1. Upgrade notes

2. New Features

2.1. Acquisitions

2.1.1. Lineitem Detail Audit Table

The default schema has added an audit table for the acq.lineitem_detail table. The audit table is NOT created during database upgrade. If you wish to add the audit table to your Evergreen installation, you can run the following SQL in your database:

SELECT acq.create_acq_auditor ( 'acq', 'lineitem_detail' );
CREATE INDEX acq_lineitem_detail_hist_id_idx ON acq.acq_lineitem_detail_history( id );
CREATE INDEX acq_lineitem_detail_hist_lineitem_idx ON acq.acq_lineitem_detail_history( lineitem );
CREATE INDEX acq_lineitem_detail_hist_fund_debit_idx ON acq.acq_lineitem_detail_history( fund_debit );

2.2. Administration

2.2.1. Enhanced Concerto dataset

An alternative sample dataset called "Enhanced Concerto" is now available. This dataset includes a more realistic organizational unit structure and additional data with which to test Evergreen functionality.

The "Enhanced Concerto" dataset supports human-driven testing and community demonstrations. However, at present, automated tests, particularly the ones found in Open-ILS/src/sql/Pg/live_t and Open-ILS/src/perlmods/live_t, are not guaranteed to pass with the "Enhanced Concerto" dataset; instead, they remain targeted to working with the original "Concerto" dataset.

Installing the dataset

The dataset can be loaded when installing Evergreen by using the --load-concerto-enhanced option of eg_db_config. For example:

perl Open-ILS/src/support-scripts/eg_db_config --update-config \
       --service all --create-database --create-schema --create-offline \
       --user <user> --password <password> --hostname <hostname> --port <port> \
       --database <dbname> --admin-user <admin-user> --admin-pass <admin-pass> \
       --load-concerto-enhanced

The --load-concerto-enhanced switch is mutually exclusive with the --load-all-sample switch to load the full original "Concerto" dataset and the --load-concerto switch to load just the bibs, items, and authorities from the original "Concerto" dataset.

Updating the dataset

An experimental script called make_concerto_from_evergreen_db.pl is included to support

  • updating the dataset as Evergreen’s schema evolves

  • incorporating changes to the dataset from a running Evergreen system

The script compares data between different versions of an Evergreen database depending on how it is invoked. This script has known bugs and its output must be manually reviewed before being committed.

2.2.2. --pipe option for marc_export

Add a --pipe option to marc_export to force reading of record ids from standard input when one or more of the --library, or --descendants options are used. This permits more flexibility when exporting records with holdings for given libraries or subsets of a collection.

It is an error to specify the --all or --since and --pipe options in the same command line.

        --pipe

Some examples:

        --pipe --library BR1   [Filter the list of bib ids by those
                                with BR1 as the Owning Library]

        --pipe --descendants BR1 [Filters the list of bib ids by those
                                  with BR1 or descendants as the Owning Library]

2.2.3. --strip option for marc_export

The --strip option is used to suppress generation of specified elements from the marc_export output. The option, which can be specified more than once, is in one if these forms:

        --strip <field RE>/<subfield RE>
        --strip /<subfield RE>
        --strip <field RE>

Some examples:

        --strip 856/0   [Delete subfield 0's in fields with tag 856.]

Regular expressions are accepted:

        --strip 8../0   [Delete subfield 0's in fields with tag 800-899.]

If the field is omitted, it is as if you specified "..." for the field RE.

    --strip /0      [Delete subfield 0's in all fields.]
    --strip /[abc]  [Delete subfield a, b or c in all fields.]

If the slash and subfield are omitted, it means to delete the given fields.

    -strip 856     [Delete fields with tag 856]

If the slash is present, but the subfield is omitted, it means "all subfields"

    --strip 856/    [Delete all subfields with tag 856]

2.2.4. Applying Missing Database Upgrades

This patch fixes a situation where an Evergreen database that had been been upgraded to 3.6.0 at some point in its past using the 3.5.1-3.6.0 DB update script may be missing some DB revisions.

The following bugfixes are affected:

2.2.5. Improvements to Mapping Permissions to Groups

The dialog to add permission mappings to a group in the Permission Groups administrative interface now allows multiple permissions to be added in one invocation. This reduces the number of clicks and keyboard interactions required when adding a large number of permission mappings.

2.3. Architecture

2.3.1. Malicious Search Protection

Evergreen sometimes sees some "novel" query strings in the wild that cause the search backend to time out or worse. These are sometimes malicious and sometimes accidental, but the effect on users is the same.

The changes here improve query compilation in several respects in order to reduce the chances of an overly complex query causing problems for the search subsystem.

More work is done up front to simplify and combine parts of the resulting SQL, allowing more work to be done closer to the data. This change allows Evergreen to handle many more tested or chained boolean expressions, and negated terms are now handled directly in line with other adjacent terms. Phrases (exact matches) are now searched for using Postgres' adjacency tsearch operator.

All of these changes work together to improve performance by getting more search work done in fewer database operations while protecting against certain query constructs that have caused problems in the past.

2.3.2. Angular and Bootstrap Updates for Staff Interface

The Angular staff interface now uses Angular 15 and Bootstrap 5. This introduces various changes to how the staff interface styling is done; developers should take note.

2.3.3. Fix SQL Injection Vulnernability

An SQL injection vulnernability related to the implementation of search term highlights is now closed.

2.3.4. Queued Ingest

This feature allows for the separation of bib and authority record updates and the search (and other) indexing that occurs when a record is modified in some way. Prior to this feature, bib and authority records would be indexed immediately upon an update.

While individual record ingest has not become a problem with regard to system performance or interface usability, there exist several batch operations which aggregate many inserts or updates and whose aggregate ingest time cost can be significant. These include, but are not limited to, reingest caused by authority control propagation, reingest required by the addition or modification of indexing configuration, cataloging and acquisitions record import and overlay from the staff interface, and upgrade-time reingest required by structural changes to the underlying indexing and search system.

New Utility

When Queued Ingest is enabled, a new control script, ingest_ctl, is available to perform several functions:

  • Run in the background to process the queues of indexing requests

  • Display statistics of queued ingest activity

  • Specify that a set of records should be reindexed.

Here are some examples of how it is used:

# Enqueue records 1-500000 for reingest later, just one worker for the queue
/openils/bin/ingest_ctl --queue-threads 1
    --queue-type biblio
    --queue-run-at tomorrow
    --queue-owner admin
    --queue-name "slowly updating records due to new RDA attributes"
    --start-id 1 --end-id 500000

# Start the background worker
/openils/bin/ingest_ctl --coordinator --max-child 20

# Stop the background worker
/openils/bin/ingest_ctl --coordinator --stop

# Process whatever you can Right Now
/openils/bin/ingest_ctl --max-child 20

# Process a single queue Right Now
/openils/bin/ingest_ctl --queue 1234 --max-child 20

# Stats on Queued Ingest processing so far today
/openils/bin/ingest_ctl --stats --since today --totals-only

This script also requires the following switches (or environment variables) in order to connect to the database:

  • --db_user (or environment variable PGUSER)

  • --db (or environment variable PGDATABASE)

  • --dbpw (or environment variable PGPASSWORD)

  • --db_port (or environment variable PGPORT)

New Settings

This feature adds several new global flags:

Global Flag

Enabled

Queued Ingest: Abort transaction on ingest error rather than simply logging an error

no

Queued Ingest: Queue all bib record updates on authority change propagation, even if bib queuing is not generally enabled

no

Queued Ingest: Use Queued Ingest for bib record ingest on insert and undelete

no

Queued Ingest: Use Queued Ingest for authority record ingest on insert and undelete

no

Queued Ingest: Use Queued Ingest for bib record ingest on update

no

Queued Ingest: Use Queued Ingest for authority record ingest on update

no

Queued Ingest: Use Queued Ingest for bib record ingest on delete

no

Queued Ingest: Use Queued Ingest for authority record ingest on delete

no

Queued Ingest: Maximum number of database workers allowed for queued ingest processes

yes; default value 20

Queued Ingest: Use Queued Ingest for all bib record ingest

no

Queued Ingest: Use Queued Ingest for all bib and authority record ingest

no

Queued Ingest: Do NOT use Queued Ingest when creating a new bib, or undeleting a bib, via the MARC editor

yes

Queued Ingest: Use Queued Ingest for all authority record ingest

no

Queued Ingest: Do NOT Use Queued Ingest when editing bib records via the MARC Editor

yes

This feature does not add any new library settings or permissions.

Upgrade Notes

Queued Ingest is not automatically turned on upon upgrade. to enable it, at minimum the following actions should be taken:

  • Enable at least the "Use Queued Ingest for all bib and authority record ingest" global flag

  • Ensure that ingest_ctl is running with the --coordinator flag.

2.4. Cataloging

2.4.1. Catalog Staff View

Adds a Staff View tab to the record details page in the staff catalog. In addition to showing various fields from the bib record, it also shows a count of hold requests and a break-down of available items versus total items, and it shows a breakdown of similar records by formats and language editions, based on the associated metarecord.

2.5. Circulation

2.5.1. Adjust Hold Cancel Reasons

The dialog to confirm canceling a hold in the staff interface now lists only hold cancel reasons that can be manually applied, excluding the ones that can only be automatically applied.

In addition, two more hold cancel reasons are added:

  • Patron via email

  • Patron via SMS

2.5.2. New Permission for Viewing Holds Pull List

Access to the holds pull list is now restricted to users with the VIEW_HOLD_PULL_LIST permission. By default, this new permission is granted to all users with the VIEW_HOLDS permission.

2.6. Client

2.6.1. Angular navbar keyboard navigation

Adds basic keyboard navigation support to the Angular web client navbar as follows:

  • Use the Tab key to move between the top-level menu items.

  • Press Enter, Space, or the down arrow on a top-level item to open its submenu. Pressing Esc will close it.

  • Shift-tab from the first submenu item back to the top level, and use Enter or Space to toggle the button again and close the submenu.

  • Within the submenu, use either Tab and shift-Tab to move up and down, or the up/down arrow keys.

2.7. OPAC

2.7.1. Multi-term Did You Mean search suggestions

Expanding on the previous single-class, single-term search suggestion development, this feature provides suggestions for single-class searches with multiple terms.

  • The Library Settings that were previously used to control the global behavior of search suggestions have been moved to search class configuration fields. This was done because the data in each search class benefits from different setting values. If the Library Settings had been set, they will be used to update the corresponding search class configuration fields. Regardless, Evergreen administrators are recommended to review the settings.

  • If a patron’s search matches a variant or non-preferred heading from an authority record, if the main heading of that authority is linked to at least bibliographic record, the system will provide that main heading as a suggestion as well, along with spelling-corrected suggestions.

  • Quoted phrases in user input require strict term order and adjacency for the phrase portion of the suggestion generated for the phrase(s), whereas unquoted input (or the portion that is not quoted) does not.

MARC Search/Facet Class field additions
  • variant_authority_suggestion Whether this class should attempt variant authority suggestions based on search-class/browse-axis mapping

  • symspell_transfer_case Whether suggestions should retain user-supplied letter case

  • symspell_skip_correct Only supply suggestions to misspelled words

  • symspell_suggestion_verbosity Setting that controls the amount of effort, and therefore time, spent on suggestion generation

  • max_phrase_edit_distance Maximum average per-word edit distance when evaluating suggestions

  • suggestion_word_option_count Maximum alternate suggestions per word

  • max_suggestions Maximum suggstions to present

  • low_result_threshold Maximum hit count beyond which suggestions are not provided

  • min_suggestion_use_threshold Minimum number of times a suggestion must exist in the corpus

  • pg_trgm_weight Weight of the trigram similarity metric; 0 avoids calculation costs

  • soundex_weight Weight of the soundex similarity metric; 0 avoids calculation costs

  • keyboard_distance_weight Weight of the keyboard distance similarity metric; 0 avoids calculation costs

Upgrade

If the databse has authority records that are linked to bilbiographic records, a reingest of the search suggestion dictionary is recommended.

Instructions for performing that reingest are included in the database update scripts and will be output to the log when those scripts are run.

2.7.2. Restrict login redirect

As a security best-practice, Evergreen should not allow arbitrary redirection on successful login, but instead limit redirection to local links or configured domains and schemes.

This feature is controlled by a new global flag called opac.login_redirect_domains which must contain a comma-separated list of domains. All hostnames under each domain is allowed for redirect, and the scheme of the redirect URL must be one of http, https, ftp, or ftps.

2.7.3. Accepting payments with SmartPAY

SmartPAY is a payment processing service that lets sites take credit card payments without payment card information ever touching the sites' own servers.

Library Settings

The following settings need to be set at the appropriate org level for sites wanting to use SmartPAY.

  • "Allow Credit Card Payments" (should be true)

    credit.payments.allow
  • "Enable SmartPAY payments" (should be true)

    credit.processor.smartpay.enabled
  • "SmartPAY location ID" (value provided by Comprise)

    credit.processor.smartpay.location_id
  • "SmartPAY customer ID" (value provided by Comprise)

    credit.processor.smartpay.customer_id
  • "SmartPAY login name" (value provided by Comprise)

    credit.processor.smartpay.login
  • "SmartPAY password" (value provided by Comprise)

    credit.processor.smartpay.password
  • "SmartPAY API key" (value provided by Comprise)

    credit.processor.smartpay.api_key
  • "SmartPAY server name" (value provided by Comprise)

    credit.processor.smartpay.server
  • "SmartPAY server port" (value provided by Comprise)

    credit.processor.smartpay.port
  • "Name default credit processor" (should be SmartPAY)

    credit.processor.default

2.8. Miscellaneous

  • Add patron home library code as a column to the View Holds grid in the staff catalog record details page (LP#1991726)

  • Include template ID in the template table in the Reporter (LP#1998386)

  • Remove the pub flag from the biblio.record_note table (LP#1978978)

  • Add the publication date to the Staff Catalog’s Shelf Browse (LP#1999432)

  • Resolve search performance degradation with PostgreSQL version 12 and up (LP#1999274)

  • Improved styling of paid line items in acquisitions screens (LP#1999270)

  • Improved styling of the keyboard shortcut info modal (LP#1999955)

  • (Developer) Add Emacs mode to fm_IDL.xml (LP#1914625)

  • autogen.sh can now accept a -c switch to specify the location of opensrf_core.xml. This is useful for certain multi-tenant setups of Evergreen. (LP#2003707)

  • Better organization of acquisitions line item alert fields (LP#2002977)

  • Prevent templates from applying or changing magical status in angular holdings editor (LP#1999401)

  • Prevent directly editing the shelving location deleted field in the Shelving Locations Editor (LP#2002435)

  • The "Strict Barcode" checkbox is now closer to the barcode input on the Check Out, Check In, and Renew Items pages (LP#1990968)

  • LP1929593 UPDATE_COPY_BARCODE permission

    • This adds the permission UPDATE_COPY_BARCODE and a new API call,

      open-ils.cat.update_copy_barcode
    • which explicitly tests for both UPDATE_COPY_BARCODE and UPDATE_COPY, with either being sufficient for allowing a barcode change. Existing Replace Barcode UI’s in both Angular and AngularJS have been modified to use this API call instead of the pcrud service. One side-effect of this has been better surfacing of errors, as errors in pcrud were uncaught and bypassing the normal error handling. This addresses LP1951469.

    • The upgrade script gives any permission groups that already have the UPDATE_COPY permission the new UPDATE_COPY_BARCODE permission at the same depth, though it’s technically not needed.

  • Patron and staff login forms now include a button to reveal the password input. (LP#1977554)

  • Adds new Local Administration entries for Item Statistical Categories Editor and Patron Statistical Categories Editor, which are angularized interfaces.

  • Tweaks eg-grids to underline hyperlinks within cells. This potentially affects multiple interfaces.

  • eg-org-family-select now supports persistKey

  • LP1965446 Option to Disable Title-Level Holds on Bib Records with Parts

    • This feature adds one global flag and one library setting, respectively:

      • circ.holds.api_require_monographic_part_when_present Holds: Require Monographic Part When Present for hold check.

      • circ.holds.ui_require_monographic_part_when_present Require Monographic Part when Present

    • Normally the selection of a monographic part during hold placement is optional if there is at least one copy on the bib without a monographic part. A true value for this setting for any involved owning library for the bib or for the global flag will require part selection even under this condition. This essentially removes the All/Any Parts option from the part selection drop-down, for both versions of the public catalog (TPAC and BOOPAC), and for the Angular staff catalog interface. It should be noted that if the library setting is set below the consortium level, Title level holds may be allowed for some libraries and not others.

    • At the API level, we consider just the global flag and will throw a TITLE_HOLD_WHEN_MONOGRAPHIC_PART_REQUIRED event for a title hold request when there are items with monographic parts on the bib. It is possible for the library settings and the global flag to differ, but the global flag will catch every instance of hold placement including those by third party callers, SIP, etc.

  • Links the lineitem id link in Acquisitions search results to the lineitem detail page rather than the purchase order. (LP#2003946)

3. Acknowledgments

The Evergreen project would like to acknowledge the following organizations that commissioned developments in this release of Evergreen:

  • CW MARS

  • Evergreen Community Development Initiative

  • Evergreen Indiana

  • King County Library System

  • Pennsylvania Integrated Library System

  • Westchester Library System

We would also like to thank the following individuals who contributed code, translations, documentations patches and tests to this release of Evergreen:

  • John Amundson

  • Scott Angel

  • Jason Boyer

  • Dan Briem

  • Andrea Buntz Neiman

  • Eva Cerninakova

  • Galen Charlton

  • Garry Collum

  • Elizabeth Davis

  • Jeff Davis

  • Britta Dorsey

  • Bill Erickson

  • Jason Etheridge

  • Ruth Frasur

  • Jeff Godin

  • Blake Graham-Henderson

  • Rogan Hamby

  • Elaine Hardy

  • Stephanie Leary

  • Clayton Liddell

  • Shula Link

  • Tiffany Little

  • Mary Llewellyn

  • Debbie Luchenbill

  • Llewellyn Marshall

  • Terran McCanna

  • Chrystal Messam

  • Gina Monti

  • Christine Morgan

  • Michele Morgan

  • Susan Morrison

  • Susasn Morrison

  • Dan Pearl

  • Jennifer Pringle

  • Mike Risher

  • Mike Rylander

  • Jane Sandberg

  • Chris Sharp

  • Ben Shum

  • Jason Stephenson

  • Josh Stompro

  • Elizabeth Thomsen

  • Jennifer Weston

  • Beth Willis

  • Carol Witt

  • Adam Woolford

  • Jessica Woolford

We also thank the following organizations whose employees contributed patches:

  • BC Libraries Coop

  • Bibliomation

  • Catalyte

  • CW MARS

  • Equinox Open Library Initiative

  • Evergreen Indiana

  • Georgia Public Library Service

  • Kenton County Library

  • King County Library System

  • Lake Agassiz Regional Library

  • LibraryMarket

  • Linn Benton Community College

  • MOBIUS

  • NC Cardinal

  • NOBLE

  • Princeton University

  • Sigio

  • Westchester Library System

We regret any omissions. If a contributor has been inadvertently missed, please open a bug at http://bugs.launchpad.net/evergreen/ with a correction.