1. Upgrade notes

2. New Features

2.1. Administration

2.1.1. Add Date Header to Action Trigger Email/SMS Templates

The Date: header specified in RFC 2822 has been added to the seed data for the example Action Trigger email and SMS templates, but no attempt has been made to automatically modify existing templates. To add this header (and end any "Why are my library emails from 1969/70?" questions you may have heard) make sure the following lines are in all templates that use the SendEmail or SendSMS reactors:

The first is already in most sample templates, but you may need to add it to the top of any custom templates: [%- USE date -%]

And this line should be inserted into the header block of each template: Date: [%- date.format(date.now, '%a, %d %b %Y %T -0000', gmt => 1) %]

2.1.2. Tablefunc Extension No Longer Required

Changes in the behavior of the connectby function in PostgreSQL 9.5 have prompted its removal from the database. It is easier for Evergreen to maintain compatibility with previous versions of PostgreSQL without this function.

By eliminating the use of the connectby function, we eliminate the requirement for the tablefunc database extension. It is no longer installed when the database is created. If you are upgrading and wish to remove it from your database, you may run the following statement in the database to drop it:

DROP EXTENSION tablefunc;

2.1.3. Purge User Activity

User activity types are now set to transient by default for new Evergreen installs.. This means only the most recent activity entry per user per activity type is retained in the database.

This change does not affect existing activity types, which were set to non-transient by default. To make an activity type transient, modify the Transient field of the desired type in the staff client under Admin → Server Administration → User Activity Types.

Setting an activity type to transient means data for a given user will be cleaned up automatically if and when the user performs the activity in question. However, administrators can also force an activity cleanup via SQL. This is useful for ensuring that all old activity data is deleted and for controlling when the cleanup occurs, which may be useulf on very large actor.usr_activity tables.

To force clean all activity types:

SELECT actor.purge_usr_activity_by_type(etype.id)
    FROM config.usr_activity_type etype;
Note
 This could take hours to run on a very large actor.usr_activity table.

2.2. Cataloging

2.2.1. Authority Record Import Updates Editor, Edit Date.

Importing an authority record via MARC Batch Import/Export now causes the authority record’s editor and edit_date fields to be updated. The editor value may come from the MARC 905u field or, if none is present, the user performing the import.

2.2.2. Authority Propagation Updates Bib Editor / Edit Date

When a bib record is automatically updated as a result of the modification of a linked authority record, the bib record’s "Last Edit Data/Time" and "Last Editing User" fields will be updated to match the time of the update and the editor of the modified authority record.

A new global flag is available to control this behavior called ingest.disable_authority_auto_update_bib_meta ("Authority Automation: Disable automatic authority updates from modifying bib record editor and edit_date"). When enabled, theses fields will not be updated. By default, this setting is disabled.

An additional speed improvement is included in this feature. No attempt will be made to update linked bib records when the normalized heading of the modified authority record is unchanged by the authority record update.

2.2.3. Bibliographic record source now copied to 901$s

If a bibliographic record has a source set, the name of that source is now copied to the 901$s whenever the record is created or updated. This allows the source to be used for record matching and MARC field queries.

2.2.4. Selectable Bibliographic Source Update

During vandelay, the bib source has always recorded the record source, the time of the update and the identity of the user associated with the operation. This is not really desired for match-only merges.

This feature provides a way to control whether the bib source data is updated or not.

In MARC Import, select the "Merge / Overlay" tab. Each entry in the table has a value in the new "Update bib. source" column. If that value is "true", then the bib source data will be updated.

The two system-defined entries have been pre-set to appropriate values (Full Overlay = true; Match-Only Merge = false).

2.3. Circulation

2.3.1. Staff Client Honors Aged Circulations

The browser and XUL clients now better represent copy checkout history by honoring and displaying information from aged circulations.

  • Browser client Recent Circ History and the analogous XUL client Circulation History tabs show summary data for aged circulations as well as regular/active circulations. When aged circulation data is displayed, any references to patron names are replaced by the string "<Aged Circulation>".

  • Browser client Circ History List and the analogous XUL client Last Few Circulations tabs behave as above, plus their Add Billing buttons are disabled when displaying aged circulation data.

  • XUL client Retrieve Last Patron actions from various UI’s report, "Item XXX circulation is an aged circulation and has no linked user". Browser client analog uses Circ History List instead, no additional changes required.

2.3.2. "Canceled Transit" Item Status

Previously, when a transit was aborted, the transited item would either go into "Reshelving" status or would return to whatever status it was in when it went into transit, even when the item itself was in a different status (including "Checked out"). Now, for most transits that get aborted, the item is put into a new status, "Canceled Transit", which signals to staff the actual state of the item. This feature only affects items with a status of "In transit" and does not affect items that were in the following statuses at the time they were sent into transit:

Any custom statuses

This change should help clear up confusing situations caused by the previous "abort transit" behavior, such as items showing "Available" when they are actually en route, and patrons' items mysteriously disappearing from their accounts and showing "Available" at the item-owning library without evidence of check-in.

2.3.3. Copy Status "Is Available" Flag

Adds a new boolean field for copy statuses to indicate when copies having a given status should be considered available. The field has 2 main effects:

  1. Checking out an "available" copy will no longer result in an override-able "COPY_NOT_AVAILABLE" alert for staff. The copy will checkout without status warnings.

  2. "Available" copies will appear in catalog searches where "limit to available" is selected as a search filter.

By default, the "Available" and "Reshelving" statuses have the "Is Available" flag set. The flag may be applied to local/custom statues via the copy status admin interface.

2.3.4. Email Checkout Receipts

This feature allows patrons to receive checkout receipts through email at the circulation desk and in the Evergreen self-checkout interface. Patrons need to opt in to receive email receipts by default and must have an email address associated with their account. Opt in can be staff mediated at the time of account creation or in existing accounts. Patrons can also opt in directly in their OPAC account or through patron self-registration. This feature does not affect the behavior of checkouts from SIP2 devices.

Patrons can opt in to receipt email checkout receipts by default via a new "Email checkout reciepts by default" patron setting.

This feature also enhances the patron staging tables so that patron settings can be chosen during self-registration.

The web staff interface’s checkout screen now includes a "Quick Receipt" button that allows staff members to generate a receipt at any time.

2.3.5. Set Per-OU Limits on Allowed Payment Amounts

Two new OU Settings have been added to prevent clerks from accidentally clearing all patron bills by scanning a barcode into the Payment Amount field, or accidentally entering the amount without a decimal point (such as you would when using a cash register).

The first setting is the amount above which staff will be asked if they’re sure they want to apply the payment, the second is the maximum amount of money that can be accepted through the staff client.

These settings only effect the staff client, not credit cards accepted through the OPAC, or direct API calls from third party tools.

2.4. Client

2.4.1. Additional Fields Available for Display in Some Interfaces

The holds age protection field will now be available for display in the following interfaces:

  • Item status list view column picker

  • Item status alternate view

  • Holdings maintenance column picker

The asset.copy.cost field, which records the amount paid for an item when an invoice is processed, will be available for display in the following interfaces:

  • Items status list view column picker

  • Item status alternate view

  • Copy editor

2.5. OPAC

2.5.1. Merge Notification Preferences Tables in TPAC

The patron notification preference page in the public catalog used to have two tables, separating notification settings based on their source. Since that distinction does not matter to patrons, and since the two tables aren’t styled consistently, they are merged together.

2.5.2. Improved Holds Screens in My Account

The grids in the My Account Items on Hold and Holds History interfaces are simplified. Data previously contained in their own Activate, Active, and Date Fulfilled columns are now incorporated into the Status columns. To further declutter the interface, the holds queue position will only show when the user most needs the information - before the hold has been captured.

Distinct CSS classes have also been added for each hold status and each date that could potentially display in these holds interfaces. A new default style highlights the Available status in green and the Suspended status in red.

2.5.3. Statistically generated Record Ratings (Popularity)

Summary

For the purpose of supplying non-bibliographic popularity adjustment to the ranking of search results, this feature implements a set of statistical modelling algorithms which will identify bibliographic records of particular note based on derivable parameters.

Generally, factors such as to circulation and hold activity, record and item age, and item ownership counts will available for statistical consideration. Each factor will embody a "popularity badge" that the bibliographic record can earn, and each badge will have a 5-point scale, where more points indicates a more popular record. The average of the badge points earned by each record will constitute a "popularity rating". The number and types of badges will break ties for average popularity, and relevance will sort items with like popularity.

A new sort axis of Popularity is created to sort first on the weighted average popularity of each record, followed by the query-specific relevance available today. A new option is created in the dropdown that sorts on the combination of "activity metric" (aka badge ranking, aka popularity) first and the existing, stock relevance ranking when those are equal. For instance, given two records that both have a badge ranking of "4.5", they will be sorted in the order of the query relevance ranking that is calculated today as a tie breaker. Those two records will sort above other records with lower badge rankings regardless of what today’s relevance ranking says about them.

In addition, a new sort axis of Popularity and Relevance is created that augments the normal Relevance sort with a normalized popularity value by multiplying the base relevance by a value controlled by a new global flag, generally set to a decimal number between 1 and 2.

Finally, there will continue to be a pure Relevance sort option, which is the version that exists today.

A global flag will allow the selection of the default sort axis.

The basics

There will exist two classes of non-bibliographic popularity badge: point-in-time popularity, such as the number of items held or the number of open hold requests; and temporal popularity, such as circulations over the past two years or hold requests placed over the last six months.

Each popularity badge will have a definition. The badge’s value are calculated for each bibliographic record within the definition’s bibliographic population. The population circumscribes the bibliographic records that are eligible to receive the badge. Each record within the population of a badge definition will receive a ranking adjustment, based on its "popularity rating" if the appropriate thresholds are met.

The set of existing popularity badges is displayed as a grid. A library selector defaulting to the workstation location will allow scoping the grid contents to specific organizational units. Creating or editing a badge is performed within a dedicated modal popup interface that will float above the grid, disabling access to the underlying interface until the action is completed or canceled.

All popularity badge definitions will describe a set of configuration, population criteria, and statistical constraints:

  • Badge Name: The administrator-assigned name of the popularity badge definition. This is presented as a text input box, and the value is used in the OPAC.

  • Scope: The owning org unit of the badge. Badges are cumulative, and are included in ranking when the Scope is equal to, or an ancestor of, the search location. Therefore branch-specific searches will include branch, system and consortium badges, but consortium-wide searches will only make use of consortium-scoped badges. For item-specific metrics, this also limits the population of the statistical group to those records having items whose owning library is at or below the Scope in the org tree. This is presented as a standard Library selector.

  • Weight: A multiplier defining the importance of this particular badge in relation to any other badges that might be earned by a record. This is presented as a number spinner with a default and minimum value of 1.

  • Recalculation Interval: How often to recalculate the badge’s popularity value for each record. For temporal popularity badges that may change quickly, such as in-house use or short-duration circulation-over-time, this may be nightly. For slowly changing metrics such as count of items held, this may be monthly or quarterly. This is presented as a text input that will accept an interval string such as "2 years", "30 days", or "6 weeks, 2 days". Numeric values without a timespan qualifier are considered to be a number of seconds. For newer items that may have rapidly changing metrics, a mechanism is created to adjust the "last calculated date" so that library staff can clear the date and force a recalculation overnight. However, because the badge value each record receives is relative to all the other records in the population, the badge as a whole will need to be recalculated. This feature stores individual record raw stats, where possible and reasonable, to speed up recalculation.

  • Population Filters: Optional, and any combination may be used.

    • Attribute Filter: Filter bibliographic record population based on record attributes. This will use an interface similar to the Composite Attribute editor.

    • Bibliographic Source: Filter bibliographic records to those with a specific source. This is presented as a dropdown of sources.

    • Circulation Modifier Filter: Include only those items that make use of certain Circulation Modifiers in statistical calculations. This is only applicable to item-related badges. This is presented as a dropdown of modifiers.

    • Copy Location Group: Include only those items that are assigned to shelving locations within specific location groups. This is only applicable to item-related badges. This is presented as a dropdown of location groups available within the Scope for the badge.

  • Popularity Parameter: One of a set of predefined types of popularity measure. This is presented as a dropdown. These will include, but may not be limited to:

    • Holds Filled Over Time

    • Holds Requested Over Time

    • Current Hold Count

    • Circulations Over Time

    • Current Circulation Count

    • Out/Total Ratio

    • Holds/Total Ratio

    • Holds/Holdable Ratio

    • Percent of Time Circulating — Of the time between the active date of the copies on the record and the badge calculation time, the percentage of that time during which the items have been checked out. This is meant to be an indicator of high-demand over the lifetime of the title, and not just a temporary spike in circ count for, say, a book club or school report. Recent temporary spikes can be represented by circs over time with a time horizon. It’s the difference between an "always popular" title and a "just recently popular" title.

    • Bibliographic Record Age (days)

    • Publication Age (days)

    • Available On-Line (for e-books, etc)

    • Copy Count

  • Fixed Rating: An optional override supplying a fixed popularity value for all records earning this badge. For some popularity types, such as "Available On-Line", it is preferable to specify, rather than calculate, the popularity value for a badge. This is presented as a number spinner with a value ranging from 1 to 5.

  • Inclusion Threshold Percentile: Cumulative distribution percentile. This is presented as a number spinner of "percentile" values ranging from 50 to 100, indicating the number of how much of the population a record’s score must be better than in order to earn the badge. Leaving this value unset will allow the entire population to earn the badge.

  • Discard most common: A value that, if greater than 0, will ignore records with extremely common values so that outliers are more readily identified, and the distribution of values can be assumed to be more normal. Many popularity parameters, such as those for circulation counts, benefit from this input filter. This is presented as a number spinner initially set to 0 that indicates the number of distinct, low values — but not the values themselves — to exclude from the population.

This new feature comes with a starter badge based on the top 97th percentile of holds requested over the past five years.

A word about Inclusion Threshold Percentile

In order to limit the amount of data that must be retained and queried during normal search operations, to allow limiting the popular population to truly exceptional records when appropriate, and to limit the speed cost of popularity ranking, many popularity types will provide thresholds that must be passed in order to store a record’s badge-specific popularity value.

The administrator will be presented with a choice of "percentile" that defines the threshold which must be crossed before the value of the variable for any given record is considered statistically significant, and therefore scaled and included in ranking. For instance, a record may need to be in the 95th percentile for Holds/Total Items before it is considered "popular" and the badge is earned.

Additionally, in order to normalize populations that exhibit a "long tail" effect, such as for circulation counts were most records will have a very low number of events, the administrator will be able to instruct the algorithm to ignore the most common low values.

Type-specific input modifications

For temporal popularity badges, two time-horizons are required. The first is the inclusion horizon, or the age at which events are no longer considered. This will allow, for instance, limiting circulation calculations to only the past two years. The second is the importance aging horizon, which will allow recent events to be considered more important than those further in the past. A value of zero for this horizon will mean that all events are seen to be of equal importance. These are presented as text inputs that will accept interval strings such as "2 years", "30 days", or "6 weeks, 2 days". Numeric values without a timespan qualifier are considered to be a number of seconds.

For those badges that have the Fixed Rating flag set, no statistical input is gathered for records in the population of the badge definition. Instead, all records in the population are assigned this fixed value for the badge.

Rating process

For badges other than those with the Fixed Rating set, the collected statistical input parameters are used to derive the mean, median, mode, min, max, and standard deviation values for the bibliographic record population. Each record passing the requisite thresholds are assigned a badge-specific value based on the quintile into which the value falls. This value, interpreted as a one-to-five rating, is stored for static application, instead of being calculated on the fly, when the badge is in scope and the record is part of a search result set. Thus records with values in the bottom quintile will have a rating of one, and records with values in the top quintile will have a rating of five.

All badge values for all records are calculated by a secondary process that runs in the background on the server on a regular basis.

Display in the OPAC

Ratings are displayed in two places within the OPAC. Like the rest of the TPAC, this is templated and display can be modified or disabled through customization.

First, on the record result page, the overall average popularity rating is displayed with a label of "Popularity" below the record-specific data such as call number and publisher, and above the holdings counts.

Second, on the record detail page, the list of badge names earned by the record that are in scope for the search location, and the 1-5 rating value of each badge, is displayed in a horizontal list above the copy detail table.

Future possibilities

This infrastructure will directly support future work such as Patron and Staff ratings, and even allow search and browse filtering based on specific badges and ratings. Additionally, bibliographic information could be leveraged to create metadata badges that would affect the ranking of record, in addition to the non-bibliographic badges described here.

Performance

It is expected that there may be some very small speed impact, but all attempts have been made to mitigate this by precalculating the adjustment values and by keeping the search-required data as compact as possible. By doing this, the aggregate cost per search should be extremely small. In addition, the development will include a setting to define the amount of database resources to dedicate to the job of badge value calculation and reduce its run time.

2.5.4. Removal of Advanced Hold Options link when part holds are expected

If a user attempts to place a metarecord hold when all eligible copies contain parts, the hold will fail. To help prevent the user from reaching a dead end while placing holds, the Advanced Hold Options link is removed from the Place Hold page in cases where all copies on the record contain parts. The Advanced Hold Options link will remain for records that have a mix of parted and non-parted copies.

2.6. SIP

2.6.1. SIP Renewals

Renewals attempted via SIP will now consider whether a penalty is configured to block renewals before blocking the renewal. Previously, any penalty, even if it wasn’t set to block renewals, would prevent a renewal from succeeding via SIP.

2.6.2. Treat SIP Location Field as Login Workstation

When using a version of SIPServer that supports the feature, the Location (CP) field of the Login (93) message will be used as the workstation name if supplied. Blank or missing location fields will be ignored. This allows users or reports to determine which selfcheck performed a circulation.

2.7. Translations

2.7.1. Translation Updates

Translations in this release have been significantly increased. In particular, Spanish has received a huge update with over 9,000 new translations, Czech has received a sizeable update of over 800 translations, and additional smaller updates have been added for Arabic, French (Canada), and Armenian.

3. Acknowledgments

The Evergreen project would like to acknowledge the following organizations who commissioned developments in this release of Evergreen:

  • TO DO

We would also like to thank the following individuals who contributed code, documentations patche and tests to this release of Evergreen:

  • TO DO

We also thank the following organizations whose employees contributed patches:

  • TO DO

We regret any omissions. If a contributor has been inadvertantly missed, please open a bug at http://bugs.launchpad.net/evergreen/ with a correction.