The piece that everyone wants added to Evergreen is an acquisitions system. I (Dan Scott, from Laurentian University) am one of the people trying to make that happen. What follows is a snapshot of the latest acquisitions and serials development sprint weekend. I apologize in advance for any of the activities I missed or details that I have confused, especially as the weekend progressed and my mind grew slightly skewed from cramming 30-odd hours of hard work into three days.
Thursday: The Canadians Have Landed
On the afternoon of Thursday, December 13th, three Canadians (Brandon Uhlman from BC Pines, Art Rhyno from the University of Windsor, and I) arrived in Atlanta for AcqFest II. No international flight would be complete without some missing luggage (Brandon’s) and border-crossing hassles (Brandon again, and Art too – but he expects it by now). Bill Erickson rounded us up and brought us into Equinox’s shiny new office in Norcross, GA. This gave us an opportunity to meet the two newest members of Equinox (Don McMorris and Bob Molyneux) and kick off the event with Art’s presentation, cleverly titled Well, now what?.
Whereas the focus of the first AcqFest was to gain a common understanding of the problem domain, the focus of AcqFest II was on Getting Things Done, and Art had mapped out a number of tasks for us to tackle on building the acquisitions and serials system, nicknamed “Woodchip”, around the opentaps open source ERP and CRM system. A great benefit of opentaps (an extension of the Apache OFBiz ERP project) is that all of the financials business logic is already implemented according to generally accepted accounting principles, which meant that we would just have to figure out how to customize and augment its capabilities for the library world and hook it up to Evergreen. A drawback is that the Java base and many layers of functions in opentaps/OFBiz are a deterrent for some potential contributors, and Art had expected to be in a more benevolent position with availability than has happened this Fall due to mysterious and confidential events in his newspaper business.
Friday I’m in (Bots) Love
On Friday, we rolled into the office at 9:00 am and settled into our self-assigned tasks.
Art disappeared into an office for the bulk of the day, muttering something about giving thanks for being able to spend extended periods of time in an office without a ringing telephone. He worked on adding library attributes to the generic opentaps catalog so that one could lookup items by fields like ISBN, ISSN, and publisher in addition to the standard opentaps attributes. The additions turned out to be trivial but exposing the attributes for searching was far trickier. Like seemingly every second system on earth, opentaps/OFBiz uses lucene for indexing but implements a constraint-based searching mechanism that hooks into the relational database modeller, making for a rather complex coding exercise when trying to map attribute entries without impinging on the general indexing. The advantages of ample whiteboards and focused environments became evident when this was sorted out.
Mike created an XSLT script to transform MARC21XML into the desired subset of bibliographic attributes to feed into the Woodchip “desiderata” catalog, then began building out the infrastructure required for broadcasting a search against multiple Z39.50 targets. Bill tackled the Woodchip build with gusto, trying to minimize the number of modified opentaps files stored in the Woodchip Subversion repository by generating the modifications at build time. Woodchip requires a whopping 250 MB download because of the opentaps/OFBiz dependencies, and changes are scattered among many different directories and glue technologies, for example, beanshell and freemaker templates, which present challenges for maintaining a coherent tracking regime. Bill achieved a strategy and implementation for keeping this in sync.
I began investigating what would be required to integrate EDI capabilities into opentaps, but after a few hours discovered that there were no actively maintained, fully open-source EDI packages available in Java or Perl. Fortunately, however, I found Bots – an actively maintained EDI parser and generator package that seems more than capable of fulfilling the requirements of EDI for the library domain with support for EDIFACT and X12 standards.
As a Python application, Bots will introduce a new language requirement to the Evergreen stack – but one of Evergreen’s design philosophies is to integrate the best of breed components where possible, and Bill had already created basic OpenSRF bindings for Python, so the fit is right. In addition, as the volume of EDI transactions for any given library will be relatively small, we will be able to run Bots as a stand-alone application that periodically converts documents between the native EDI format and JSON.
Now that we had a real Python application to integrate, and realizing that the OpenSRF 1.0 release was just around the corner, I began to adapt the OpenSRF bindings to follow standard Python coding conventions. It’s both a code hygiene issue and a recruitment tactic. We want to encourage more developers to join the project, and if a Python developer encounters a properly Pythonic API, they can focus on learning how OpenSRF and Evergreen works rather than dealing with the cognitive dissonance of an unconventional API. Thus began several days of running pylint, making global changes, and running pylint again.
The day wrapped up at approximately 10:30 pm – a full day of putting our noses to the grindstone, with breaks only for ordering and consuming delivered food. We had the pleasure of meeting Sally “Murph” Murphy, the Quality Assurance Engineer for Evergreen who was hired by GPLS back in September 2007. It was great to meet another key part of the Evergreen team, and for those who have not yet had the pleasure, I can assure you that Murph not only packs strong technical chops, she also wields a dry wit that fits right in with the rest of the Evergreen team.
Serials Saturday
Friday’s efforts left us a little drained, so Saturday resumed at a slightly slower pace. We assembled at the office around 10:00 am and managed to get in a few hours of work before a number of friends from Georgia PINES and Evergreen family members dropped by for a holiday lunch. After lunch, Art and Mike broke off for an hour of discussions in Mike’s office before returning with the declaration “Serials is done!”. And there was much rejoicing.
From what I understood, the basic approach for serials would be to overlay multiple instances of the production schedules built into opentaps, plus exceptions, to establish the serials patterns – for example, a serial published on a monthly basis, with an extra issue in December, would combine a monthly schedule plus an annual schedule. Some details were captured on a cell phone image of the whiteboard, but the full details will require the development of a brain-to-information interface; something like an orange juice extractor, or perhaps a slightly less destructive approach such as an forced writing session for one of the collaborators.
Bill plugged Mike’s XSLT script into the Woodchip “desiderata” catalog search widget, enabling a search against the Evergreen catalog to return the desired attributes and laying the groundwork for searching Z39.50 sources directly. I persevered with my Python pedanticism, pausing only to hurl a “MULTIZ KTHXBYE” at poor Mike on IRC as he was involved in a conversation not involving acquisitions or serials. After a trip to Fry’s and a fabulous dinner, Mike and I returned to the office to put in another couple of hours on our respective tasks.
Sunday, Blissful Sunday
On Sunday morning, Bill and Mike helped me debug a few problems that had crept into my Python refactoring work. As a reward for having gone through the OpenSRF API with a fine-toothed comb, I was able to quickly add locale support to the Python bindings (requiring all of six lines of additional code). Mike completed his effort of making the Evergreen catalog able to appear as just another Z39.50 target for the purposes of the acquisitions search, exposing an interface that can be used to wrap other services, such as an Amazon search. Bill added a multi-session manager to the Java OpenSRF bindings.
And with that, AcqFest II drew to a close. The progress that we made this time around was quite tangible – almost 40 commits to the Evergreen, OpenSRF, and Woodchip repositories, including improved infrastructure and new core functionality. We also found the solution to our EDI needs and clarified our functional requirements and technical plans. In follow-up conversations, we continued our dialogue on the idea of shifting the role of opentaps/OFBiz to more of a prototype engine and fostering more rapid development with one of the OSS ERP engines that could fit into a Pythonic space, possibly TinyERP. Watch this space — I’m looking forward to the next few months of development in this area.
A special thanks to Art Rhyno for his comments and contributions to my first draft of this blog post!