Evergreen ILS

Open Source Library Software

The birth of OpenSRF

by on

After months of work, buckets of sweat, and one snapshot of code seemingly unrelated to libraries, we are proud to present our first subproject: OpenSRF (Open Scalable Request Framework), pronounced “open surf”. The 0.1 release of OpenSRF represents the culmination of everything we have posted about here, and some things we haven’t.

One of the main differences between our initial snapshot tarball and the shiny new OpenSRF code is the removal of the dependence on an Auth Application. We have decided that the security provided by Jabber is sufficient for connection to the OpenSRF network, and application level authentication and authorization should be handled either by each individual Application, or by an Authentication Application built specifically for the target problem domain.

I’ll start of with a bullet list of our current features, then give you a rough roadmap of where we want to take OpenSRF in the future.

Current Features

  • Secure message passing via Jabber
    • SSL connections
    • Authenticated clients and servers
    • Built in anti-spoofing
  • Scalable architecture
    • Built in load balancing at both the Messaging and Application layers
    • Built in fault tolerance through error and timeout detection
    • Lighting fast, very low overhead; requests against simple Applications show a 35 millisecond maximum overhead for session setup on a cluster of 2 servers and less than 10 milliseconds of overhead for additional requests on established sessions
    • Cluster can grow simply by adding more servers
    • Any cluster member can run any Application
    • Instances of Applications implement Servers of a particular Server Class; these Servers can register with multiple Routers
    • Supports multiple Jabber domains, and each domain’s Router can accept registrations from Servers in other domains, providing for topological redundancy as well as local Server failover
  • Simple API for Application developers
    • Applications are implemented as Perl modules
    • Network communication is standard XML
    • Data structures are serialized from language-native objects to JSON, which is language-agnostic, more compact than XML, and easily converted back to language-native objects, no matter what languages each end of the transaction are implemented in
    • Client libraries for Perl, C and ECMAScript (JavaScript)
    • Applications can respond with atomic results, or stream large result sets asynchronously
    • Client/Server communication can be used in both synchronous and asynchronous modes from the client’s perspective
    • Failover, session failure and “max requests” situations are handled by OpenSRF, allowing developers to focus on Application logic

The future is not written … well, kind of

  • Version 0.2 (RSN)
    • Distributed Object Persistence Application implementing Stacks, Queues and Object Stores
  • Version 0.3 (March/April 2005)
    • Centralized configuration management
    • Partial implementation of a BROADCAST mode for addressing all Servers of a particular Class on one Router
  • Version 0.5 (July/August 2005)
    • Full implementation of a BROADCAST mode for addressing all Servers of any number of Classes on any number of Routers
  • Version 0.7 (October/November 2005)
    • Basic inter-server session migration support in addition to session failure recovery
  • Version 0.9 (Some time in 2006)
    • Built in distributed transaction support.
  • Version 1.0 (in the future)
    Who knows?

What does it all mean?!?!

Why is this subproject important to the success of Open-ILS and Evergreen? This can best be explained through example:

Around the 10th of December ’04 we started work on the Demo OPAC linked from the January Executive Committee report. We started with code that was significantly less complete compared to what we eventually posted as the pre-alpha snapshot tarball, and didn’t really expect to get too much done before the end of the year. We all had plans to be out of town, and in the US everyone generally loses about half of December to Holiday-itis.

Well, imagine our surprise when sometime during the first week in January we had a working — albeit ugly(er) and slow — OPAC going! We were expecting to have to put in at least two weeks of work into debugging and reworking the transport code (ney! OpenSRF), but even in its initial state we could forget about the underlying network and simply focus on tuning the SQL and getting the interface right.

As you might imagine, we are pretty proud of OpenSRF even now, before it has most of it’s enterprise class features. Our hope now is to get more people interested the framework as it has already served us very well. With the addition of a Broadcast mode for informing multiple Persistence Servers of global object updates, and eventually real ACID style transaction support built directly into the messaging layer we are targeting a sort of self-tuning, loosely coupled, transparent cluster architecture that will make large, hairy, complex applications much easier to write, and cut time spend on building scalability into each individual application down to a minimum.

As always, we welcome any feedback you have. You can reply here, or join one (or all) of our new mailing lists from here. You can grab this shiny new code from our anonymous CVS repository, or browse the source online.