evergreen-ils blog

In case anybody happens to need some C that can take a char* and encode any UTF-8 byte sequences in there (as I needed to — our cribbed UTF-8 encoder proved brittle), here’s a little bit of code that may help. It’s probably more verbose than it needs to be, but it works — and working code wins!
(more…)

In an effort to maintain some mental stimulation while we go through our post-migration shake-down (i.e. Bugzilla and feature request wrangling), I’ve decided to learn Python in the off-hours. I also figured while I was at it, I might as well get some useful code out of it.
(more…)

I’ve been particularly productive today, at least as far as externally visible services are concerned. SuperCat is proving to be a mighty beast, and is now the basis of three (count ‘em, three) shiny new web APIs.

UPDATE: Bill and I set up the services described here on our externally accessible dev server. The URLs in the body have been updated to links that point at that server.

(more…)

I mentioned in a previous post that we were planning to create a script to extract API documentation from our new OpenSRF method signature layout. Well, I was inspired by Aaron Krowne, and found guidance in Thom Hickey, to create an XSLT wrapper to the existing OpenSRF introspection API using the OpenILS XML Gateway.

It’s still only available for use on my development server, but it’s in cvs for all to see, or you can click on the unreadably tiny image at the top left of this post for a screenshot. It’s a (some would say scary) mix of XSLT and Apache SSI, but it’s fast and does what we need. It is currently based, as I mentioned, on the OpenILS XML Gateway, so it’s not really part of OpenSRF proper. I plan to remedy that shortly by pulling the OpenILS specific parts of the XML Gateway out into their own library, and creating a generic OpenSRF XML Gateway.

If anyone out there with free time, a love of XSLT and a desire to help out with OpenSRF or OpenILS/Evergreen would like to poke at it, be my guest. Be forewarned, though; if you are experienced with XSLT (which I most certainly am not) it may make your eyes bleed.

UPDATE: For those interested, I asked Bill to push the docgen XSLT out to our dev site, and, well, he did. You can see it in action here.

I began work on open-ils.supercat this weekend, and things are going swimmingly so far. I created the open-ils.supercat.record.marcxml.retrieve and open-ils.supercat.metarecord.mods.retrieve methods with no problem, and added open-ils.supercat.record.mods.retrieve as well. The metarecord building code is mostly a big pile of DOM, but you can see the code here in the sub called retrieve_metarecord_mods if you are so inclined. The record methods, both marcxml and MODS, are much leaner, so I’ll show those inline:

sub retrieve_record_marcxml {
	my $self = shift;
	my $client = shift;
	my $rid = shift;

	return
	entityize(
		$_storage
			->request( 'open-ils.storage.direct.biblio.record_entry.retrieve' => $rid )
			->gather(1)
			->marc
	);
}

__PACKAGE__->register_method(
	method    => 'retrieve_record_marcxml',
	api_name  => 'open-ils.supercat.record.marcxml.retrieve',
	api_level => 1,
	argc      => 1,
	signature =>
		{ desc     => < <"		  DESC",
Returns the MARCXML representation of the requested bibliographic record
		  DESC
		  params   =>
		  	[
				{ name => 'bibId',
				  desc => 'An OpenILS biblio::record_entry id',
				  type => 'number' },
			],
		  'return' =>
		  	{ desc => 'The bib record in MARCXML',
			  type => 'string' }
		}
);

sub retrieve_record_mods {
	my $self = shift;
	my $client = shift;
	my $rid = shift;

	my $marc = $_storage->request(
		'open-ils.storage.direct.biblio.record_entry.retrieve',
		$rid
	)->gather(1)->marc;

	return entityize($_mods_sheet->transform( $_parser->parse_string( $marc ) )->toString);
}

__PACKAGE__->register_method(
	method    => 'retrieve_record_mods',
	api_name  => 'open-ils.supercat.record.mods.retrieve',
	api_level => 1,
	argc      => 1,
	signature =>
		{ desc     => < <"		  DESC",
Returns the MODS representation of the requested bibliographic record
		  DESC
		  params   =>
		  	[
				{ name => 'bibId',
				  desc => 'An OpenILS biblio::record_entry id',
				  type => 'number' },
			],
		  'return' =>
		  	{ desc => 'The bib record in MODS',
			  type => 'string' }
		}
);

As you can see, in both cases the method registration call containing the method signature is larger than the actual sub even with a great deal of whitespace in the code. I will take that as a good sign, since that signature allows scripted API documentation, basic argument checking (that I don’t have to handle inside the method) and quick-glance documentation inline in the code.

While we don’t have the many millions of records that OCLC does (what, 60M+?), we do have a local database to play with, an extensible ILS, and the desire to do fun stuff like OCLC does. So, I wrote an xISBN clone while working on the open-ils.supercat OpenSRF application. Instead of grouping records into FRBR work-sets, we group records into OpenILS/Evergreen metarecords. That is the algorithm that works best for PINES, but the metarecord fingerprinting algorithm will soon be scriptable via server-side JavaScript (complete with DOM and XPath for processing MARCXML records), and we plan on creating a FRBR-ish version at some point based on that work.

One big advantage of our version is that it works using the live data we already have indexed, so there’s no need to create a specialized database that can get stale over time. In fact, every extension we write will work that way if at all possible.

I think it’s a pretty good example of an OpenSRF extension to OpenILS that a developer with a mild familiarity with the open-ils.storage server could create. I’ll be posting the code for the mod_perl handler that creates the XML version soon — all 10 lines that I expect it will take. Anyway, here’s the code:

sub oISBN {
	my $self = shift;
	my $client = shift;
	my $isbn = shift;

	throw OpenSRF::EX::InvalidArg ('I need an ISBN please')
		unless (length($isbn) >= 10);

	# Create a storage session, since we'll be making multiple requests.
	$_storage->connect;

	# Find the record that has that ISBN.
	my $bibrec = $_storage->request(
		'open-ils.storage.direct.metabib.full_rec.search_where.atomic',
		{ tag => '020', subfield => 'a', value => { like => $isbn.'%'} }
	)->gather(1);

	# Go away if we don't have one.
	return {} unless (@$bibrec);

	# Find the metarecord for that bib record.
	my $mr = $_storage->request(
		'open-ils.storage.direct.metabib.metarecord_source_map.search.source.atomic',
		$bibrec->[0]->record
	)->gather(1);

	# Find the other records for that metarecord.
	my $records = $_storage->request(
		'open-ils.storage.direct.metabib.metarecord_source_map.search.metarecord.atomic',
		$mr->[0]->metarecord
	)->gather(1);

	# Just to be safe.  There's currently no unique constraint on sources...
	my %unique_recs = map { ($_->source, 1) } @$records;
	my @rec_list = sort keys %unique_recs;

	# And now fetch the ISBNs for those records.
	my $recs = $_storage->request(
		'open-ils.storage.direct.metabib.full_rec.search_where.atomic',
		{ tag => '020', subfield => 'a', record => \@rec_list }
	)->gather(1);

	# We're done with the storage server session.
	$_storage->disconnect;

	# Return the oISBN data structure.  This will be XMLized at a higher layer.
	return
		{ metarecord => $mr->[0]->metarecord,
		  record_list => { map { ($_->record, $_->value) } @$recs } };
}
__PACKAGE__->register_method(
	method    => 'oISBN',
	api_name  => 'open-ils.supercat.oisbn',
	api_level => 1,
	argc      => 1,
	signature =>
		{ desc     => < <"		  DESC",
Returns the ISBN list for the metarecord of the requested isbn
		  DESC
		  params   =>
		  	[
				{ name => 'isbn',
				  desc => 'An ISBN.  Duh.',
				  type => 'string' },
			],
		  'return' =>
		  	{ desc => 'record to isbn map',
			  type => 'object' }
		}
);

In an effort to help others (and ourselves) write cleaner, happier, more eco-friendly code, we have added optional argument count and signature checking to OpenSRF. To make using this facility as simple and unobtrusive as possible there are three implementation levels (well, four if you count “a short note without parameter information”), each one more descriptive than the previous, so developers can choose how much (if any) time they will invest in documenting method signature and use and whether runtime argument checks are appropriate for that method. The other part is the argc setting that can be passed along with the signature. This specifies how many arguments are required, and the code will check their type and class against the actual parameters supplied if the signature gives that much information. Eventually, signature level 3 will allow individual parameters to be marked as required, and it will allow the handling of named (instead of positional) parameter description and validation. I’ll go ahead and lay out the structure of each method signature level here, and wait patiently for my thrashing to ensue.
(more…)

• Alternative OPACs
• Personal Card Catalogs
• Bookbags
• Record Tagging
• Reading Lists
• User built RSS Feeds of reading lists and searches
• Interactive searches
• Saved searches
• Search history
• … tons of other things we’ve collectively imagined …

What do they conceptually have in common? (more…)

This will be a quick one because I’m feeling a bit under the weather, but I wanted to post a quick update on some of our tornado-like progress post-alpha.

(more…)

I’m glad to see we’ve stirred the pot with our discussion of JSON. It’s driven us to stop and think about our decisions, their impact on our project, and how others may be able to interact with our system.

For the sake of those interested, I thought I might offer some clarity regarding our use of JSON. When OpenSRF (our core information passing framework, discussed here) was first designed, it was XML everywhere. As we were doing our benchmarking of the system, we thought things were good, but sluggish with large datasets. OpenSRF is designed as a set of independent applications which often must communicate with one or more other applications to accomplish any particular task. For example, to retrieve information on a patron, the system must verify the requesting party has such permission, then it must actually retrieve the information. With a lot of such conversation going on across the backbone, you can see there will be a great deal of data flying around the network.

(more…)

After months of work, buckets of sweat, and one snapshot of code seemingly unrelated to libraries, we are proud to present our first subproject: OpenSRF (Open Scalable Request Framework), pronounced “open surf”. The 0.1 release of OpenSRF represents the culmination of everything we have posted about here, and some things we haven’t.

One of the main differences between our initial snapshot tarball and the shiny new OpenSRF code is the removal of the dependence on an Auth Application. We have decided that the security provided by Jabber is sufficient for connection to the OpenSRF network, and application level authentication and authorization should be handled either by each individual Application, or by an Authentication Application built specifically for the target problem domain.

(more…)