Library Journal "Digital Libraries" Columns 1997-2007, Roy Tennant
Please note: these columns are an archive of my Library Journal column from 1997-2007. They have not been altered in content, so please keep in mind some of this content will be out of date.
Open Source Metasearch
Digital Libraries: Open Source Metasearch By Roy Tennant -- Library Journal, 10/15/2007 Software that searches multiple remote sources and merges the results for the user has been available for several years. Whether called federated searching, cross-database searching, or metasearching, it aims to make it as easy as possible for library users to find what they want from many different sources. Now, joining several commercial offerings in this space is a free, open source solution: LibraryFind. LibraryFind, which is under development by Oregon State University (OSU), Corvallis, is also being used by the University of Houston, TX. As of August 2007, the program has been downloaded nearly 2000 times. Jeremy Frumkin, the Gray Chair for Innovative Library Services at OSU, is a driving force behind it, as is Terry Reese, also at OSU and well known for the very useful MARC utility MARCEdit. What you get One of the most significant things you get with LibraryFind is the code itself--because it's open source, you're free to add to it, change it, or simply examine how it works. In contrast, commercial solutions do not allow you to view or change the software. Along with the LibraryFind software, you get some preconfigured resource connections to access about 50 specific databases (though there are many others). To make it faster and easier for searchers to find items of interest and quickly retrieve them, LibraryFind implements a "two-click workflow"--one click to find, one to access. When users perform a search, an integrated OpenURL resolver gathers all of the links for a screen of search results and puts links on the page that lead the user directly to the item when it's available. This avoids sending the user through a series of pop-up windows to retrieve an item. "Our goal is not to remove the richness and value that we have in the data in our collections," Frumkin says, "but to move the complexity of dealing with that rich data 'under the hood.'" As anyone who has ever used a metasearch application knows, it often takes a long time for the application to respond with the search results. To help alleviate this, LibraryFind caches search results, saving a local copy. This cache persists beyond a single user to enhance the speed for a second query. Another technique to enhance speed is to index locally whatever you can, and LibraryFind offers this capacity. Libraries with repositories of digitized content or faculty papers could use LibraryFind to harvest and index records from any repository that supports the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH). Challenges, progress LibraryFind is not (yet) a shrink-wrapped application--that is, it still takes someone with experience at installing UNIX software to configure and load it, though set up may become easier. Also, the number of preconfigured resource connections remains small, so any adopter of LibraryFind with needs beyond the included connections will have to set up new connections. LibraryFind has not yet made it to release 1.0 (expected in April 2008), so more features are coming. Frumkin would like to add a better way for libraries to share connection information for various databases. But Frumkin's designs go well beyond the standard metasearch tool. In his universe, LibraryFind would simply be one tool among several that support the entire scholarly workflow. It still needs work Although LibraryFind may one day be an excellent choice for a large academic library, the lack of preconfigured resource connections means that any current adopters will have to create and maintain a significant number of them. Also, since LibraryFind remains at a relatively early stage of development, it is probably best for those libraries that are willing to be early adopters, e.g., libraries ready and able to contribute bug fixes and enhancements to the code base. For now, LibraryFind likely will be most easily implemented by small to medium-sized libraries that have the technical expertise to configure and install it as well as create missing resource connections. Over time, if development continues and the community of adopters grows, LibraryFind could become an excellent choice for many libraries. For more on the wired library, see the netConnect supplement mailed with the January, April 15, July, and October 15 issues of LJ __________________________________________________________________ LINK LIST LibraryFind libraryfind.org MARCEdit oregonstate.edu/~reeset/marcedit/html/index.php OAI-PMH openarchives.org __________________________________________________________________