Please note: This document is kept here for historical purposes, but it has not been kept up to date. Also, you may also like to see another document frozen in time, "Specifications for Metadata Processing Tools" (PDF)

Bitter Harvest: Problems & Suggested Solutions for
OAI-PMH Data & Service Providers

Roy Tennant, California Digital Library * roy.tennant@ucop.edu
Last text revision 14 May 2004

Background

 

The Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH, http://www.openarchives.org/) specifies a method for digital repositories (also called "data providers") to expose metadata about their objects for harvesting by aggregators (also called "service providers"). Metadata is exposed via "sets," or collections of metadata that data providers decide to make available for harvesting. Service providers harvest sets from data providers of interest, and provide search services for the resulting collections of metadata (for a good example of a service provider, see http://www.oaister.oclc.org/). Data providers also decide which metadata formats to expose for harvesting, beyond the one required data format of simple Dublin Core (see http://dublincore.org/).

 

The OAI-PMH is relatively new, and both data and service providers are still learning the best methods for exposing metadata for harvesting and gathering that metadata into centralized search services. This paper attempts to outline some of the major harvesting issues that service providers have discovered, and outlines some suggested solutions and next steps for data providers, service providers, and those involved with revising and extending the Open Archives Initiative infrastructure. For additional issues specific to service providers, see "Service Provider Issues" by Kat Hagedorn[1].

 

The OAI-PMH was specifically designed to make participation relatively easy for data providers, so that those with useful content to share can experience as little pain as possible to comply with the protocol. As stated in an official OAI document, "The OAI-PMH is designed to provide a low barrier to implementation for repositories and this means that in places burden has placed on harvesters in order to simplify repository implementation."[2] This is why, for example, simple Dublin Core is the only metadata schema required. But this low barrier does not preclude a much higher ceiling, and the OAI-PMH specifically allows the use of much richer metadata schemes. The fact that few data providers are taking advantage of this flexibility may be at least partly due to the relatively recent development of OAI-compliant systems, the small number of OAI service providers, and the relative inexperience of most of those involved.

 

As service providers gain more experience in OAI harvesting, and run into some of the problems identified below, pressure may build on data providers to expose richer sets of metadata. However, as the OAI directorate intended, service providers will likely continue to shoulder a greater burden and responsibility to perform the types of normalization and transformation routines that are touched on in the final section of this paper.

Harvesting Issues

Sets

There are presently no guidelines for data providers regarding set specification. Since no guidelines exist, data providers create sets that may only make sense for that particular institution, and that make no sense whatsoever to a service provider attempting to provide discovery services for a group of data providers. Some specific set difficulties are:

 

• At least one institution makes sets available only on the basis of object type; that is, image, text, etc. However, a text document that has been digitized as page images but not fully converted into machine-readable text, is considered to be a part of the "image" set -- regardless of the fact that no one outside of that institution would consider a text document to be an "image".
• A number of institutions offer sets that comply with their internal organizational structure, but which have little meaning outside of that institution (e.g., university departments).
• At least one institution surfaces a set comprised of every assigned keyword in the collection; although this provides for a highly granular collection of sets, it also presents problems for using the "list sets" verb and processing the resulting list.

 

Metadata Issues

Simple DC is Too Simple

Most data providers are not exposing the richest metadata format possible -- most expose only simple Dublin Core. This presents a number of problems:

 

• Mapping -- "Dumbing down" metadata from an internal (and typically richer) metadata schema to simple Dublin Core requires mapping elements from the richer format to the simpler. The decisions that data providers make about mapping richer element sets into simple Dublin Core are not always the best ones for service providers -- or even, at times, logically defensible.
• Lack of qualification -- Simple Dublin Core lacks even the most basic metadata element qualification that could potentially offer the service provider important information about a given element. For example, some data providers make available two date fields for a digital object -- the date of the original object and the date the object was digitized. Without information that specifies unambiguously which date field is which, the service provider must perform sophisticated metadata processing techniques to discover this information, and in some cases even that may fail.

 

 

Metadata Artifacts

Artifacts are anachronistic practices that have been inherited from other metadata schemas or uses to which the metadata was put. For example, at least one data provider provides Dublin Core <title> elements with "[electronic resource]" as part of the title. This is a hold-over from MARC, and outside the context of library catalogs (and one could argue even within the context of library catalogs) this information is, at least, misplaced. Metadata from other providers can be found to include HTML markup such as "<br>" for a line break -- clearly a hold-over from some user display requirement.

 

Granularity

Object Level -- Some data providers expose individual metadata records for components of one intellectual object. For example, the individual digitized pages of a diary may be represented by individual metadata records by one data provider, and represented by one "set record" by another. Reconciling these differences present particular problems for service providers.

 

 

Element Level -- Unqualified Dublin Core provides opportunities for all sorts of disasters. One such disaster is the loss of granularity. A specific example is "dumbing down" the various constituent components of a personal name into one unstructured field. Once granularity is lost, it can be extremely difficult to recover it unless the records are consistent and can be parsed by a reasonably competent algorithm.

 

 

Encoding Variances

The Dublin Core "date" element is a prime example of the challenges service providers face with variant encoding practices. Even among only five data providers, the following date entries have been discovered:

• 1991-10-01
• ca. 1920.
• (ca). 1920)
• 2001.06.08 by CAD
• Unknown
• ca. June 19, 1901.
• (ca). June 19, 1901)
• [2001 or 2002.]
• 1853.
• c1875.
• c1908 November 19
• c1905
• 1929 June 6
• [between 1904 and 1908]
• [ca. 1967]
• 1918 ?
• [1919 ?]
• 191-?
• 1870 December, c1871
• 1920, 1921, 1922, 1923, 1924, 1925, 1926, 1927, 1928, 1929

 

Such wide variability in the method of encoding dates may render this field unusable for searching, unless service providers can create well-crafted normalization routines. So far, however, even such experienced service providers as the Core Integration team of the National Science Digital Library do only simple transformations of the date field (i.e., from CCYY, CCYY-MM, and CCYY-MM-DD to the W3C Date and Time Format[3]).

 

 

Best Practices for Data Providers

 

• Expose the richest metadata you have -- besides complying with the OAI-PMH standard by exposing simple Dublin Core for harvesting, also make available for harvesting the richest metadata you have available in your internal system.
• Expose as many metadata formats as you can easily manage -- if you can expose metadata using a variety of schemata, do so.
• Normalize your metadata -- be consistent in your encoding.
• Wash your metadata clean of any anachronistic or unique encoding practices.

Best Practices for Service Providers

 

• Do not accept the harvested metadata "as is" -- metadata from a variety of sources is very likely to differ in significant ways. Harvested metadata will need, at minimum, to be normalized. For example, varying methods of encoding dates should usually be normalized into the same encoding for proper searching and display.
• Request that data providers provide the richest, most granular form of metadata possible, in addition to the required Dublin Core format.
• Work with other service providers to create a set of shareable algorithms, techniques, and strategies for metadata.

 

A Suggested Harvesting Model for Service Providers

 

Service providers will be required in almost all cases to at least normalize the harvested metadata, if not perform more sophisticated functions such as adding fields (e.g., the source of the data), increasing the metadata granularity (e.g., splitting out personal names into the constituent parts), or qualifying fields (e.g., specifying that a subject term comes for a specific vocabulary).

 

Some of these transformations can be accomplished without conversation with the data provider (e.g., date normalization), but in a number of cases service providers will find it fruitful to communicate with the data provider. If a richer set of metadata cannot be exposed by the data provider, at least the service provider may be able to discover information that would be useful in a metadata transformation process. Therefore, an iterative process to setting up a repository for harvesting is proposed.

 

The process begins with an initial prototype harvest, which leads to analysis of the metadata and specification of a harvesting profile and required metadata transformations (see Figure 1). The Core Integration Team of the National Science Digital Library has found software applications such as Microsoft Excel and SpotFire DecisionSite to be useful in visualizing the harvested metadata for spotting anomalies and characteristics[4].

 

 

Figure 1. Metadata analysis, creation of harvest profile and metadata transformation process

 

The second stage consists of subsetting activities -- operations required to select the appropriate set of records for a given search service (see Figure 2).

 

 

Figure 2. Creation of subsetting profile and procedures

 

The final stage is the establishment of a production harvesting service, which uses the created profile and the established subsetting and transformation processes to create the appropriate set of harvested and transformed metadata (see Figure 3).

 

Figure 3. The production harvesting process.

 

Properly accomplished, at least some of aspects of the production harvesting process could be deployed to perform metadata transformations for other metadata ingest activities. Libraries will increasingly be called upon to accept, process, and make available metadata in a variety of formats (e.g., ONIX), and the more skilled we become at ingesting and processing metadata the better service we will be equipped to provide our clientele.