6. Retrieval Facility

YAZ version 2.1.20 or later includes a Retrieval facility tool which allows a SRU/Z39.50 to describe itself and perform record conversions. The idea is the following:

The Retrieval facility is driven by an XML configuration. The configuration is neither Z39.50 ZeeRex or SRU ZeeRex. But it should be easy to generate both of them from the XML configuration. (Unfortunately the two versions of ZeeRex differ substantially in this regard.)

6.1. Retrieval XML format

All elements should be covered by namespace http://indexdata.com/yaz . The root element node must be retrievalinfo.

The retrievalinfo must include one or more retrieval elements. Each retrieval defines specific combination of syntax, name and identifier supported by this retrieval service.

The retrieval element may include any of the following attributes:

syntax (REQUIRED)

Defines the record syntax. Possible values is any of the names defined in YAZ' OID database or a raw OID in (n.n ... n).

name (OPTIONAL)

Defines the name of the retrieval format. This can be any string. For SRU, the value is equivalent to schema (short-hand); for Z39.50 it's equivalent to simple element set name. For YAZ 3.0.24 and later this name may be specified as a glob expression with operators * and ?.

identifier (OPTIONAL)

Defines the URI schema name of the retrieval format. This can be any string. For SRU, the value is equivalent to URI schema. For Z39.50, there is no equivalent.

The retrieval may include one backend element. If a backend element is given, it specifies how the records are retrieved by some backend and how the records are converted from the backend to the "frontend".

The attributes, name and syntax may be specified for the backend element. The semantics of these attributes is equivalent to those for the retrieval. However, these values are passed to the "backend".

The backend element may include one or more conversion instructions (as children elements). The supported conversions are:

marc

The marc element specifies a conversion to - and from ISO2709 encoded MARC and MARCXML/MarcXchange. The following attributes may be specified:

inputformat (REQUIRED)

Format of input. Supported values are marc (for ISO2709), xml (MARCXML/MarcXchange) and json (MARC-in-JSON).

outputformat (REQUIRED)

Format of output. Supported values are line (MARC line format); marcxml (for MARCXML), marc (ISO2709), turbomarc, marcxchange (for MarcXchange), or json (MARC-in-JSON ).

inputcharset (OPTIONAL)

Encoding of input. For XML input formats, this need not be given, but for ISO2709 based input formats, this should be set to the encoding used. For MARC21 records, a common inputcharset value would be marc-8.

Note

If inputformat is marc and inputcharset is marc-8, then effective inputcharset is UTF-8 if leader position has value 'a' (MARC21 rule).

outputcharset (OPTIONAL)

Encoding of output. If outputformat is XML based, it is strongly recommended to use utf-8.

leaderspec (OPTIONAL)

Specifies a modification to the leader for the resulting output record. The leaderspec is a comma separated list of pos=value pairs, where pos is an integer offset (0 - 23) for leader. Value is either a quoted string or an integer (character value in decimal). For example, to set leader at offset 9 to a, use 9='a'. This has same effect as -l for yaz-marcdump(1).

select

The select selects one or more text nodes and decodes them as XML. The following attributes may be specified:

path (REQUIRED)

X-Path expression for selecting text nodes.

This conversion is available in YAZ 5.8.0 and later.

solrmarc

The solrmarc decodes solrmarc records. It assumes that the input is pure solrmarc text (no escaping) and will convert all sequences of the form #XX; to a single character of the hexadecimal value as given by XX. The output, presumably, is a valid ISO2709 buffer.

This conversion is available in YAZ 5.0.21 and later.

xslt

The xslt element specifies a conversion via XSLT. The following attributes may be specified:

stylesheet (REQUIRED)

Stylesheet file.

In addition, the element can be configured as follows:

param (OPTIONAL)

A param tag configures a parameter to be passed to the XSLT stylesheet. Multiple param tags may be defined.

rdf-lookup

The rdf-lookup element looks up BIBFRAME elements in some suitable service, for example http://id.loc.gov/authorities/names and replaces the URIs for specified elements with URIs it finds at that service. Its configuration consists of

debug (OPTIONAL)

Attribute to the rdf-lookup tag to enable debug output. A value of "1" makes the filter to add a XML comment next to each key it tried to look up, showing the URL, the result, and timing. This is useful for debugging the configuration. The default is not to add any comments.

timeout (OPTIONAL)

Attribute of the rdf-lookup tag which defines timeout in seconds for the HTTP based rdf-lookup.

namespace (OPTIONAL)

A namespace tag declares a namespace to be used in the xpath below. The tag requires two attributes: prefix and href.

lookup (REQUIRED)

A section that defines one tag to be looked up, for example an author.The xpath attribute (REQUIRED) specifies the path to the element(s).

key (REQUIRED)

A tag withing the lookup tag specifies the value to be used in the lookup, for example a name or an ID. It is a relative Xpath starting from the tag specified in the lookup.

server (OPTIONAL)

Specifies the URL for server to use for the lookup. A %s is replaced by the key value to be looked up. If not specified, defaults to the same as the previous lookup section, or lacking one, to http://id.loc.gov/authorities/names/label/%s . The method attribute can be used to specify the HTTP method to be used in this lookup. The default is GET, and the useful alternative is HEAD.

See the example below.

This conversion is available in YAZ 5.19.0 and later.

6.2. Retrieval Facility Examples

Example 7.19. MARC21 backend

A typical way to use the retrieval facility is to enable XML for servers that only supports ISO2709 encoded MARC21 records.

     <retrievalinfo>
       <retrieval syntax="usmarc" name="F"/>
       <retrieval syntax="usmarc" name="B"/>
       <retrieval syntax="xml" name="marcxml"
		  identifier="info:srw/schema/1/marcxml-v1.1">
         <backend syntax="usmarc" name="F">
	   <marc inputformat="marc" outputformat="marcxml"
		 inputcharset="marc-8"/>
	 </backend>
       </retrieval>
       <retrieval syntax="xml" name="dc">
         <backend syntax="usmarc" name="F">
	   <marc inputformat="marc" outputformat="marcxml"
		 inputcharset="marc-8"/>
           <xslt stylesheet="MARC21slim2DC.xsl"/>
	 </backend>
       </retrieval>
     </retrievalinfo>

     

This means that our frontend supports:

  • MARC21 F(ull) records.

  • MARC21 B(rief) records.

  • MARCXML records.

  • Dublin core records.


Example 7.20. MARCXML backend

SRW/SRU and Solr backends return records in XML. If they return MARCXML or MarcXchange, the retrieval module can convert those into ISO2709 formats, most commonly USMARC (AKA MARC21). In this example, the backend returns MARCXML for schema="marcxml".

     <retrievalinfo>
       <retrieval syntax="usmarc">
         <backend syntax="xml" name="marcxml">
	   <marc inputformat="xml" outputformat="marc"
		 outputcharset="marc-8"/>
	 </backend>
       </retrieval>
       <retrieval syntax="xml" name="marcxml"
		  identifier="info:srw/schema/1/marcxml-v1.1"/>
       <retrieval syntax="xml" name="dc">
         <backend syntax="xml" name="marcxml">
           <xslt stylesheet="MARC21slim2DC.xsl"/>
	 </backend>
       </retrieval>
     </retrievalinfo>

     

This means that our frontend supports:

  • MARC21 records (any element set name) in MARC-8 encoding.

  • MARCXML records for element-set=marcxml

  • Dublin core records for element-set=dc.


Example 7.21. RDF-lookup backend

This is a minimal example of the backend configuration for the rdf-lookup. It could well be used with some heavy xslt transforms that make BIBFRAME records out of MarxXml.

        <backend syntax="xml" name="rdf-lookup">
          <rdf-lookup debug="1" timeout="10">
            <namespace prefix="bf" href="http://id.loc.gov/ontologies/bibframe/" />
            <namespace prefix="bflc" href="http://id.loc.gov/ontologies/bibframe/lc-extensions/"/>
            <lookup xpath="//bf:contribution/bf:Contribution/bf:agent/bf:Agent">
              <key field="bflc:name00MatchKey"/>
              <key field="bflc:name01MatchKey"/>
              <key field="bflc:name11MatchKey"/>
              <server url="http://id.loc.gov/authorities/names/label/%s" method="HEAD"/>
            </lookup>
          </rdf-lookup>
        </backend>

     

The debug=1 attribute tells the filter to add XML comments to the key nodes that indicate what lookup it tried to do, how it went, and how long it took.

The namespace prefix bf: is defined in the namespace tags. These namespaces are used in the xpath expressions in the lookup sections.

The lookup tag specifies one tag to be looked up. The xpath attribute defines which node to modify. It may make use of the namespace definitions above.

The server tag gives the URL to be used for the lookup. A %s in the string will get replaced by the key value. If there is no server tag, the one from the preceding lookup section is used, and if there is no previous section, the id.loc.gov address is used as a default. The default is to make a GET request, this example uses HEAD


6.3. API

It should be easy to use the retrieval systems from applications. Refer to the headers yaz/retrieval.h and yaz/record_conv.h.