All data in the Torus can be fully maintained using a Web Service endpoint. The Web Service has been designed to quite closely follow RESTful principles in that it operates on resources (realms, parent, records) exposed via URLs using the four basic HTTP methods (GET, POST, PUT, DELETE) to model CRUD operations.
At the moment, the Torus Web Service is not great at following HATEOS (Hypermedia as the Engine of Application State) principle in REST design, but that might very well change in the future versions.
For short the Torus Web Service Protocol is referred to as WSAPI (Web Service Application Programming Interface).
So without further ado let's get started.
The actual records stored within the Torus are created by combining layers of key-value pairs that come from different locations.
Once the records are loaded from the remote parents, they form the so called original layers. Original layer resides in the world record and cannot be modified in any way. Instead a new layer, called override, can be posted using a local record that refers to the said original. Torus will store such layer as-is but the next time the given record is requested by a client it will contain a new, automatically constructed layer called final. Final layer is a result of merging the two mentioned layers: original and override (which is placed on top). In case one of the layers is not present, final assumes the existing one.
Each torus record is a collection of name/value pairs called fields. Beside a strict set of magic (system) fields, fields can take any names and values chosen by the client. The (current) magic fields are as follows:
Client software should avoid clashes between user and magic fields.
User fields can have arbitrary values but, additionally, fields specially formatted to represent IP addresses/ranges and dates (RFC/ISO) will support extended searching (within/encloses) and sorting (date) capabilities.
Torus is well abstracted from any specific record representation and cares only about records that resemble key-value pair bags. Nevertheless, the currently used on-the-wire format for encoding requests and responses is a strictly defined XML format. It's entirely plausible to extend the range of supported representations to e.g. JSON, if ever needed. For now just make sure to include Content-Type: application/xml header in the WSAPI requests (especially if you're using curl to interface with it).
No XML schemas are provided at the moment but the actual XML representations are well described in relation to the appropriate operations below.
Torus uses standard HTTP error codes across all resource interactions. The meaning of the error codes follows closely their generic HTTP meaning.
200 OK, operation went fine (GET, PUT, DELETE)
201 Created, operation went fine and a new entity was created in return, the Location header should be inspected for the URL of the new entity (POST)
301 Moved Permanently, by default all resource URLs end with a trailing slash and this style of requests should be favored by a client. In cases when no trailing slash is supplied but a resource exists, a redirect will be issued by the Torus.
304 Not Modfied, for conditional GET requests that use If-Modified-Since header in case no record(s) were modified
400 Bad request, for all invalid requests that cannot be understood, like bad queries, illegal identifiers, malformed entity data in the body etc. It usually means that the client request should be reformulated before it is going to be repeated.
404 Not found, for requested things that cannot be found (e.g deleted)
500 Internal error, when things go wrong in the Torus server, may indicate misconfiguration or simply, a bug
Resources (in the RESTful sense of the term) are objects exposed via URLs that can be manipulated with HTTP methods.
What follows is a list of resources (and applicable HTTP methods with parameters) that can be used to interface with the Torus2. The sample interactions have been included, in the form:
REQUEST BODY (if any) REQUEST QUERY STRING (simplified) RESPONSE CODE RESPONSE BODY (if any)
The Realms is the top level Torus resource that allows listing all realms that reside within a given Torus virtual host:
http://host/torus2/
METHODS
GET http://host/torus2/ 200 OK <realms count="6" start="0" total="6"> <realm name="admin.admin" type="identity"/> <realm name="admin.wizard" type="identity"/> <realm name="bench-realm" type="searchable"/> <realm name="bench-realm-child" type="searchable"/> <realm name="global.admin" type="searchable"/> <realm name="identity.USERS" type="identity"/> </realms>
http://host/torus2/somerealm/
The Realm is the resource that allows reading, creating and removing particular realms.
METHODS
The realm type is, at the moment, an opaque string and is used by the client to match records to specific profiles.
The autoInherit (true when set to 'yes') flag enables automatic inheritance of all world records, that is all world records are selected without posting overrides. Any manual overrides still apply. This can be further configured on the per-parent basis, if more granularity is needed.
The de-duplication matchKey can be freely configured by the user to refer to any fields that may appear in the records (the behaviour for missing fields is controlled with the required flag). In case duplicates are detected in parents' records a parent priority is consulted before choosing a representative record that is then placed in the world.
GET http://host/torus2/test-realm-A/ 200 OK <realm name="test-realm-A" type="searchable"> <matchKey> <field name="displayName" required="yes"/> </matchKey> </realm>
Above example shows a realm of type searchable with a matchKey configured to de-duplicate records with the same displayName field.
<realm type="searchable"> <matchKey> <field name="displayName" required="yes"/> <field name="serviceProvider" required="no"/> </matchKey> </realm> PUT http://host/torus2/test-realm-A/ 200 OK
<realm type="searchable" autoInherit="yes"> <matchKey> <field name="displayName" required="yes"/> <field name="serviceProvider" required="no"/> </matchKey> </realm> POST http://host/torus2/test-realm-A/ 204 No Content
DELETE http://host/torus2/test-realm-A/ 200 OK
All three resources: world, records and merged have similar syntax and representations as they all contain list of Torus records, albeit in different configurations.
http://host/torus2/somerealm/world/ http://host/torus2/somerealm/records/ http://host/torus2/somerealm/merged/
PARAMETERS - list of parameters common to GET on the three list resources:
Default index. Search with an empty index will result in searching all of the available fields. This is equivalent to using the following CQL special indexes: cql.allIndexes, cql.anyIndexes, cql.anywhere, cql.keywords and cql.serverChoice.
Default relation. When none or '=' relation is specified the Torus assumes adjacency relation, no matter the term used. When adjacency relation match is performed the Torus will split the search term into words. To match, all of the words in the search term must appear in the record, and must be adjacent to each other in the order of the search term. The adj relation maps here.
Other relations. Additionally, the Torus supports the following relations:
E.g GET ../?query=ip+within%2Fnet.ipaddress+%2210.0.1.1+10.0.1.255%22
GET ../?query=lastModified+within%2FrfcDate+%22Sat%2C+01+Jan+2012+21%3A49%3A06+CET..Thu%2C+01+Jan+2015+21%3A49%3A06+CET%22&start=0&count=20
E.g GET ../?query=ip+encloses%2Fnet.ipaddress+%22192.168.1.117%22
Term wildcards. The Torus supports all three CQL wildcards:
Backslash (\) can be used to escape any special character.
For more information on CQL 1.2 indices and relations see http://www.loc.gov/standards/sru/resources/cql-context-set-v1-2.html. Not all features are, however, supported by the Torus.
Note on date searching: since v2.14 it's possible to express dates (and date ranges) using partial ISO forms (e.g just a year, year and month, etc) and, for convenience, the relation within/isoDate got a shorthand notation of @ (or literal at). Here'sa few rules to use the date range syntax succesfully:
1. Relations @ or within always operates on ranges, but allows unbounded following ( 2014.., the following dots are optional) and unbounded preceding (..2014) ranges.
2. Partial ISO date notation is always expanded to the full form before matching takes place, so 2014-01-01 becomes 2014-01-01 00:00:00, no matter if it's the range's lower or upper bound.
3. Range expression is inclusive on both ends.
4. Following from 2.) a range expression like originDate @ 2014-01-01..2014-01-01 is expanded to originDate 2014-01-01 00:00:00..2014-01-01 00:00:00 so it does not match anything outside of that (impossibly narrow) range.
5. Range expression originDate @ 2014-01-01..2014-01-02 is expanded to originDate @ 2014-01-01 00:00:00..2014-01-02 00:00:00, so it potentially matches records created on 2014-01-02 00:00:00, practically it won't since internal dates are granular down to milliseconds and the expansion really is 2014-01-02 00:00:00,000.
6. The client doesn't need to perform date calculations, e.g to match records created/update within a single day, since it can use time span expressions in the range, like so: originDate @ 2014-05-11..+1d.
Here are some date range expression examples:
lastModified @ 2012 matches records modified after and including exactly 2012-01-01 00:00:00
lastModified @ 2010..2012 matches records modified between 2010-01-01 00:00:00 and 2012-01-01 00:00:00, including those exact time points
lastModified @ 2012-05-06..2012-07-06T13:25 matches records modified between 2012-05-06 00:00:00 and 2012-07-06 13:25:00, including those exact time points
ISO dates also support additional separators (or none at all):
lastModified @ ..2012/05/06 matches records modified before 2012-05-06 00:00:00, including that exact time point
lastModified @ "20120607 13:24..20120604 15:48" matches records modified between 2012-06-07 13:24:00 and 2012-06-04 15:48:00, including those exact time points
It's also possible to express time spans (and mix them with specific dates):
lastModified @ -7d modified in the last 7 days
billingDate @ 2012..+1y2M15d billable 1 year 2 months and 15 days from 2012-01-01 00:00:00
billingDate @ +0..+7 billable in the next 7 days
Sorting The Torus honors sorting requests as part of the CQL query (see http://zing.z3950.org/cql/sorting.html#3). The Torus implements a limited subset of the sort modifiers that can be expressed in CQL. Currently supported are sort.ascending, sort.descending, sort.locale and date. The Torus thus supports alphabetical sorting, but not numerical sorting, and will implicitly do sort.missingLow (sort the missing values first when sorting ascending). As for sort.ignoreCase, sort.respectCase, sort.ignoreAccents, and sort.respectAccents the actual settings will currently depend on the defaults for the locale. When using date modifier, only fields with dates expressed in the RFC format can be understood. For fields with ISO dates use the standard alphabetical sorting.
E.g GET '/torus2/test-realm-A/world/?query=displayName=Dupe%20sortBy%20displayName'
GET '/torus2/test-realm-A/world/?query=%22%22%20sortBy%20lastModified/date'
E.g GET '/torus2/test-realm-A/world/?facets=displayName,categories:;' would yield
<records count="3" start="0" total="3"> <facets> <facet name="displayName"> <term content="Library of Congress" count="3"/> <term content="Library of Texas" count="1"/> </facet> <facet name="categories"> <term content="Libraries" count="2"/> <term content="Catalogs" count="2"/> </facet> </facets> ... </records>
The major difference between world resource and the other record-type list resources is the fact that the world is immutable (so no PUT/POST is allowed) as it is automatically created to form the combined view of all parent records in a given realm.
Each world record is assigned an ID, unique and valid only within a particular Torus realm. This ID can be then used to create override records that point to this given world record. More on this in the record resource paragraph.
METHODS
PARAMETERS
GET http://host/torus2/test-realm-A/world/ 200 OK <records count="2" start="0" total="2"> <record type="searchable"> <layer xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="original" xsi:type="searchableTypeLayer"> <id>P-1.local-0</id> <realm>world</realm> <displayName>Record from parent 1</displayName> </layer> </record> <record type="searchable"> <layer xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="original" xsi:type="searchableTypeLayer"> <id>P-2.local-1</id> <realm>world</realm> <displayName>Record from parent 2</displayName> </layer> </record> </records>
http://host/torus2/somerealm/world/{itemId}/
Allows accessing a world record directly.
METHODS
GET http://host/torus2/test-realm-A/world/P-1.local-0/ 200 OK <record type="searchable"> <layer xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="original" xsi:type="searchableTypeLayer"> <id>P-1.local-0</id> <realm>world</realm> <displayName>Record from parent 1</displayName> </layer> </record>
The Records resource allows for listing all of the realm records created locally: both the pure local ones (meaning not referencing or overriding any world records), ones that only "select" world records for use (empty overrides with only worldId field filled and pointing at an existing world record) and records that fully or partially override data in the original world records (field-specific overrides).
METHODS
GET http://host/torus2/test-realm-A/records/ 200 OK <records count="3" start="0" total="3"> <record type="searchable"> <layer xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="final" xsi:type="searchableTypeLayer"> <id>P-1.local-0-0</id> <realm>test-realm-A</realm> <worldId>P-1.local-0</worldId> <displayName>Record from parent 1</displayName> </layer> </record> <record type="searchable"> <layer xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="final" xsi:type="searchableTypeLayer"> <id>P-2.local-1-0</id> <realm>test-realm-A</realm> <worldId>P-2.local-1</worldId> <displayName>Record from parent 2</displayName> </layer> </record> <record type="searchable"> <layer xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="final" xsi:type="searchableTypeLayer"> <id>local-0</id> <realm>test-realm-A</realm> <displayName>Purely local record</displayName> </layer> </record> </records>
POST http://host/torus2/test-realm-A/records/ <record type="searchable"> <layer xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="override" xsi:type="searchableTypeLayer"> <worldId>P-1.local-0</worldId> <displayName>Record overriding displayName in the parent record</displayName> </layer> </record> 201 Created Location http://host/torus2/test-realm-A/records/P-1.local-0-0/
The Record item resource allows to directly manipulate the existing local record: that is reading it (GET), updating (PUT) and removing (DELETE).
METHODS
PARAMETERS
GET http://host/torus2/test-realm-A/records/local-1/ 200 OK <record type="searchable"> <layer xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="original" xsi:type="searchableTypeLayer"> <id>P-1.local-0</id> <displayName>Record from parent 1</displayName> </layer> <layer xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="override" xsi:type="searchableTypeLayer"> <id>P-1.local-0-0</id> <worldId>P-1.local-0</worldId> <displayName>Record overriding displayName in the parent record</displayName> </layer> <layer xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="final" xsi:type="searchableTypeLayer"> <id>P-1.local-0-0</id> <worldId>P-1.local-0</worldId> <displayName>Record overriding displayName in the parent record</displayName> </layer> </record>
<record type="searchable"> <layer xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="override" xsi:type="searchableTypeLayer"> <id>P-1.local-0-0</id> <worldId>P-1.local-0</worldId> <displayName>Updated record overriding displayName in the parent record</displayName> </layer> </record> PUT http://host/torus2/test-realm-A/records/local-1/ 200 OK
DELETE http://host/torus2/test-realm-A/records/local-1/ 200 OK
The merged resource is meant to give a combined view of all the records associated with a particular realm; both records created or referenced locally (as in the records resource) and unused world records that reside in the world. All common parameters apply. The merged resource accepts ONLY GET method and does not contain any sub-resources (as all records can be addressed directly through world and records resources). Querying is always done on the final layer.
The Masters resource allows for listing all of the master-override records, a special kind of records that get overlaid over a set of records matching a specified query. Master-overrides can be treated as "filters" that allow to override (or provide) a set of fields to a large (and possibly dynamically changing) set of records. Master-overrides should be used only in a situation where mass override functionality is needed and where standard record overriding method may lead to maintenance issues.
Each master-override record may contain a 'matchQuery' attribute -- a standard CQL query is used to limit the set of overridden records to a matching subset.
There is a couple of rules to be remembered when using master-overrides records:
As noted master-overrides should be used only in special circumstances and theiry number should be kept low as they have significant performance footprint. No searching is available on the 'masters' resource.
METHODS
GET http://host/torus2/test-realm-A/masters/ 200 OK <records count="1" start="0" total="1"> <record type="master-override"> <layer name="master-override" matchQuery="recordEncoding = "marc8*""> <fieldMap>some global realm fieldmap</fieldMap> </layer> </record> </records>
POST http://host/torus2/test-realm-A/masters/ <record type="master-override"> <layer name="master-override" matchQuery="queryEncoding = UTF-8"> <comment>This target must be queried using UTF-8</comment> </layer> </record> 201 Created Location http://host/torus2/test-realm-A/masters/master-0/
The Master-override item resource allows to directly manipulate the existing local record: that is reading it (GET), updating (PUT) and removing (DELETE). Per usual the sub-resource is available under masters/id/.
METHODS
The parents resource allows listing all parents that are in use for a particular realm. The combined, de-duplicated records of all parents constitute the realm's world.
http://host/torus2/test-realm-A/parents/
METHODS
GET http://host/torus2/test-realm-A/parents/ 200 OK <parents count="2" start="0" total="2"> <parent id="P-0" lastRefreshed="Wed, 13 Jul 2011 17:39:27 CEST" name="test-realm-1" priority="99" refreshAfter="10" url="http://localhost:8181/torus2/test-realm-1/records/"/> <parent id="P-1" lastRefreshed="Wed, 13 Jul 2011 17:39:27 CEST" name="test-realm-2" priority="90" refreshAfter="0" url="http://localhost:8181/torus2/test-realm-2/records/"/> </parents>
It's worth to know that when a new parent is created Torus will attempt to fetch its record and if it fails - e.g. because of the connection problems - the whole realm creation is aborted.
<parent name="test-realm-3" priority="10" refreshAfter="10" url="http://localhost:8181/torus2/test-realm-3/records/"/> POST http://host/torus2/test-realm-A/parents/ 201 Created Location http://host/torus2/test-realm-A/parents/P-2/
The parent resource allows accessing any of the realm's parents directly: reading, updating and deleting it.
http://host/torus2/test-realm-A/parents/P-1/
METHODS
GET http://host/torus2/test-realm-A/parents/P-1/ 200 OK <parent id="P-1" lastRefreshed="Wed, 13 Jul 2011 17:39:27 CEST" name="test-realm-2" priority="90" refreshAfter="0" url="http://localhost:8181/torus2/test-realm-2/records/"/>
<parent priority="90" refreshAfter="10000" url="http://localhost:8181/torus2/test-realm-2/records/"/> PUT http://host/torus2/test-realm-A/parents/P-1/ 200 OK
DELETE http://host/torus2/test-realm-A/parents/P-1/ 200 OK