The Hitch-Hiker’s Guide to the Connector Platform

(Please note that this manual is incomplete. Those portions that have been written, we are making available in the hope that they are helpful. The missing sections will be added as and when we have the opportunity.)

Introduction

• Purpose
• Overview of components
• Community sourcing the connectors

Architecture

• The Montreal Model: connectors, tasks and steps
• Templates (which define the arguments, etc.)

Components

• The Builder
• The Engine (not downloadable, hosted by us)
• The Repository

Connector Builder Tutorials/HOWTOs

• My first connector (The AMNH Digital Library)
• Refining the AMNH connector
• Is JavaScript Necessary?
• Sites without Hit Counts
• Parsing with XPattern
• Normalizing dates and URLs
• Transforming data with regular expressions
• Parsing with regular expressions (when XPattern can’t cut it)
• XPath refinement (making “Next” work on the 2nd page)
• Alt steps
• Page load detection: Waiting and retrying
• Coping with AJAX, and knowing when to give up
• Finding the next page of results when there is no Next Page button
• Debugging Connectors
• Working with XML
• Advanced searches
  • Overview
  • Years and other limits
  • Analyzing a search page
  • Query types: Fullquery and Listquery
• XPattern issues
  • Or-bags
• Content connectors

Connector Platform Reference Manual

• Data Model
  • Advanced
• Key Shortcuts
• Individual steps
  • Any/All query
  • Assert
  • Click
  • Constant
  • Delay
  • Extract value
  • Extract chronology
  • Fullquery
  • Fullquerylimit
  • Join
  • Listquery
  • Listqueryelement
  • Mouse
  • Map
  • Next if
  • Normalize URL
  • Normalize date
  • Go to URL
  • Parse by XPattern
  • Parse by regex
  • Proxy URL
  • Retry
  • Set form value
  • Set preference
  • Simple query
  • Split result
  • Submit Form
  • Transform
• File formats
  • File-format overview
  • Connector file format
  • Connector templates
• Software Architecture
• The Step API, for those who want to write new steps

Into the future

• A general WSAPI to the engine
• Operations other than searching
  • Recovering circulation information
  • Posting blog entries
  • Booking train tickets and other maybe-some-day ideas (The new Z39.50 server will be an application of the generic engine.)