We now want to refine the connector that we made for the AMNH digital library. There are two improvements that we might wish to make: supporting advanced search (i.e. searching specifically for an author or title); and cleaning the parsed data more fully. We will consider each of these in turn.
We noticed earlier that the AMNH site has an Advanced Search page. We can use this to make field-specific searching available to clients – title, author and subject. (The site also supports searching for series, but the Builder does not have a search parameter for that, so we can’t make use of it.)
To take advantage of the Advanced Search page, we’ll make a second
search task – one that uses the title, author and subject arguments
rather than the keyword argument. Once both
search tasks are in
place, we’ll be able to tell which is which, when we pull down the
Current Task dropdown, because one will be called
and the other will be called
search (title, author, subject) – the
Builder knows which parameters each task uses and, names the task accordingly.
To create the new task, click on the Add Task button – the large plus
sign in the toolar at the top of the Builder sidebar. From the Add
Task browser, double-click on search (or click on it once, then
hit the Add button at the bottom of the Task browser). We need to
add some sample parameters for testing, so in the Test Arguments
area of the new task, let’s set title to
relationships, author to
colbert and subject to
dinosaurs. Now we’re ready to start adding steps.
As before, we’ll start by having the Builder navigate to the search page. Add an Open URL step to the new task, and set the Constant
The next part of the process is to populate the search form with the title, author and subject parts of the query. This can be awkward to do in search-pages like this one, where the choice of which fields to search is not statically determined but must be made by selecting from dropdowns on the search page. But in this case it’s not so bad because there are three search fields available: we can set the dropdowns to constant values and use them to implement the three query parameters. If there were fewer search fields available, we would need to use conditional logic to determine at run time what selections to make from the dropdowns that control them.
Let’s start by fixing the first dropdown to specify a title search.
Add a Set form value step, hit the Select node button in the
step configuration pane, and then click on the first of the three
field-name dropdowns in the search page. When you do this, the Form
field to populate field in the step configuration pane is filled in,
and the dropdown itself opens up to offer the list of options. From
this list, click on
Title, and the step configuration pane’s
Populate with constant field is also filled in. Now test that the
step works by manually changing that dropdown to one of its other
values, then hitting the Play button to revert it to
Next we need to set the value of the
title argument into the form.
Add another Set form value step, hit the Select node button in
the step configuration pane, and then click on the first of the three
entry boxes on the AMNH search page, the one next to the dropdown that
we’ve set to
Title. Now, in the step configuration pane, go to the
Populate with task argument dropdown and choose
title. Click on
the Play button to verify that this step does indeed set the
What you have done for the title parameter in the Advanced Search
page’s first search field, you can now do for the author and subject
parameters in the other two fields. Go ahead: add two more pairs of
Set form value steps, and test them. (Somewhat confusingly, the
value that the AMNH Digital Library’s Advance Search page uses for
field-selection dropdown when it’s set to “Subject” is
Don’t worry, you didn’t make a mistake.)
Finally, we can submit the search form. This is done exactly the same
as for the simple search: add a Click action, Select node and
hit the search form’s
Test the new task using the Play button, and a single record should be
found. This can be parsed using the existing
Tidying up the titles
The author strings that we extracted with the XPattern parser are rather ugly, and we can usefully do more work on them.
To get set up for this work, return to the original
task, and hit the Play All button to re-do the search that finds 85
hits. Return to the
parse task, and re-run it to get a set of
results into the Builder.
Now we can get to work.
Splitting and cleaning author names
The first thing to do is split the author strings apart into separate authors: this is useful because when multiple values are returned separately rather than as part of a glued-together string, they can be used in facet lists.
Add a Split result step, and configure it as follows:
In the Result list dropdown, select
Set Result to transform to
Set the Regular expression to
;\s*(semicolon, backslash, lowercase letter ‘s’, asterisk). This matches a literal semicolon followed by any number of spaces, tabs, etc.
Set Result to set to
author, so that the split values will replace the existing author string. (Alternatively, they could be copied into a separate field.)
Running this step does not change the single authors at all, but splits compound author strings like “Brown, Barnum.; Schlaikjer, Erich Maren, 1905-” into multiple authors.
Removing trailing periods
We see that some author names have terminating periods, while others
do not – for example, in the field that was split in the previous
step, “Brown, Barnum.” ends with a period, while “Schlaikjer, Erich
Maren, 1905-” does not. We can tidy this up with a Transform
result step: add the step, set the Result list, Result to
transform and Result to set as with the previous step
author), and set the Regular expression to
\.$ (backslash, period, dollar). This matches a period at the end
of a value only. Leave the Replace with field blank, since we
want to replace the trailing period with nothing. Press the Play
button to check that this step does indeed remove the period from the
end of “Brown, Barnum.”.
Removing birth and death dates
We now see that some of the author names have birth dates, or both birth-and-death dates, after them, whereas others do not. For example, the fourth record has three authors, Osborn, Henry Fairfield, 1857-1935, Brown, Barnum, and Lull, Richard Swann, 1867-. To remove these dates, we use a regular expression that matches a comma, followed by zero or more whitespace characters, then any number of digits and minus signs at the end of the string.
Add another Transform result step, as before with Result list
results, Result to transform set to
author and Result
to set set to
author. Set the Regular expression to
,\s*[0-9-]+$ and leave the Replace with string empty.
Press the Play button, and watch all the dates disappear from the ends of the author names.
Reversing the “Last name, First name” format
Now we have author names like Osborn, Henry Fairfield, Brown, Barnum, and Lull, Richard Swann. These are all in a consistent format within the AMNH Digital Library database, but normalised names in this last name, first name format are generall unusual, and so client software that uses these names to generate facet lists will not recognise that Osborn, Henry Fairfield in this database is the same author as Henry Fairfield Osborn in another. So in order to make our connector a better citizen in the metasearching world, we’ll finish up by switching the names into the more common form.
For this, we will use yet another Transform result step on the
author field (so set up Result list, Result to transform and
Result to set as before). This time, we need a more sophisticated
regular expression that captures both the surname and the forenames
separately. Set Regular expression to
matches and captures any sequence of characters, followed by a comma
and zero or more spaces, followed by another sequence of any
characters, which is also captured. Set Replace with to
which simply emits the two captured substrings in reverse order.
Hit the Play button: all the author names are converted into conventional form.
Putting it together
Now that the
parse task is complete, you can test it as a whole.
next task to step on to hits 11-20, then go back to the
parse task, and hit the Play All button. All of the URLs and
authors should appear correctly. in the Results area.