I know next to nothing about MARC,though being a shambrarian I have to fight it sometimes. My knowledge is somewhat binary, absolutely nothing for most fields/subfields/tags but ‘fairly ok’ for the bits I’ve had to wrestle with.

[If you don't know that MARC21 is an ageing bibliographic metadata standard, move on. This is not the blog post you're looking for]

Recent encounters with MARC

• Importing MARC files in to our Library System (Talis Capita Alto), mainly for our e-journals (so users can search our catalogue and find a link to a journal if we subscribe to it online). Many of the MARC records were of poor quality and often did not even state the item was (a) a journal (b) online. Additionally Alto will only import if there is a 001 field, even though the first thing it does is move the 001 field to the 035 field and create its own. To handle these I used a very simple script to run through the MARC file – using MARC::Record – to add an 001/006/007 where required.
• Setting up sabre – a web catalogue which searches the records of both the University of Sussex and the University of Brighton – we need to pre-process the MARC records to add extra fields, in particular a field to tell the software (vufind) which organisation the record was from.

Record problems

PC Tools

marcview

 

USEMARCON is the final utility. It comes with a GUI interface, both of which can be downloaded from The National Library of Finland. The British Library also have some information on it. Its main use is to convert MARC files from one type of MARC to another, something I haven’t looked in to, but the GUI provides a way to delve in to a set of MARC records.

Back to the problem…

So we were pre-processing MARC records from two Universities before importing them in to vufind using a Perl script which had been supplied by another University.

It turns out the script was crashing on certain records, all records after the problematic record were not being processed. It wasn’t just that script, any perl script using MARC::Record (and MARC::batch) would crash when it hit a certain point.

By writing a simple script that just printed out each record we could as least see what the record was immediately before the record causing it to crash (i.e. the last in the list of output). This is where the PC applications were useful. Once we know the record before the problematic record, we could find it using the PC viewers and then move to the next record.

The issue was certain characters (here in the 245 and 700 fields). I haven’t got to the bottom of what the exact issue is. There are two kinds of popular encodings: MARC-8 and records in UTF-8, and this can be designated in the Leader (9th character). I think Alto (via it’s marcgrabber tool) exports in MARC-8 but perhaps the characters in the record did not match the specified encoding.

The title (245) on the orignal catalogue looks like this:

One work around was to use a slightly hidden feature of MarcEdit to convert the file to UTF:

I was then able to run the records through the perl script, and import it in to vufind.

But clearly this was not a sustainable solution. Copying files to my PC and running MarcEdit was not something that would be easy to automate.

Back to MARC::Record

The error message produced looked something like this:

utf8 "\xC4" does not map to Unicode at /usr/lib/perl/5.10/Encode.pm line 174

I didn’t find much help via Google, though did find a few mentions of this error related to working with MARC Records.

The issue was that the script loops through each record, the moment it tries to start a loop with a record it does not like it crashes, so there is no way to check for certain characters in the record as it will already be too late.

Unless we use something like exceptions. The closest to this perl has out-of-the-box is eval.

By putting the whole loop in to an eval, if it hits a problem the eval simply passed the flow down to the or do part of the code. But we want to continue processing the records, so this simply calls the eval again, until it reaches the end of the record. You can see a basic working example of this here.

So if you’ve having problems processing a file of MARC records using perl MARC::Record / MARC::batch try wrapping it in a eval. You’ll still loose the records it can not process but it wont stop in it’s tracks (and you can output an error log to record the record number of the records with errors).

Post-script

So, after pulling my hair out, I finally found a way to process a filewhich contains records which cause MARC::Record to crash. It had caused me much stress as I needed to get this working, and quickly, in an automated manner. As I said, the script had been passed to us by another University and it already did quite a few things so I was a little unwilling to rewrite using another language (though a good candidate would be php as the vufind script was written in that language and didn’t seem to have these problems).

But in writing this blog post, I was searching using Google to re-find the various sites and pages I had found when I encountered the problem. And in doing so I had found this: http://keeneworks.posterous.com/marcrecord-and-utf 

Yes. I had actually already resolved the issue, and blogged about it, back in early May. I had somehow – worryingly - completely forgotten any of this. Unbelievable! You can find a copy of a script based on that solution (which is a little similar to the one above) here.

So there you are, a few PC applications and a couple of solutions to perl/MARC issue.

Related posts:

1. ircount : Repository Record Statistics

I’ve created a screencast showing how easy it can be to install VuFind. Here I go from nothing (no server, no OS) to full VuFind install in under 8 minutes.

It would probably take less than two minutes (under 10mins in total) to add MARC records to the installation, but I didn’t have any to hand at the time.

This demo cheats a bit by using a script that does the heavy work, the script was a mash up I created taking existing scripts and commands that come with VuFind with a few tweaks. It probably would have been only slightly slow to run most commands manually.

The script in question is at http://www.nostuff.org/vufind-install.txt and of course anyone is free to download (and improve, please share changes). There’s lot of potential to improve it’s ability to work on different flavours of Linux.

Multi Instance

One of the key aspects of the script is that is allows you to easily install multiple instances of VuFind on to a server. By default VuFind installs in to /usr/local/vufind and has other things (databases, apache conf) names vufind. The script prompts for an ‘instance name’ and then uses that in place of ‘vufind’.

The rational for this is my feeling that VuFind is an excellent tool for creating niche catalogues that are a subset of the full Library collection (or as someone put it a ‘tranche of the catalogue’). A branch Library, particular collection, rare books, a particular School, Core reading (short loan books), specialist resources (AV / laptop items for loan) etc. The idea of a organisation’s records being in system, rather than many (of varying quality) makes sense, but it’s reasonable for those moving their records to a central system to want to be able to search their records independently of other records (and expecting users to go to an Advanced search of using a refine option of the main catalogue is not really not an option). VuFind seems like an obvious answer to this. Especially if new instances can be set up quickly.

In fact it seems to be a failing of most of the main Library Manage Systems (ILS) and their web interfaces that being able to create lots of interfaces (slicing and dicing the underlying single large pool of records). Most require a large amount of effort to create a second web interface to a catalogue. This seems like such an obvious flaw and a barrier to moving to one system to manage resources such as books and other items for an entire organisation.

Amazon EC2

Amazon AWS is a great tool to use for this. A small instance will cost around $0.10 an hour, the micro instance is even cheaper (just over $0.02). Create an instance for ten hours and you have spent around a dollar. Mess it up, just destroy it and create a new one. No risk and no hassle (for first time users the hardest thing is probably the ssh private keys).

Related posts:

1. Amazon AWS EC2 and vufind
2. MARC Tools & MARC::Record errors

Today I submitted a JISC bid on behalf of a team, as part of the recent JISC call Infrastructure for Education and Research (’15/10′ to its friends). The call was actually a set of broadly related different strands, we submitted (with a whole 30 mins to spare) under a strand called Infrastructure for Resource Discovery, and there’s a nice web based version of the call on Jiscpress.

Jiscpress was created in part by Joss Winn and a post of his I saw today inspired me to knock out this this rambling.  Go read it before you read this, thanks.

I admire his openness and I should strive to do the same. Funny that I try – and to an extent automatically do – make much of what I do open, but with this sort of thing there is a tendency to keep it close to your chest. There were very few tweets in the run up to the bid. Why are we not more open? He also talks about his JISC bid writing and tips, here’s mine.

My first experience was attending a ‘town hall meeting’ in Birmingham about the JISC Capital programme, around 2006. For a starts I didn’t even know what a Town hall meeting was (I think it means a briefing day, everyone presumed you should know this). I do remember it felt daunting, there were lots of people in suits. Lots and lots of sentences I didn’t understand (We’re going to innovate to take vertical solutions and silos and break them in to a horizontal landscape to help foster a new wave of disruption to help inject in to the information environment testbed) and no one I knew. I looked at the massive documents that accompanied the programme, many of them, many times. And looked at what I needed to do to write a bid. Budgets, signatures, partners, matched funding. I didn’t submit one.

Since then the community has developed, in no small part thanks to Twitter, but also to things like mashlib and many one day events (which either never used to exist in the field that I work, or I was just more ignorant then than I am now). Beer has been a big part of forging links in the HE Library / tech community. Seriously. It really needs its own cost code.

I looked at a number of potential calls over the last few years – often they required a report or development that I had no really knowledge in, I almost came close to putting something in for the JISC rapid innovation call (and helped mark it). When the JISC LMS call came out about a year a go the time and subject were right to submit a bid. I knew the subject matter, I had a natural interest and passion, and I knew the people who would be involved in these sorts of things.

These are tips for people who are thinking of putting in a bid, especially those who are stupidly disorganised like me:

• Time between a call being released and the submission deadline is short, normally about a month, which in HE terms is not long. Use the JISC funding timeline to get a heads up of future funding opportunities so that you can prepare for working on a bid (including blocking off time during a month, and arranging initial meetings with others) before it comes out and not taken by surprise. The JISC 15/10 call had a blog post a few weeks before the call came out giving a feel for the call and confirming the date it should be released. It helped me to start thinking about ideas and block out time to read it (even if some of that time was in the evening) on the day it was released.
• Every organisation is different (that applies to everything I say) but for us, setting up a meeting a couple of days after the call was out was very useful. It included those who it could affect and relevant senior staff. The call had lots of areas which matched our goals (and some, not always the same, that matched my personal interests), it was good to prepare a briefing and then bounce those ideas around to see what had potential and see what other ideas came up. It helps in many ways, to quickly focus and refine potential ideas (and drop those that people show no interest in), keep everyone in the loop and see whose willing to work on it. It stops it being one person or departments little side project.
• The briefing day was very useful, especially for talking to people, finding potential partners and getting great advice.
• Now I have an incredible amount of bad points, but two of them are leaving everything to the last minute and working in a very linear fashion. Often things that feel like they are the last things you need to do are actually things you need to set in motion earlier on. This seems so simple typing it now but I’ll probably (be warned colleagues) do the same next time. These include budget, project name and supporting letters.
• The budget is hard. See if your org offers support in doing this. The problem is certain magic numbers (the wonders of trac and fec) can only be calculated once you know all your other costs. However I tend to find that near the end of the bid writing process you suddenly think of some work a particular group/person/org will need to do so you need to factor in those hours and costs, or you invite someone from the other end of the country to be on a panel and need to cater for their travel and hotel costs. In best ‘do as I say don’t do as I do’ tradition I would try and bash this out well in advance so it can be sent to those who can then check it over and fill in the magic numbers.
• Asking a friend at another Uni if they don’t mind asking their (P)VC to drop everything so that he/she can write a nice supporting letter for your project is hard. So try and avoid it by getting it done sooner. Again often easier said than done as projects tend to evolve during the bid writing process which can make letters reflect out of date ideas or stress the wrong areas.
• Letters and other stuff need a project name. I’m guilty of really not thinking a name is that important. The acronym will be meaningless to all. On my first bid I just used a working name (all of 5 seconds thought) and right at the end asked everyone if they are happy to go with it. Mistake. Changing project name at the last minute is a pain.
• A key point. You need a good idea. And a good idea is one that is a good fit to the call. You may have a perfect methodology but if the idea doesn’t fit with the call then you could be in trouble. I’m guessing ‘It’s not really what you’re after but it’s such a good idea you must want to fund it’ is not a good sign.
• Speak to people, I mentioned the briefing day above, but also speak to the programme manager, they’re nice people! Talk about it on twitter.
• You don’t need to be an expert. I was put off for years from writing a bid about things I was interested in but didn’t think I knew enough about. You can ask people to work with you! People who know how to do stuff. I’ve just submitted a bid about Linked Data. Now I’ve followed the rise of Linked Data for years and tried to learn about it, but taking an XML file and ‘converting’ (is that the right term?) in to Linked Data, I had no idea how to even start. But I spoke to some people, who recommended someone, and they do know what to do
• Approaching others out the blue is difficult, especially if you don’t feel ‘part of it’. All I can say is ask. And if you don’t know who to approach ask people (either at the JISC or via twitter) for advice.
• If you have a clear(ish) idea of what you are going to do, broken down in to mini packages of work, andwho is going to do each one of them, then writing the actually bid is easy. Treat it like a job application. We all know that when writing a job app use the Person spec as a structure, a paragraph for each entry of the person spec, perhaps even with headings to help those marking your application. A bid is just the same, the clearly laid out structure of a bid is worth sticking to, it’s the same thing the markers will have to use to score each section. If the JISC Call document ‘proposal outline’ refers to a section which talks about Project Management, leadership, copyright, evaluation and sustainability. Then write about those things together as clearly as possible. Long winded paragraphs which ramble on about everything and make subtle implied passing references are a bain to the marker and no help to you.
• But, I have been involved in marking and assessing bids, what impressed me was the impartial way bids were judged, and the real desire of wanting to fund Good Ideas, even if the actually bid document needs a little clarity in some parts. Especially from first bidders. To stress, there was a real desire to see first time bidders (with a good idea) be successful.
• So the actually bid write up can in a way be left later than other tasks mentioned above, as it mainly involves just you, (and probably a couple of people who work closely with you to check it).
• In an ideal world this would all be done weeks before it needed submission. In real life other factors (and work) can mean it can be a last minute dash. That’s fine. But make sure a few days before it has to be submitted you put in to an email to yourself (and others involved in writing it up): The email address to send it to, the submission date time, cut and paste things such as the exact format it needs to be submitted in (how many fiels, how big), number of pages. Add a link below each of these facts to the actually source of information (jiscpress is excellent for this), so when you’re panicing and presume everything you know is wrong you can follow a link and see for yourself that it really is eight pages max for the proposal, direct from the horses mouth.

The whole process of submitting a bid, and running a project, is good experience. It often involves working with people you would not normally, and doing differently to your normal job. Now if I get a chance in the next few days I will follow Joss’ example and blog about our proposal.

Related posts:

1. JISC Library Management System Review
2. JISC, Monitter and DIUS (Department of Innovation, Universities and Skills)

Talis Aspire is a new-ish Reading List system used at the University of Sussex Library.

On Aspire, a url for a Department looks like this:
http://liblists.sussex.ac.uk/departments/anthropology.html

A page for a course looks like this (for course l6061):
http://liblists.sussex.ac.uk/courses/l6061.html

The nice thing is that you can replace the ‘.html’ with .json or .rdf – while the html only has a link to the related list, the json and rdf expose other links and relationships, such as the department.

For us, most (but not all) courses only have one list. URLs for lists are not predictable in the same way as the courses URL. E.g.
http://liblists.sussex.ac.uk/lists/EEC1E2AA-C350-DAFC-BDE4-1E9EF5EC69E5.html

So what if you want to link to a list from an external system.

The University has a portal which includes for students a list of their courses and provides a link to those that have a reading list. Two issues: it needs to know which courses have a reading list on Talis Aspire, and it needs a way to link to them.

The easiest way to achieve the former is for the central admin/portal server to query (make a http request) for each course provided by the University, if it gets a 404, no list exists for that course. It can then flag each course that has a list (using a field in the database).

And what about linking to a list. We actually have three options. The first is to simply link to the course link above (second link from the top), this will show any lists for that course, but will normally require the user to click on the single reading list link to see the list items, which isn’t ideal. The second method is to process the json data in the step above (where we were checking for the existence of a course page), the data will include the URL for the list(s) which could then be stored in the portal DB.

But the third option is the easiest. Turns out there is a nice URL you can use, similar to the one above, which will take the user directly to a list for a course, if there is only one, if there is more than more it will show a list (of, ummm, lists) instead, and this URL is predictable for any given course code.

Here’s an example of both scenarios. Thanks for Chris Clarke at Talis for the information on this.

http://liblists.sussex.ac.uk/courses/l6061/lists.html [one list, norm]
http://liblists.sussex.ac.uk/courses/v1367/lists.html (json) [two lists]

Finally, I knocked up some bad php to demonstrate this. It queries the url above, and returns either the urls of the list(s) or a message to say there is no course or lists for that course code.

http://www.lib.sussex.ac.uk/coursechecker.php?course=v1367
http://www.lib.sussex.ac.uk/coursechecker.php?course=l6066
http://www.lib.sussex.ac.uk/coursechecker.php?course=abc [doesn’t exist]

<?php

# This script is past a course code
# it returns any reading lists attached to that coursecode

$baseurl = “http://liblists.sussex.ac.uk”;

$coursecode = $_GET["course"];
if ($coursecode == “” OR $coursecode == null) {
echo “no course”;
exit;
}

# this url will exist if there are lists for this course,
# and will not exist (404) if there are not
$url = $baseurl . “/courses/” . $coursecode . “/lists.json”;
# …also available in html and rdf

# Get webpage using curl
$ch      = curl_init( $url );
# OPTION: don’t put it on screen
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$page = curl_exec($ch);
$header  = curl_getinfo( $ch );
curl_close($ch);
$statuscode = $header['http_code'];

# if status code 404 then no page
if ($statuscode == ’404′) {
echo ‘NOREADINGLIST’;
}

# we have a page for this course
elseif ($statuscode == ’200′) {
# use json to return url to list(s), though if only one the original url,
# ending in html, will take you to it.
get_list_url($page);
}

# or maybe we have another status code
else {
echo ‘NOTSURE’;
}

function get_list_url ($page) {
# JSON
$data = json_decode($page, true); #true=array

foreach ($data as $listurl => $stuff) {
# we only want lists. not courses or departments
if (preg_match(“/\/lists\//”, $listurl)) {
echo “$listurl<br>\n”;
}
}

}

?>

So once a week, the central portal will query Talis Aspire for each course to see if there is a reading list, and store the result. And it will link to lists using the URL http://liblists.sussex.ac.uk/courses/{coursecode}/lists.html

Related posts:

1. nostuff library catalogue using the Talis Platform

Way back I wrote a piece about the changing model for library catalogues, you can see it here. The main premise was that trying to maintain records in a Library Management System (LMS/ILS) for all the items you want your users to discover is no longer feasible. This is especially true in this here digital age, trying to maintain records for all the e-journals a University has access to is an almost impossible task, and LMS were not designed for thousands of MARC records to be dropped and then re-imported (i.e. sync’d) with another source. And what about all the free stuff, is an e-book not worth being discovered by users because it is free?

So let the LMS be a record of what your library physically holds, and your discovery service a place where users can find (and see how to access) resources that are of interest to their research and work. The former being just one element (albeit a major one) of the latter. Meanwhile your LMS physically holdings can be shared with other discovery systems (such as union catalogues) to show what your library physically contains.

The blog post included a diagram. Some said it was rather crude, badly drawn and basic. Its critics weren’t so kind.

New Library catalogue model

But I want to focus on the bit near the bottom, a line and ‘Endnote search’ with not even a box around it. And the point has nothing to do with Endnote. (well a little bit but who’s counting).

Story 1

I had a phone call from a trainer here at Sussex. Their problem went a bit like this: When showing postgrads Endnote and reference management, a common query arose, they wanted to import multiple records from the Library catalogue in to Endnote. This seemed reasonable. She had discovered a number of ways to import records: (a) from our discovery service – Aquabrowser (b) from our traditional library catalogue that comes with our LMS – Talis Prism (c) from within Endnote.

Aquabrowser has a nice ‘export to Endnote’ button on each record, we’d set that up. But there was no way to export multiple records. Talis Prism was no better. Endnote has a feature for connecting directly to a database (in this case our catalogue) via Z39.50. (This feature can be a real bane for libraries as it looks like a real time saver to researchers but often database vendors and ‘platform’ middle men don’t support this type of connection for institutional subscribers.) This connects directly in to our catalogue and uses the stupidly complex and aging Z39.50 to retrieve results. And from within Endnote you can then easily add them to your Endnote library.

But the problem was these search results in no way reflected the results they had when using our normal catalogue, in fact they just seemed  to be of no real order. So they had to either use the web catalogue to download one record at a time, or use Endnote but with a strange set of search results which does not reflect what they found (and what they wanted to save) on the catalogue

Story 2

Sussex is about to launch an iphone app using CampusM. Very exciting. One of the developers was chatting to me and asked if our catalogue had an API which they could perhaps use in the future to search the catalogue from within the app. Now we Library geeks love standards and apis but the only one I could think of was Z39.50.

Problem: aside from Z39.50 being the most difficult thing to get your head around and use – is that, like Endnote above, the results would be not ordered, with no relevance ranking.

The issues here are the same. Neither Endnote, nor a University iphone app is trying to be a fully fledged catalogue. Even if they did have the resources to develop search relevance and rank, it would be reinventing the wheel (which can only be bad) and ultimately would differ from the results seen within our main catalogue (Aquabrowser).

What’s more, in the model described at the top, the resource discovery layer (e.g. Aquabrowser) may hold records for items not on the catalogue (e.g. if we were starting again, our e-journal records would not be in our LMS, but indexed straight in to Aquabrowser).

So these ‘thin clients’ really just want to connect to a catalogue and get back some results in XML or similar which they just have to transform in to HTML (or display in an iphone app). They don’t want to implement a full bodied search interface, and nor should they have to.

The answer is, of course, that these clients need to an API provided by the Resource Discovery system.

The good news is that I think this is already happening, a number provide XML interfaces which developers can use. Aquabrowser for example has a SRU interface (you may need to ‘browse source’ to see this) and its own XML interface, though this is not very well documented at all (thanks to Ed Chamberlain for the help in finding the url for it).

Until our front end discovery systems provide a way to allow other systems to search and retrieve results – preferably via a common standard such as SRU – we can’t expect third parties to develop new interfaces to our catalogues. Which is a shame and frankly an embarrassment. When next look at a resource discovery or web catalogue system, the ability for other systems to connect to its API and bring back ordered results will be an requirement.

Related posts:

1. Library search/discovery apps : intro
2. Academic discovery and library catalogues
3. Library catalogues, search systems and data

One Sunday morning in January this year I got an email sent automatically from the webhosting company. It contained the output of the script that ran weekly, when all ran fine the script produced no output. When something went wrong the error messages were emailed to me. Judging by the length of the email something big had gone wrong.

The script collected data from http://roar.eprints.org/ – to be used as this weeks ‘number of records’ for each repository.

The reason became clear quickly. A major revamp to ROAR had just been launch, showing off a new interface, which used the Eprints software as a platform (essential a repository or repositories). This was a great leap forward but unfortunately removed the simple text file I used to collect the data, and what was more, the IDs for each IR had changed.

I finally got around to fixing this in May. The most fiddly bit was linking the data I collected now with the data I already had. This involved matching URLs and repository names.

Anyways. Things should be more or less as they were. A few little tweaks have been added. A few bugs still remain.

As ever you can view the code and changes here: http://trac.nostuff.org/ircount/browser/trunk

And checkout the svn here: http://svn.nostuff.org/ircount/

ircount can be found here: http://www.nostuff.org/ircount/

No related posts.

I attended an event at Huddersfield looking at their and Northumberland’s experience of Summon. http://library.hud.ac.uk/blogs/summon4hn/?p=22 These are my rough notes. Take all with a pinch of salt.

The day reaffirmed my view of Summon, it is ground breaking in the Library market, and with no major stumbling blocks. They are very aware that coverage is key and seem to be adding items and publishers. It searches items that a organisation has access to (though users can tick a option to search all items in the kb, not just those they can access). They have good metadata, merging records from a number of sources, and making use of subject headings (to refine or exclude from the search).

There was general consensus that it made sense to maintain only one Knowledge-base, and therefore in this case, using 360 Link if implementing Summon. There was also general dissatisfaction for federated search tools.

To me, and I must stress this is a personal view, there are two products that I have seen which are worth future consideration: Summon and Primo. Summon’s strength is in the e-resources realm and as a resource discovery service. Primo’s strength, while offering these features/services, is as a OPAC (with My account features etc) and personalisation (tags, lists). Both products are in a stage of rapid development.

In my view, one decision to implement one of these products – which ever it is –  will have a chain reaction. And I think this is an important point. Using Sussex as an example, it currently has Aquabrowser (as a Library Catalogue), Talis Prism 2 (for Borrower Account, reservations, renewals), SFX (Link Resolver) and Metalib (Federated Search).

Two example scenarios (and I stress there are other products on the market and this is just my personal immediate thoughts):

One: Let’s say Sussex first decide to replace Metalib with Summon. They would probably cancel Metalib (Summon replaces it). Probably move from SFX to 360 Linker (one Knowledge base). May then wish to review our Library Catalogue in a years time: Primo is no longer on the cards (too much cross over with Summon, which we now already have), so they either stick with Aquabrowser (but the new SaaS v3 release) or perhaps move to Prism 3 (Talis’ new-ish SaaS Catalogue). Sussex would end up with no Ex Libris products, but would potentially subscribe to several Serial Solutions products.

Two: Let’s say Sussex decide to replace Aquabrowser with Primo (which acts more like a Catalogue than AB). They cancel Aquabrowser. Primo would (in addition to being the primary OPAC) have Summon-like functionality, allowing users to search a large database of items instantly, with relevance and facets. So Summon would not be an option. Stick with SFX (Metalib would be a side feature of Primo, with a Primo-like interface). With a number of Ex Libris products they would want to keep an eye on the Ex Libris URM (next genration LMS), they would have no Serials Solutions products.

The following are some notes from the day:

Sue White from Huddersfield Library started the day, saying it is probably the best decision they have ever made.

No related posts.

This isn’t a comprehensive review of dev8d, checkout the twitter stream (that link uses twapperkeeper by the way, the guy who created it was there and it was great talking to him), and wiki for and much else on the web for what went on.

Things I saw that I liked:

• An iphone app developed which allowed you to point your iphone in different directions (think augmented reality) and show what was currently on in the room your iphone was pointing in.
• This was possible because the wiki and programme exposed data in a machine readable way, using RDF & ical for each event, geo co-ordinates for rooms. With out this the above would not have been more difficult to produce. It proves the saying: the coolest thing to be done with your data will be thought of by someone else. And likewise, set your data free.
• Another app, little more than a simple iphone switch, allowed you to turn a light on and off. Of course the iphone could have been anywhere in the world when it turns the light on and off, the app calls a REST web interface on a notebook running a webserver which was connected to the light. Simple implementation of a nice idea, which much potential (someone was discussing how they want to be able to control their heating from their phone if they are out for the night, or if they are coming home early).
• RFID tags connected to the door of each room, each attendee had a small RFID tag so they could record if they entered or passed a room. A web interface was built to display this data and lots of ideas of how we could build on this. The RFID readers where a simple consumer model (from Touchatag), connected to netbooks and similar near by, and a mac mini acting as a central place to collect all the data. It was coded in ruby and used Redis as a quick and neat way to collect the data.
• A Developer Happiness iphone app which shows a number out of 10 representing on average how happy those at Dev8d are. It builds on the Happiness Pipe site built last year.
• I’m not sure how far they got, but some explored an open source OCR code library. They had the idea they could use these to try and OCR a frames from a videos on the web. This would have massive potential. indexing words from videos (say on youtube) whether super implanted on the screen, or signs and general wording in the image, would help improve searching for videos no end for starters. (of interest to us whose work includes resource discovery).
• A cheap 3D printer, which can almost replicate (print) itself. Clearly a device that can build a huge array of items (from wine glasses to children’s shoes, though made out of a plastic like substance) has potential to change the world we live in. Was very impressive to see it.
• Lots of hardware hacking going on and clever genetics stuff too, in fact loads not covered here.

I created a twitter account which posts the latest blogs, videos and photos. You can read about it here (this is more for info, sounds stupid when coming after the amazing things above). I also started to learn Ruby on Rails and python.

I was somewhat envious of those who by the end of the few days were able to say ‘look what I’ve built’, but then I’m no a developer as such, and certainly not as clever or geeky as some at the event. That is not to say I wasn’t busy, trying out lots of little things, discussing things with others, meeting lots of new people, attending expert talks and a couple of coding labs and trying out lots of things.

An example of 45 minutes: On seeing a tweet saying Ed Summers (LCSH Linked Data fame) was starting his expert talk for 15 minutes I went running in to that room to catch it (with power cable tailing behind me), great talk and was followed by a couple of guys from Mendeley – a great app and like last.fm they are collecting a lots of data which we should be able to do some interesting things with using the new API. This was followed by a great talk from Kate Pekacar from the MLA explaining their efforts to open the data of various collections and their keenness that the likes of Dev8D create useful new ways for people to discover these collections. Three different talks yet all interesting stuff and related to the day job.

I was lucky enough to attend the awards ceremony on the Friday night. One of those receiving an award was Garry Bulmer, who had run the popular Arduino workshops, who said teaching and working with those who were so enthusiastic and smart was a joy. This seemed to sum up the event well. It works because those attending want to challenge themselves, try something new, work with new people and move sightly outside their comfort zone. It would be easy to not do much at barcamp style events, but the reverse actually happens as people try and squeeze so much in to the few days, often working on tasks late in to the night.

If I had to sum up what I got out of this – a difficult task not due to lack of quantity but the sheer broadness and soft qualities of what has been gained – I’d add to what I’ve already out above:

• There were 15 minute expert lightening talks that have informed and got me thinking more than entire one day workshops and conferences.
• In terms of hard facts, getting started with two programming languages both very useful when working and developing for the web. Learning these would have taken days in themselves, it’s hard to stress how much you can pack in to an afternoon when you have an excellent expert leading the session working with bright people in the room (my brain hurt a lot throughout).
• Motivation is hard to identity, quantify and where exactly it originates, but I come away from this even with a higher opinion of the industry I work in, the Tech/dev community around Higher Education and our ability to break through old ways of working and needless bureaucracy. I think it helps those attend see a bigger picture to what they are doing. Tired yet with a fresh urge to build new things and Get Things Done.
• I ended up talking to lots of people I wouldn’t do normally, neither personally or socially. This is useful in itself, but I think the experience itself is useful (especially for the socially inept like me). I found myself a few times suddenly talking to people (often from the States) who have created high profile web apps, including those which have nothing to do with HE or libraries.

All in all. Tired. Good event. Good food. Metropolitan line crap. Well organised. Nice people. Brain hurts. Learnt lots. Shambrarian meeting a success . Chatted Lots. Drank a moderate amount while adhering to medical daily recommended limits. Obviously.

Related posts:

1. Dev8D

I’ve finally got around to spending a bit of time on the ircount code.

This post goes through some of the techy stuff behind it. If you’re just interested in features, I’m afraid there’s none yet, but you can now compare more than 4 repositories, but that’s as far as you’ll want to read.

Subversion and Track

The first thing I did a few months a go was create a Subversion repository. I seemed to time this quite well in learning how to use subversion just as every man and his dog gets excited by GIT.

I also installed Trac (using Dreamhost‘s useful one-click install), which I haven’t really used other than to browse the code and view changes (time-line).

The repository is publicly available from here: http://svn.nostuff.org/ircount/

The trac site (which can be used to browse the code) is found at: http://trac.nostuff.org/ircount/

It took me a while to get svn working well. Originally I would edit the files in a local working copy of my Macbook, and then use subversion to load these to the server to test. Of course, this meant every little change had to be manually checked in to test it. I got apache and php working on the Macbook, and setup the Mysql db (on the Dreamhost server) to allow connections remotely which allowed me to test/use files located in my local copy. This seems to work well.

I use svnx as a GUI Mac client. It’s free and is easy to use.

The code. The rewrite

I don’t do much code writing or developing. Anyone glancing at my work will take that as plain obvious.

I’ve realised when writing code I have a tendency to be very linear and unconsciously put efficiency before good design. Anything more than one database call per page was unthinkable, loading in other pages a sin, which often led to large unwieldy while loops processing the results from a massive database call. Database calls are mixed with html output mixed with logic.

This is just about ok when showing information about one Institutional Repository. But when comparing a number, it becomes unreadable, and not in any state to be reused. The key aspect of comparing a number of IRs is that you need to ascertain a number of facts for the page as a whole (the earliest data collection date for the page, the higher record count for the page – for the chart for example – which could be from any of the IRs).

My  aim was that the page files should be little more than calls to a few discrete functions.

I’m not quite there yet, but it’s a start. archive.php is mainly a set of function calls, but there is still too much in there, and random bits of html code dotted around. There’s also too many arrays holding information the repository (php) objects can provide. include.php holds the functions, but is now a little bit unwieldy itself, with functions ordered randomly. The third file of note is class.archive.php. This is a repository object, which can grab data from the db for the repository, and return it to the calling page in various ways.

My original plan was to merge the code for showing one repository, with the code to show more than one, to make it easier to implement changes (not having to update two files). By the end of it, I’m now wondering if it would be easier to have two files again, for the little changes, which both call the same core functions.

Google Chart

Google Chart is a great API and I recommend anyone to play with it.

However one of the problems of the last version of ircount is that the URLs for the chart images often became too long (more than 2,000 characters) resulting in no chart being shown. The Chart URL includes each data point separated by a comma, so four repositories multiplied by 100 weeks (for example), multiplied by 4 or 5 digits per datapoint (4 digit number plus a comma), it soon adds up.

One solution was to only pass data per month rather than per week (roughly reducing the number of data points to a quarter of the original). Another would have been (and probably will do in the future) to make use of Google Chart’s encoding function, made easy using these helpful functions.

Overcoming this in an efficient way was a challenge. Originally I had a Google Chart PHP object. IR data would be passed to it in one method, and another method would return the URL.

This seemed sensible at the time, but deciding which php object did what became confusing. For example, the first thing the chart object needed to do was decide if the URL would be too long for the Repositories in question, taking in to account the data we had for each repository. Should the chart object loop around each data point for each repository to first decide how many there are? or should the repository object be handling this by telling the chart object how much data it has? How to avoid the need to loop through the same data several times. Does it matter which object does the work? It’s for the chart, so the chart object should do it, yet other parts of the page may want this info about the repositories, so the repository objects should provide it for all.

In the end, I did away with the chart object and used a function instead, which is passed an array of repository objects, which in turn handle a lot of work.

Future

The foundations are about there. For any page I (or anyone else) wishes to create. A couple of lines are all that are needed to take one or more repository IDs passed in the URL and load in all the data for them, ready to be used as needed. We can then easily call a table or chart to display for these repositories (or a subset of them).

As I mentioned above, the only real improvements are the ability to show more than 4 repositories at once (the chart stops showing once you get to about nine repositories), and the chart is more robust and will now show when it would have failed to do so in the past.

Google Chart does have an encoding which allows far more to be passed in a condensed way, and this php function looks very useful for using it.

If I was starting again today I would look to use a framework such as Zend Framework or CakePHP, or maybe even have a go at Ruby on Rails. But perhaps a third rewrite is a little over the top for now.

I need to tidy up the table view a bit (some nasty code there) and then look to a few new features, may be collecting more data, and exposing it in some computer friendly formats such as atom.

So ircount is really no more than a play thing for a bad coder to make mistakes and learn a little bit along the way. slowly. but if you have any ideas or thoughts I would love to hear them.

Related posts:

1. ircount : new location, new functionality
2. ircount : Repository Record Statistics
3. ircount : update

Today I saw a tweet from juliancheal, which mentioned he was setting up his virtual server on slicehost. I hadn’t heard of this company but their offering looks interesting. This got me thinking about cloud hosting and I decided it was time to actually try out Amazon’s AWS EC2.This allows you to run a virtual server (or multiple servers) in the Amazon cloud, servers can be created and destroyed by a click of a button.

First thing is to get a server ‘image’ to run in the cloud. Thankfully many have already been created. I went for a recent ubuntu server by Eric Hammond. This is basically a ubuntu server vanilla install, but with a few tweaks to work nicely as a EC2 virtual server. Perfect!

Signing up is quick and easy, it just uses your Amazon (the shop) credentials. Once created, you are taken back to the main control panel where you can see your new instance, including details like the all important public DNS name.  Just save a private key to your computer and use it to ssh in to your new server.

e.g.: ssh -i key1.pem root@ec2-174-129-145-xx.compute.amazonaws.com

(you may need to chmod 400 the key file, but all this is documented)

Once in, well it’s a new server, what do you want to do with it?

I installed a LAMP stack (very easy in ubuntu: apt-get update and then tasksel install lamp-server). I initally couldn’t connect to apache (but could from the server itself using ‘telent localhost 80′). I presumed it was a ubuntu firewall issue, but it turned out you also control these things from the AWS control panel. The solution was to go to ‘security groups’ and modify the group I had created when setting things up and adding HTTP to ‘Allowed Connections’. This couldn’t of been easier. And then success, I could point my browser at the DNS name of the host and saw my test index page from the web server.

So now what? I pondered this out loud via Twitter, and got this reply:

Excellent idea!

Good news: vufind has some good – and simple – documentation for installing on ubuntu;

http://vufind.org/wiki/installation_ubuntu

Following the instructions (and editing them as well, they specified an earlier release and lacked a couple of steps if you weren’t also reading the more general install instructions) I quickly had a vufind installation up and running. Took around 20-25 minutes in all.

Now to add some catalogue data to the installation. I grabbed a MARC file with some journal records from one of our servers at work and copied it across as a test (I copied it just by using a scp command logged in to my ec2 server). After running the import script I had the following:

If the server is still running when you read this then you can access it here:

http://ec2-174-129-145-75.compute-1.amazonaws.com/vufind/

EC2 is charged by the hour, and while cheap, I can’t afford to leave it running for ever.

So, a successful evening. Mainly due to the ease of both Amazon EC2 and Vufind.

A final note that if you are interested in EC2 you may want to look at some notes made by Joss Winn as part of the jiscpress project: http://code.google.com/p/jiscpress/wiki/AmazonWebServices

Both Ec2 and vufind are worth further investigation.

Related posts:

1. VuFind in 8 minutes using Amazon EC2
2. Amazon : censor reviews of the high arts

I put this in to a seperate post. It continues on from my previous post, but didn’t want my notes of the day to be taken over by my ill thought views.

Personal Thoughts

Reluctant to give some thoughts as I know so little about the service. However… (!)

There seems to be two clear areas here: Data formatting and Data storing. There is some linkage (Preserving surely covers both, formats can become obsolete, Servers die), yet the two seem to be somewhat seperate.

Both require IT skills, but IT is a broad church, the former is technical metadata (and is very much IT and library) and in the general area that I sees covered in the Eduserv efoundations blog.

The latter in its simplest form is hard core infrastructure. Disks, sans, servers, security, but also has elements at the application level (how do we access it, using what software, repositories? CRIS? Fedora?).

On another issue, while it is easy to say that libraries should take the lead, I think we need to be cautious. With the current climate of frozen or decreasing budgets nationally, and journal subscription pressure, how wise is it to go to the University’s executive and demand funding for resources/staff for data management. We know it’s important and could make the process of research more efficient, but there are other things higher up a Universities list of priorities (NSS/atracting good students, REF, research funding). Even at a library level, journals help researchers do research (which brings funding), and keep students happy because we have the stuff they need (NSS). How many journals should we cancel to focus on Research Data? Why? The recent JISC call will help with providing a business case.

The problem at the moment is that there are not enough clear benefits for most Universities to steam ahead with this. Let’s clarify this: not enough benefits for the institution itself. The benefits are for the UK as a while (actually, the while world). It’s the UK-wide economy and research that will benefit. So maybe it needs UK-wide funding. It’s easier to convince someone (or something) to spend money when the benefits for them are clear. In this case the benefits are for UK so it should the UK which sets aside explicit cash (via HEFCE, JISC, and so on).

And this is happening, with the JISC call (talked about today), amongst other things it will help build examples.

But I’m not sure if the institutional level is the best one. Australia has been successful with a centralised approach. We have a number of small Universities, and those which only have one or two departments which are research active. Yet the resources/knowledge required of them will be similar to that of a large institution. Will this leave them at a disadvantage?

On another note, it seems the range of data is vast. When dicussing this, I always – incorrectly – picture text based data, of vearying size, perhaps using XML. Of course this is blinkered. For auido, images and similar should a data service just provide a method to download, or a method to browse and view/listen? When it comes to storage and delivery, should we just treat all data as ‘blobs’ – things to be downloaded as a file, and we no nothing more with it? This makes it easy and repository softwareapplications (eprints/dspace/fedora) are well placed to cater to this need. But I get the impression that this is somewhat simplistic. Perhaps this means a data service needs a clear scope, otherwise we could end up building front end applications which mimic flickr, youtube and last.fm all in one. A costly path to go down.

[all views are my own. are wrong, badly worded, ill thought, why are you reading this?, just think the opposite and it will be right, etc]

Related posts:

1. Event: The Data Imperative: Libraries and Research Data
2. Linked data & RDF : draft notes for comment
3. Research in the Open: How Mandates Work in Practice

Today I’m at the one day event ‘The Data Imperative: Libraries and Research Data’ at the Oxford e-research Centre. As usual, these are my own rough notes. There are mistakes, gaps and my own interpretation of what was said.

Paul Jeffreys : Director of IT, Oxford University.

Started off giving an overview of where this has come from. e-Research is more than just e-Infrastructure. e-Research is not just about outputs, but outputs (articles/data) are a part of this, and an discreet area to work on.

This is a cross-discipline area, it needs academics, University executive, research office, IT and Library. Libraries have skills that have to be fed in to this.

EIDCSR : ‘Enough talking,  let’s try and do it’, selected two research groups to work with, but not a pilot, a long term commitment. He talks about Oxford’s commitment to a data repository, it stresses cross agencies, mentions business models and feeds in to a senior research committee (the quote is far too long to add here!).

As each HEI is facing the same issue, it makes sense for national activity. but how much is done locally and how much is done nationally.

What is the vision of research management data? To what extent is managing research data the role of the Library/librarians? Is data management and data repositories a new kind of activity? Is it Librarians or Information Professionals who are charged to take this forward? [cjk: i thought they were one and the same]

John K Milner : Project Manager UKRDS

Can’t just use existing subject specific data centres. Need for cross-discipline (eg climate change) and therefore universal standards and methods so one subject can use another subject’s data with ease.

Feasibility study:

Understand what is happening today? where are the gaps. Avoid re-inventing the wheel.

Four Case studies (Bristol, Leeds, Leicester, Oxford), views of ~700 researchers over all disciplines (inc the arts).

What did they learn?

About half of the data has a useful life of 10 years? 26% has ‘indefinite’ value, ie keep for ever’ Nearly all kept locally (memory stick, departmental server, [cjk: not good!]).

21% use a national/international data centre. 18% share with them.

UK has rich landscape of facilities, skills and infrastructure.

The management of data from a research project are now starting to be directly funded, which is important.

What are others doing? Are we in step with other countries? Yes. US spending $100 million on 5 large data centres. Australians are leading in this area, and have a central approach to it. Canada and Germany also have similar developments.

Aim: to set up a framework for research data.

Why Pathfinder: not a pilot but the start of a long term commitment.

[my notes miss a bit here, had to deal with a urgent work issue]

Service must be useful and accessible. Need a framework for stakeholder engagement.

This is non-trivial. Lots of parties involved, a lot of effort needed.

Citation of datasets is of growing interest to some researchers, this may help engage the research community.

Showing a diagram of UKRDS Basic processes. Split between ‘Research Project process’, Research data sharing process and UKRDS Services and Administration

Diagram doesn’t focus on curation but on accessibility (inc discovery, stable storage, identity) as this seems like the most important part. Discovery:Google, Identity(auth):Shibboleth.

Making it happen.

Need clearly defined service elements, will involve DCC, RIN and data centres.

HEIs need a reliable back-office service to handle working with data.

UKRDS is extremely challenging, nothing is easy and it is expensive. Needs support of funders and HEIs, need the right bodies to show leadership and shape policy. It will take time.

Q: Is it limited to HEI or public sector (museums etc). A: a more complicated issue, but they are working with the liked of Connecting for Health and DEFRA.

Q: Copyright. A: HEI often don’t own copyright. Data Management Plan (Wellcome are funding Data planning as part of funding)

Q: Is it retrospective? A: Could be. [he did say more]

Q: Could UKRDS influence ‘reputational kick back’ [nice phase!] e.g. for the REF. A: Yes, in discussion with HEFCE.

Q: Research Councils A: they are in discussion with RCs but Wellcome very much taking the lead (leap of faith) in the area. The whole key is a ‘value proposition’ which makes a case for funding this.

Q/point: Engage government/politicians.

Q: Challenge in explaining what it is, especially for subjects which are already doing something with data. How can we tap in to those already doing it? A: there is sometimes a missing link between researchers and subject national data centres. No real relationship between the two. Which is a problem in cross-subject research.

Research data management at the University of Oxford: a case study for institutional engagement – Luis Martinez, OeRC, Sally Rumsey, Oxford University Library Service

More of a ‘in practice’ talk, rather than high level.

Luis Martinez

Scoping study: ‘DataShare project‘. Talking to researchers they found some couldn’t understand they own old data, some wanted to publish their own data, some found data was lost when academics moved on.

Requirements: Advice/support across research cycle (where to store it, how, etc), Secure Storage for large dataset. Sustainably infrastructure.

Lots of different Oxford units need to be consulted (library, it, research technology, academics, legal, repository etc).

Findings after consultation: there is actually widespread expertise in data management and curation amongst service units, and other findings. DataShare: new models, tools, workflows for academic data sharing.

Data Audit Framework: (DAF) adapted this to Oxford needs and used it to document practices in research groups.

‘Policy-making for Research Data in Repositories : a guide‘ [pdf]

The EIDCSR challenge: two units that both research around the human heart. The two groups share the data between them and agree to produce 3d models using the combined data. They are helping this groups do this, using a ‘life cycle approach’.

Using the DAF to capture the requirements. Participating in the UKRDS Pathfinder (as above).

They have a blog http://eidcsr.blogspot.com/

Sally Rumsey

Starts of by talking about the roles required regarding the library. They have Repository staff, librarians, curators, but not so sure about ‘data librarians’.

What should of data should they be responsible for? Some stuff can go to a national service. There are vast datasets (eg Oxford Supercomputing centre), who has the expertise to make these specialised datasets available. Some departments already have provision in place, fine, why rock the boat.

Long tail. Every thing else (not above). No other home, lots of it, Academics asking for it, highly individual (ie unique), hums and sciences.

Things to consider: live or changing data Freely available or restricted? Long term post project?

Showing what looks like a list of random words/letters/strings of chars, an example of some data they were asked to look after from the English department.

Showing a diagram showing that Fedora (a repository system which is strong on metadata/structure but lacks an out of the box UI) is key to the setup. many applications can sit on top of it. Institutional Repository is just one application which runs on top of Fedora.

ORA (IR) for DATA: actual data can be held anywhere in University but ORA is a place of discovery. Allows for referencing of data. Might want to link to ‘DataBank‘ (a proof of concept to show what is possible).

Databank: how do you search/discover? First things added were audio files, perhaps then photos, how do you find them?

Showing Databank. Explaining that everything has a uid so we have cool URLs, and hence you can link to it [yes!]. Explaining how you can group an audio object, a related photo object and a related text object (perhaps explaining it).

End of morning discussion (I’ll just note some points I picked up):

This seems to raise such huge resource implications.

DAF is flexible, you can pick elements of it to use.

Non academic repositories, such as flickr, preservation issues, if they go down. [unlike the AHDS then!]

The Research Data Management Workforce – Alma Swan, Key Perspectives

Study commissioned by JISC, looking at the ‘supply of DS [data scientists] skills’.

NSF Roles:

• Data Authors – produce data
• Data Managers – more technical people – often work in partnership with data authors
• Data Users
• Data Scientists – expert data handlers and managers (perhaps ‘Data Manager’ was a confusing name).

Our Definitions (but in practice the roles and names are fuzzy):

• Data creators or authors
• Data Scientists
• Data Managers
• Data Librarians

Data Creators

Using DCC Curation lifecycle model, these are the out ring. But not all of it, and do things not on the ring, such as throw data away.

Shows picture of an academics office. Data is stored in random envelops.

Data Scientists – the focus of this study

Work with the researchers, in the same lab. Do most things in the DCC model. Are computer scientists (or can be one), experts in database technologies, ensure systems are in place, format migration. A ‘translation service’ between Researchers and computer experts.

Lots of facts about this, based on the research. Often fallen in to the role by accident, often started out as a researcher. Domain (maths, chemistry) related or Computer training. Informatics Skills: well advanced in biology and chemistry. Majority have a further degree. Need People skills. Rapidly involving area.

Data Librarians

Only a handful in the UK. specific skills in data care, curation. Bottom half (or bottom two thirds) of DCC model.

Library schools have not yet geared up for training. Demand is low, no established career path. Good subject-based first degree is required.

Things are changing, eg library schools are creating courses/modules around this.

Future Roles of the library

train researchers to be more data aware

Pressing issue inform researchers on data principles, eg ownership.

Open Data : datasets

A growing recognition across all disciplines that articles aren’t enough, datasets are what are needed to be in the open.

Datasets are a resource in their own right.

Publishers do not normally claim ownership of datasets. Some are (usual suspects)

Funder may own Data, Employers may own data. No one seems sure. Several entities may own the data.

In some areas of research journals play role in enforcement.

Some journals are just data.

Using PDF for data is very very not good.

Do we leave preservation of data to publishers [cjk: no! they should have nothing to do with this, the actors are Universities, their employees and their funders]

Simon Hodson – JISC Data Management Infrastructure Programme

Something problem, not easy to tackle. Would be a mistake for institutions to wait. The Call is designed to better understand how its data management facility can be taken forward.

Detailed business cases are needed.

Needs everyone (HEI, funders, data centres, RIN, etc) to be on board.

the Call will have an Advisory Group.

‘Exemplar projects and studies designed to help establish partnership between researchers, institutions, research councils.

See DCC as playing a major role in developing capacity and skills in the sector.

Tools and technologies: tools to help managers make business case internally, institutional planning tools (building on DAT, DRAMBORA, and costing tools). Workshop 1oth June DCC to review progress/outcomes of DAT project.

Two calls planned for the early Autumn.

2 June Call: Infrastructure. To build examples within the sector. Requirements analysis -> Implementation plan -> Execution thereof -> business models.

Bids encouraged from consortia.

Briefing day 6 July. DCC will provide support for bids, including a specific helpdesk.

There may be a Digital Curation course in the next few weeks.

Libraries and Research Data Management; conclusions – Martin Lewis, Director of Library Services and University Librarian, University of Sheffield.

Martin had been chairing all day and here he sums up and bring the various threads together.

The library research data pyramid. Things at the bottom need to be in place before things higher up. At the bottom, training in library (confidence), Library schools. Then develop local data curation capacity, teach data literacy. Higher up: research data awareness, research data advice, Lead on local policy. At the very top ‘influence national data agenda’.

Summary

An excellent day and excellent knowledgeable speakers. Nice venue, and most importantly, I found the only plug socket in the room!

This is clearly an emerging area. Many are in the same posistion, they are aware of the (Opene) Research Data developments, but nothing has yet happened at their university, nor academics queuing up to demand such a service. This is a good thing and it needs to happen, and Universities need to start acting now. But there are many preasures on University resources at the moment. How high on the institutional priority list will this come?

[Very finally, I did another audioboo experiment. On the fly, with no pre-planning, I recorded about 2 minutes of talk during the lunch. It's random, with no thought, many umms, a pointless 'one more thing' and basically wrong. laugh at it here]

Related posts:

1. The Data Imperative: Libraries and Research Data : comment
2. Research in the Open: How Mandates Work in Practice

Today I’m at the RSP/RIN Research in the Open: How Mandates Work in Practice at the impressive RIBA 66 Portland Place.

Slides can be found here (not available when I made this post, as semi excuse as to why my notes miss so much). These are rough notes, which I’m making available in case others are interested, apologies for mistakes and don’t take it as gospel!

After an introduction by Stéphane Goldstein, kicking off with Robert Kiley from the Wellcome Trust.

Wellcome trust mandate since 2006, anyone receiving funding from Wellcome Trust must deposit in to pubmed, now uk pubmed central. SHERPA Juliet lists 48 funder policies/mandates.

Two routes to complying to their mandate: (route 1) publisher in open access / hybrid journal (preferred), Wellcome will normally pay any associated fees. However when paying the publisher, they expect a certain level of service in return (deposited on behalf of author, final version available at time of publication, certain level of re-use. Route 2 Author self-archives author’s final version within 6 months of publication. It was stressed that the first option is very much preferred.

“Publication costs are legitimate research costs”. To fund Open Access fees for ALL research they fund would, they estimate. take up 1-2% of their budget.

Risk of ‘Double payment’ (author fees and subscriptions). OUP have a good model here.

Still to do:

• Improve compliance (roughly 33%, significant increase after letters to VCs),
• improve mechanisms (Elsevier introduced OA workflow which resulted in significant increase in deposits, but funders/institutions/publishers all need to play a part here),
• Clarifying Publishers OA Policies  (and re-use rights, didn’t catch this).

Research Councils UK – Astrid Wissenburg, ESRC

Starts of by talking about drivers for OA in the RC. Value for money, ensuring research is used, infrastructure and more.

Principles: Accessible, Quality (peer review), preservation (she’s moving through the slides fast)

April 2009 study in to OA impact, provides options for RC to consider.Findings

• Significant shift in favour of OA over last decade
• Knowledge/awareness still limited. Confusion
• Engagement with OA varies by subject area.
• Too early to access impact of RCs policies.
• Drivers
  • Not speed of dissemination
  • principles of free access
  • co-authors views are a big influence (mandates less so!)
  • some evidence that OA increases citation just after publication
  • limited compliance monitoring by finders
  • concern about impact of learned societies (but no evidence of libraries cancelling journals)
  • little evidence of use by non-researchers (CJK comment: interesting, I would imagine this may grow, wish newspapers would link/cite journal articles)

Both models (oa journals/repositories) supported by RCs, level playing field.

Pay to publish findings: limited use, barriers, costs, awareness, not RAE. would lead to redistribution of costs from non-academic to academic areas.

OA Deposit (repositories): from grant application from 1 Oct 2006, so a three year project starting then will only be finished in Autumn 2009. Acknowledges embargos but ‘at earliest opportunity’.

75% researchers were not aware of the mandate. diversity across subjects. ‘In general, no active deposit’.

A slide showing % of awareness broken down by RC, interesting.

From the highest level RCs are committed to supporting OA (this will increase). But change takes time.

Some issues: what do to with embargo periods, difficult for funders to manage (are there incentives we could use), depends on existence of repositories, multiple deposit options confusing to researchers, awareness/understanding.

UKPubMed Central – Paul Davey, Engagement Manager, UKPubMed Central

Aims to become the information resource of choice for biomedical sector.

Principles: freely available, added to UK pubmed central, freely copied and reused.

Departmental of Health have clear policy to make research freely available.

95% of papers submitted are taken care of (deposited?) by the authors. only 0.5% submitted by academics (PIs/colleagues)

1.6 milion papers in uk pubmed central. 366 thousand downloads last month.

Core benefits: transparency, cutting down duplication, greater visibility.

Text mining, grabbing key terms from an article  (a little like  OpenCalais does)

Mentions EBI’s CiteXplore, encouraging academics to ink to other research.

Pubmed UK includes funding/grant facilities search. Can link articles to funding grants.

In short, backing from key funders, will make researchers more efficient, researcher’s visibility will increase.

Beta out in the Autumn, new site in Jan 2012.

Questions:

Worried about text mining, need for humans to moderate this. response: Limited finding in this area so human intervention also limited. really need specialist to answer this fully.

Question about increasing visibility of UK pubmed central, referring to Google, response: getting indexed by Google very much part of increasing visibility.

Question about Canadian ‘pubmed central’, response confirms this and mentions talk of a European pubmed central. Potential of European funders using UK pubmed central as a place to deposit research (like everything here, not sure if I’ve noted this right).

PEER – Pioneering collaboration between publishers, repositories and researchers – Julia Wallace

Funded by EC, not a ‘publisher project’.

Three key stages of publication: NISO Author’s original, NISO Accepted Manuscript, NISO version of record.

Starts of talking about the project, interesting stuff but failed to take notes.

From the website:

PEER (Publishing and the Ecology of European Research), supported by the EC eContentplus programme, will investigate the effects of the large-scale, systematic depositing of authors’ final peer-reviewed manuscripts (so called Green Open Access or stage-two research output) on reader access, author visibility, and journal viability, as well as on the broader ecology of European research. The project is a collaboration between publishers, repositories and researchers and will last from 2008 to 2011.

Seven members: including a publisher group, university, funders etc. Various publishers involved, big and small and about six European repositories taking part.

Approach / content:

• Publishers contribute 300 journals, plus control
• Maximises deposit and access in participating repositories
• 50% publisher submitted 50% author submitted.
• Good quality, range of impact factors. Publishers set embargo periods, up to 36 months.

Publishers will deposit articles in to the repositories via a central depot for their 50% of articles submitted (50% fulltext, metadata for the remaining 50%). Publishers will invite authors to deposit for the ‘author’ 50%

Technical: using PDFA-1 (where possible) and SWORD

Three strands: Behaviour, Usage (looking at raw log files), Economic. Also looking at Model Development (the three strands will look in to this).

Question about why they chose PDF (not very good for text mining). A: wide range of subjects and publishers means that PDF the best fit

Economic Implications of Alternative Scholarly Publishing Models, also Loughborough University’s Institutional Mandate – Charles Oppenheim, Loughborough University

‘Houghton report’ looks at costs and benefits of scholarly publishing.

Link to report http://hdl.handle.net/2134/4137

Link to main website and models http://www.cfses.com/EI-ASPM/

• Massive savings by using OA, UK would benefit from this.
• Savings include: quicker searching, less negotiations, savings not just in library budgets
• 2,300 activity items costed.
• This report currently final word in economics of OA.
• Charles Talks about the various methods and work involved in producng this report.
• a 5% incease in accessibility would lead to savings (or extra money to spend) in research/he/RCs
• Hard to compare UK toll/open access publishing costs as one pays for UK access to content from across the world, the other pays for UK content to be world wide accessible.
• Keen to role this out to other countires
• Publishers response to report: furious!

Now for something completly different: Loughborough approve a mandae a few months a go, to come in to affect Oct 09. An intergral part of academic personal research plans (only those research items in the IR will be considered at the review). Now have over 4,000 items

Lunch and audioboo

During lunch I did an experiment using audioboo. Would I be able to summarise the morning, on the fly with no planning, in a brief audio recording. The answer, as you can discover, is ‘no’, but fun to try, and made me think of what I had taken in during the morning. Link to audioboo recording. or try the embedded version below.

Institutional Mandates – Paul Ayris, University College London

Paul starts off by shoing a number of Venn diagrams, for example: 90% of its research is available online, 40% available to an NHS hospital

What do UCL academics want

• as authors: visbility / impact
• as readers: access
• delivery 24×7 anywhere

UCL madate, a case study:

Looking global is an important part of UCL (for PIs rankings etc). Number of systems in their publication system: Symplectic, IRIS, eprints, data mart (and portico, FIS, HR). Symplectic (or similar tool) and IRIS seem central in this model. Plan to automatically extra metadata from other external places (publication repositories.

How did they get the mandate? Paul spoke at UCLs senate (Academic Board), the agreed: all academic staff should record they own publication on a UCL publication system, and, teaching materials should all be deposited in their eprints systems.

UCL are going to set up a publication board to over see the OA rollout; to advise, monitor, oversee presentation and more.

Next steps: market/exploit, set standards for online publication, to advise on ongoing resource issues in this area. Also, establish processes, Statistics and management information, advise on multimedia, copyright issues.

‘Open Access is the natural way for a global university to achieve its objectives’

Question about blurring the line between dissemination and publication, and that some of UCLs aims seem more fitting of ‘publication’. Paul agrees, still trying to figure this out.

HEFCE – Paul Hubbard, Head of Research Policy, HEFCE

Policy: Research is a process which leads to insights for sharing. So Scholarly Communication matters to HEFCE. Prompt and accessible publishing is essential for a world class research system.

Supporting research: JISC, RIN, Programmes to support national research libraries (UKRR), UKRDS. Mentions Boston Spa (BL) document centre as an example of our world class sharing.

Internet opens up new ways of scholarly communication and sharing.

What do HEFCE want to see:

• Widest and earliest dissemination of public research.
• IP shared effectively with the people best placed to exploit it (CJK comment, i don’t think it is publishers!)

Committed to: UK maintaining world leading research, funding that fosters autonomy and dynamism, research quality assessment regime that supports rather than inhibits new developments.

As we move forward, things may be unclear those HEIs with repositories will be at an advantage.

Paul finishes up with a personal view of scholarly communications in 2030. He sees to forms of communication: discussion (building up ideas), and writing up a formal firm idea/conclusion based on these. HEFCE supports – through the likes of JISC – a range of tools and systems to enable this. (sorry that was an awful summary, he said much more than that!).

Answered a question as to why IRs, HEIs are the places to administrate/manage. Websites people go to see research for a particular subject need to be overlay systems harvesting from IRs.

[hmm, does 'university requirement' sound better than mandate?]

Institutional Policies and Processes for Mandate Compliance – Bill Hubbard, SHERPA, University of Nottingham

99.9% of academics do not object to Open Access, but need to show it will not change how they work. Librarians going to be much more part of the research process. Most people (including most publishers) are in favour of Open Access.

Other pressures on the systems, lack of peer reviewers, rising prices of journals, growing need for different forms of scholarly communications (e-lab books, multimedia), public demand for highest value for money ‘public should get what they pay for’,

Not if we change, but how we change. Research has to change seamlessly. Mandates have a value-added basis with fast delivery of benefits. Need integrated processes, need integrated support (we don’t want researchers to hear different messages from their Uni, funder, publisher, etc).

Authors need to know ‘what do i meed to do’. Need to make it less confusing, need to make it clear when they can get help.

First step compliance: how can funders improve compliance, how can authors be supported?

All 1994 and Russel Group now have IR (Reading, I think, just setting one up now).

Compliance for mandates makes it better for us admin/support staff, and for the University generally.

Institutions need a compliance officer (perhaps repository manager). Funders need to ensure these people have the information they need. Share compliance information

I’ve missed so much of Bill’s talk here, he moves fast (and passionately) and lots of points.

After Bill’s talk there was a panel session.

Twitter

Finally check out some of the useful tweets from the day. (Twitter search only goes back about a month or so, so this link may not work after a certain date). Jim Richardson also created a permanent copy with the (new to me) webcitation website.

Conclusions

With such dodgy note taking I feel some concise summary is in order!

• Mandates are happening, by Universities and by Funders.
• HEFCE want research to be accessible to as many as possible as quickly as possible.  Coming from HEFCE, this holds a lot of weight.
• Funders (Research Councils / Wellcome) put mandates in place several years a go. They have not sat back and said ‘job done’. They are building on this foundation. How can they check? How can they enforce/encourage? How can they assist? How can they automate? How can they work with publishers and HE to share this information? Expect more to come in this area.
• Wellcome Trust prefers submission to Open Access Journals rather than author depositing in to a repository at a later date.
• HE Mandates are coming, we alreay have a few in the UK. Making them an intergral part of an academic’s review seems like a good idea. My opinion is that this is reasonable – even if there are those who disagree – surely an employer can (and does in every other sector) ask for a record of what an employee has been working on, and a copy of the end output, i.e. the full text in an IR.
• The report ‘Economic implications of alternative scholarly publishing models : exploring the costs and benefits. JISC EI-ASPM Project‘ is a thourough comprehensive look at the economic costs of Open Access and new forms of Scholorarly Communications.
• I think we are starting to see the larger Universities developing sophisticated network of systems to manage research/publications/OA/research-funding. See slide 10 of Paul Ayris presentation, and this article about Imperial’s setup as two examples.
• It makes sense to share information (between IT systems) between funders, HE and publishers. Examples: Funders sharing (bibliographic) information to a University about publications from its researchers, Universities (or publishers) passing information to funders linking publications to funding (or even the other way round?).
• This is an area which is still developing, fast, and will of course involve a culture change. Publishers seem unsure how to handle this new world.

Related posts:

1. The Data Imperative: Libraries and Research Data : comment
2. Event: The Data Imperative: Libraries and Research Data

There’s a lot of talk in the Library world about ‘next generation catalogues’, library search tools and ‘discovery’. There’s good reason for this talk, in this domain the world has been turned on its head.

History in a nutshell:

• The card catalogue became the online catalogue, the online catalogue let users search for physical items within the Library.
• Journals became online journals. Libraries needed to let users find the online journals they subscribed to through various large and small publishers and hosts. They built simple in-house databases (containing journal titles and links to their homepages), or added them to the catalogue, or used a third party web based tool. As the number of e-journals grew, most ended up using the last option, a third party tool (which could offer other services, such as link resolving, and do much of the heavy lifting with managing a knowledge base).
• But users wanted one place to search. Quite understandable. If you are after a journal, why should you look in one place to see if there is a physical copy, and another place if they had access to it online. Same with books/ebooks.
• So libraries started to try and find ways to add online content to catalogue systems in bulk (which weren’t really designed for this).

The online catalogues (OPAC) were simple web interfaces supplied with the much larger Library management system (ILS or LMS) which ran the backend the public never saw. These were nearly always slow, ugly, unloved and not very useful.

A couple of years a go(ish), we saw the birth of the next generation catalogue, or search and discovery tools. I could list them, but the Disruptive Technology Library Jester does an excellent job here. I strongly suggest you take a look.

Personally, I think I first heard about Aquabrowser. At the time a new OPAC which was miles ahead of those supplied with Library systems and was (I think) unique as a web catalogue interface not associated with a particular system, and shock, not from an established Library Company. The second system I heard about was probably Primo from Ex Libris. At first not understanding what it was: It sounds like Metalib (another product from the same company which cross-searches various e-resource), is Primo replacing it? Or replacing the OPAC? It took a while to appreciate that this was something that sat on top of the rest. From then, VuFind, LibraryFind and more.

While some where traditional commercial products (Primo, Encore, Aquabrowser), many more were open source solutions, a number of which developed at American Libraries. Often built on common (and modern) technology stacks such as Apache solr/Lucene, Drupal, php/java, mysql/postgres etc.

In the last year or so a number of major Libraries have started to use one of these ‘Discovery Systems’ for example: the BL and Oxford using Primo, National Libraries of Scotland & Wales and Harvard have purchased Aquabrowser and the LSE is trying VuFind. At Sussex (where I work) we have purchased and implemented Aquabrowser. We’ve added data enrichments such as table of contents (searchable and visible on records), book covers and the ability to tag and review items (tag/reviewing has been removed for various reasons) .

It would be a mistake to put all of these in to one basket. Some focus on being a OPAC replacement, others on being a unified search tool, searching both local and online items. Some focus on social tools, tagging & reviewing. Some work out the box others are just a set of components which a Library can sow together, and some are ‘SaaS’.

It’s an area that is fast changing. Just recently an established Library web app Company announced a forthcoming product called ‘Summon’, which takes searching a library’s online content a step further.

So what do libraries go for, it’s not just potentially backing the wrong horse, but backing the wrong horse when everyone one else had moved on to dog racing!

And within all this it is important to remember ‘what do users actually want’. From the conversations and articles I’ve read, they want a Google search box, but one which returns results from trusted sources and academic content. Whether they are looking for a specific book, specific journal, a reference/citation, or one/many keywords. And not just one which searches the metadata, but one which brings back results based on the full text of items as well. There are some that worry that too many results are confusing. As Google proves, an intelligent ranking system makes the number of results irrelevant.

Setting up (and even reviewing) most of these systems take time, and if users start to add data (tags, reviews) to one system, then changing could cause problems (so should we be using third party tag/rating/review systems?).

You may be interested in some other articles I’ve written around this:

• Library catalogues, search systems and data – discusses the issues in putting online journal/item records on to the catalogue, and therefore why a ‘discovery systems’ can resolve these.
• Academic discovery and library catalogues – Discusses the eXtensible Catalogue, and also links to an Academics article about how they conduct research
• JISC Library Management System Review – Summarises a recent JISC report on Library Management systems
• Free e-books online via University of Pittsburgh Press – from a couple of days a go, talks about the issues of Discovery related to free academic items (books) on the web, and ensuring Library Discovery Tools include them.

There’s a lot talk about discovery tools, but what sort to go for, who to back? And many issues have yet to be resolved. I’m come on to those next…

Related posts:

1. Library Catalogues need to cater for light-weight discovery clients
2. Library catalogues, search systems and data
3. Academic discovery and library catalogues

The University of Pittsburgh Press has put nearly 500 out of print books online and Open Access. You can access them via their Digital Editions website.  This is excellent news, making work which could be lost openly available to all.

University of Pittsburgh Press Digital Editions - Open Access free ebooks

For years there has been a movement towards making Journal articles Open Access, i.e. publicly available. However some subjects (especially in the Humanities) publish much of their research in books, not journals. Letting the world gain from the (normally publicly funded) research contained within books is more complex, and it’s not an area I fully understand. The author normally receives royalties from book sales. However I understand this are normally very small 99% of the time, and normally tail down to tiny amounts after a few years. What if funders and Universities demanded that any book written with their money (or during their employment) must be made publicly available after x number of years (let’s say 10 years)? Academics and Publishers would not welcome the move, but would still allow a window where they can gain revenue, and if this became the norm it would be something they just have to accept. Meanwhile, once open access, the book becomes much easier to archive and preserve, and ensure the knowledge is available to all in the long term. Just a thought.

Back to the Pittsburgh Press website (created with the University Library), the Copyright remains with the Press and I note this somewhat restrictive quote at the bottom of their homepage: “This material is provided for scholarly, educational, and research use only.” In an ideal world they would be under a more liberal licence (creative commons for example), though I imagine this would lead to many legal complexities to resolve, and don’t want this to detract from the good work that has been done here. It’s a massive step to release these so that they are publicly available.

It occurs to me that this raises the question of ‘discovery’. Not as a criticism of this service, but as an example of the problems we face in fully utilising its existence.

How will our (a University Library) users find these books? I can think of two  situations: first, they are looking for a particular book (perhaps due a reference) and discovering the online version will be much quicker than trying to find any existing print copy in a Library around the world, second: when a book contained in this service will be relevant to their research – and they are searching (catalogues/Google) using keywords to find relevant material.

Should users find this when they search our catalogue, or when they search our online resources (using a federated search service)? With the next generation library search products appearing, should they be ‘discoverable’ within these systems, and if so, how?

The above focuses on how a library should add these items in to its systems, the other side of the coin is adding them to universal worldwide systems. I selected a particular book and searched Worldcat for the title, it shows the book twice in the search results, once for the paper version and once for the ebook at Pittsburgh, the latter linking to the free e-book. A Good start!

Searching Google for the same title, the Worldcat entry was the fifth link, so someone searching Google for this book would – if their attention span can extend to trying the fifth link – find the free online version via the Worldcat record. An even better situation would be for Google to link to the free book directly.

Some thoughts:

• While the above is promising it relies on a user searching for the title of the book, which doesn’t cater for those who don’t know it exists, yet it’s content would be useful for their research.
• Users are saying to us they want one place to search for sources of information and content useful for their work. They have to use many different systems and it can be confusing when to use which one, and each has their own interface and rules. The Library catalogue, VLE/e-learning platform, Web of Science, Wordcat, Scopus, Google, Federated (online resources) search, COPAC, etc. Which ones to use, and when? The ultimate aim of providing one search box is to highlight information (content) which could be useful – both that which the institution subscribes to, and that which is free. How will records of Open Access books such as these ‘get in to’ such a search index?  Each Library adding such content is not very efficient, better would be for it to be added in one place/database (along with similar items) which can be openly searched/harvested by other systems using an API. Libraries could then simply add such a service to their own search system to make the content available to users. DOAJ (a directory of Open Access Journals) answered a similar concern for Open Access Journals.

Content is useless unless people can discover it, and how will Libraries develop systems which help users to discover everything they can potentially access in such an easy way that it is intuitive and requires no specialist training?

Related posts:

1. Google Books API