miscellaneous factZ – The online home of Rufus Pollock

As announced on Friday on the UK Government’s data.gov.uk, I am one of the members of the UK Government’s newly formed Public Sector Transparency Board.

From the announcement:

The Public Sector Transparency Board, which was established by the Prime Minister, met yesterday for the first time.

The Board will drive forward the Government’s transparency agenda, making it a core part of all government business and ensuring that all Whitehall departments meet the new tight deadlines set for releasing key public datasets. In addition, it is responsible for setting open data standards across the whole public sector, listening to what the public wants and then driving through the opening up of the most needed data sets.

Chaired by Francis Maude, the Minister for the Cabinet Office, the other members of the Transparency Board are Sir Tim Berners-Lee, inventor of the World Wide Web, Professor Nigel Shadbolt from Southampton University, an expert on open data, Tom Steinberg, founder of mySociety, and Dr Rufus Pollock from Cambridge University, an economist who helped found the Open Knowledge Foundation.

In the words of Francis Maude:

“In just a few weeks this Government has published a whole range of data sets that have never been available to the public before. But we don’t want this to be about a few releases, we want transparency to become an absolutely core part of every bit of government business. That is why we have asked some of the country’s and the world’s greatest experts in this field to help us take this work forward quickly here in central government and across the whole of the public sector.”

I’m one of the co-organizers of a workshop on Public Domain Calculators workshop taking place next week, on the 10th and 11th of November, at Emmanuel College, University of Cambridge.

Hosted by the Open Knowledge Foundation in association with the Centre for Intellectual Property and Information Law at the University of Cambridge, it’s a meeting of European experts on copyright and the digital public domain taking place as part of the Communia project.

The purpose of the workshop is to produce materials such as legal flow charts and public domain “algorithms” which will help with the representation of different national copyright laws and the determination of public domain status.

Details of the meeting are as follows:

• When: 10-11th November 2009
• Where: Emmanuel College, Cambridge
• Wiki: http://wiki.okfn.org/PublicDomainCalculators/Meeting
• Participate: Free but space is limited. If you are interested in coming, email the organizers at: info@okfn.org

Background

There is often a tendency to talk of ‘the public domain’ and of works falling out of copyright and ‘into the public domain’ – as though there is a single set of works which are out of copyright all over the world. In fact, of course, there are different national laws about the nature and duration of copyright in different types of works – and hence what is in the public domain is different in different countries.

Efforts are currently underway to build a series of public domain calculators – which will help to determine whether or not a given work is in copyright in a given jurisdiction. At the time of writing groups and individuals in more than 17 jurisdictions are assisting in this effort.

Current methods of disseminating scholarly information focus on the use of journals who retain exclusive rights in the material they publish. Recently there has been increasing dissatisfaction with this model, with suggestions for alternative approaches such as “Open Access”.

Together with a colleague (Omar Al-Ubaydli) I’ve been working to explore the reasons for the development of the traditional journal model, why it is no longer efficient and how it could be improved upon. We’re particularly interested in going beyond the basic question of distribution (access) to that of filtering, i.e. the process of matching information with the scholars who want it.

With the volume of information production ever growing — and attention ever more scarce — filtering is becoming crucial. Digital technology offers us some radically new possibilities. In particular, distribution and filtering can be separated, in turn, allowing filtering to be decentralized and distributed — a model which promises dramatic increases in transparency, innovation and efficiency.

Below is an overview of our analysis with the full version of the current paper here: http://rufuspollock.org/economics/papers/scholars_and_journals.pdf

Overview

It is crucial to the progress of any domain of scholarship that those engaged therein are able to communicate their discoveries and activities to others. As such a variety of systems and institutions have been developed in order to support ’scholarly communication’ in one form or another ranging from personal letters to physical meetings. In recent times, the growth of scholarship, combined with its increasing geographical dispersion, have resulted in the centrality of the written word and its dissemination via ‘journals’. In this paper we consider the purposes of any system of scholarly communication and consider the current academic journal system in light of them. This examination highlights several deficiencies and also suggest various possible improvements.

When thinking about the possible mechanisms of scholarly communication it is useful to specify in more detail the criteria against which they should be measured. That is, to put it more succinctly, what do we want a good mechanism for scholarly communication to do? In particular, when we say communicate we must ask ourselves what, to whom, in what form, etc etc. For it is clear that when we talk of communication we usually mean more than the simple transmission of a piece of information. In fact, today, with so much scholarship available, the challenge may often not lie in the transmission from the author to the reader but in the matching of authors and readers — the decision of ‘what to read’. This growing focus on choice is a natural one in a world where time and attention are limited and the amount of scholarship available is ever increasing. As such it suggests that there are at least two distinct functions performed by a system of scholarly communication:

• Distribution — getting information from authors to readers (and back again)
• Selection (filtering) — deciding what to distribute and to whom

In appreciating this distinction it is illuminating to consider how practice has changed over time. Originally communication between scholars, at least in written form, primarily took the form of letters between the individuals involved. As such, the two activities of distribution and filtering would be almost completely identical. Then, as the number of authors and readers grew this became infeasible and dedicated journals would be created which would then disseminate to their particular readers a selection of what was submitted to them. Thus, what was once a direct peer-to-peer relationship became mediated by a new institutional form: the academic journal — though of course journals were often run by the very readers and authors who used them. Finally, today, thanks to digitization and the Internet peer-to-peer is once again a possibility though with important differences: unlike in the past, where a letter writer chooses the recipient, the modern peer-to-peer approach more resembles journals in that the author and reader act independently — the author uploads or publishes his/her work to a repository entirely separately from the reader finding, downloading and reading it. This last discussion suggests breaking down our original two categories a little further:

• ‘Making available’ — publishing material
• Discovery — finding out what is available
• Choice — choosing from what is available
• Reading — getting access to the material (in the form required)

Here, the first and fourth item would come under the ‘distribution’ heading while the second and third would come under ’selection’. In addition we should mention two other functions performed by such a system, both of which relate to selection: a) improvement of work via peer-review (distinct from filtering process itself); b) ‘quality signalling’ whereby the selection of work helps signal the quality of its creators which in turn is important for the purpose of resource allocation (jobs, grants etc) within the scholarly community.

With these added to the list we now have a good number of separate goals which a scholarly communication mechanism may seek to satisfy. The next stage is to consider how the current system, largely based on academic journals, fares in respect of them.

Goals, Instruments and the Current Journal System

It is well known that in order to fully address a given number of (independent) goals one needs an equal number of instruments. For example, if one is seeking to address both congestion and pollution in relation to road-traffic, a single instrument such as petrol taxes, will be insufficient.

Here too there are multiple independent goals, most notably distribution and selection (matching). These are clearly distinct goals and require distinct instruments for their achievement but journals are but a single instrument which combine distribution and filtering in one mechanism.

Originally, the restrictions of reproduction and distribution technologies, meant they were the best instrument available. Today, with the advent of the computer and the Internet, this is no longer true: distribution (the uploading and downloading) can be done by almost anyone and quite separately from recommendations and rating of that material.

As such, the traditional journal system is becoming a serious constraint, particularly in its closed access form. There are two distinct aspects of this constraint. First, on the distribution side, journals delay and restrict access as a result of higher prices arising either from simple monopoly control or the costs of the (inefficient) selection mechanism the traditional model necessitates. Second, on the selection side, the forced combination of selection and distribution and the associated monopoly control of content greatly limit the efficiency (and utility) of the selection and filtering processes used to match authors and readers together.

Unfortunately, the two-sided nature of the journal market (based on expectations), combined with the current evaluation structure of academia, continue to lock society into this inefficient restriction. Open-access journals provides are an important part of improving the current situation. However, as we discuss below, they are only a first step: in order to reap the full benefits of new technology we must move away from the traditional ‘journal’ model to a system that allow for full separation between the distribution and selection operations.

The Technological Origins of Modern Inefficiency

At this point it is worth considering in a little more detail why restricted-access journals originally came about. The answer lies in the nature of the technology available in earlier periods to manage distribution (printing and transmission). When many journals were originally started the cost of transmitting information was very high and journals acted as a club good by which the costs of reproduction and distribution could be (efficiently) shared (the efficiency arising here from economies of scale).

At the same time, given the limited ‘bandwidth’ it was natural for journals to take on some filtering role in order to economize on the scarce distribution capacity. In this situation, dissemination is limited and with only one instrument available (journals), it is natural to tie dissemination and filtering together (with filtering in many ways secondary). Once filtering is being done it is natural for journals to ‘tie’ material to the journal explicitly via copyright — though at an early stage given the scale economies of journals this explicit tying was not actually necessary and was probably done for simple legal convenience.

With the advent of digital communications, in particular the Internet, bandwidth is no longer scarce. What is now scarce is attention. In this setup the importance of a journal is not its role in efficiently sharing reproduction and distribution costs but its role as a filtering mechanism. However, there is now a problem: when distribution is central it is natural to ‘add-in’ filtering, it is not natural, or necessary, to tie distribution to filtering when filtering is central. In fact it seems clear that distribution and filtering can be done entirely separately (there are potentially lots of ways for you to download my paper quite separate from getting it from a journal — and lots of ways to do matching and filtering other than by journal editors and reviewers). The Open Access movement can be seen as largely about achieving this separation: with open access there is no longer a connection between access/distribution (which would be free) and the filtering mechanism (the choice of which articles go in a particular journal).

That said the ‘Open Access’ movement still has a large focus on journals — albeit open-access ones. This, in our view, is a mistake. Technology has also affected possibilities for filtering. In particular it is no longer clear why the centralized mechanism of official peer-review and journals is superior to alternative decentralized options. The last decade, has witnessed widespread, and often successful, experimentation with distributed voting and evaluation mechanisms (for example Slashdot’s story-ratings and Google’s link-based site rankings).

Thus, to be more radical, it makes sense not only to remove centralized control of distribution but also centralized control of filtering. A more distributed (market-like?) filtering mechanism would permit the same freedom (and same status) for reviewing and recommendation as it does in the production of scholarly information. At the same time it would deliver greater transparency and, by permitting ‘free-entry’ in filtering, would permit greater specialization, greater diversity, increased participation and the increasing efficiency flowing from greater competition.

As such, the gains from going ‘open’ are not simply wider access, but a reduction in the time and energy scholars spend finding and processing research information. Significantly, this second item, which is less frequently mentioned in discussions of ‘Open Access‘, may well be the most significant.

The Open Knowledge Foundation’s 2009 Open Knowledge Conference (OKCon), which I help organize, will take place next Saturday 28th March – less than a week away.

Full details including programme can be found either in this blog post or on the OKCon home page.

As usual this will be a fun and informal day so if you’re free this Saturday and interested in “Open” stuff come along to UCL and take part.

I should also add that for the two days before (Thursday + Friday) there is also the 5th COMMUNIA Workshop which is about Accessing, Using, Reusing Public Sector Content and Data which is being co-organized by the Open Knowledge Foundation together with the London School of Economics and taking place at LSE (all thanks to the tireless work of Jonathan Gray and Prodromos Tsiavos!).

Over the Christmas break I had a chance to make some substantial improvements/additions to our Open Economics including:

1. Improved javascript graphing.
2. Extend Millenium Development Goals package and added web interface.
3. First efforts at ‘Where Does My Money Go’
   • Aim: Dig up govt finance info and visualize the results (online)
   • http://okfn.org/wiki/projects/Where_Does_My_Money_Go

More details on each of these can be found below. Also we’d be delighted to here from anyone interested in getting involved in this, especially with the last item, so if interested do get in touch.

1. Updated javascript graphing package to use flot.

This also allows us to use javascript make the graphing stuff more interactive, in particular to select chart type and the series to plot. See e.g. the data on Daily Wages of Thatchers in the Middle Ages or Wheat, barley, oat, mutton and wool prices, and agricultural wages, 1500-1849.

2. Improved Millenium Development Goals package/dataset and added a web interface.

• MDG entry in the store
• Source for MDG package: http://knowledgeforge.net/econ/svn/trunk/econdata/mdg/

Extended ‘packagization’ of the MDG data by creating a mini-domain model and an associated sql version of data in addition to the existing csv normalized-tabular version of the data:

http://knowledgeforge.net/econ/svn/trunk/econdata/mdg/db.py

This is much more convenient for analysis (e.g. finding all countries which have at least one entry for any of these 3 series between 1995 and 2005 …). It is also essential for:

New web interface for Millenium Development Goals

Using the sql version of the data is was easy to build a quick-and-dirty web interface to enables one to browse and view the data quickly:

http://www.openeconomics.net/mdg/

For example here’s chart and data showing “Children under 5 moderately or severely underweight, percentage” for Afghanistan, China, India, United States:

http://www.openeconomics.net/mdg/view?commit=Show+Values&series=559&countries=4&countries=156&countries=356&countries=840

3. First efforts at ‘Where Does My Money Go’

Two parts to this project a) getting the data on government revenue/expenditure b) displaying it nicely in a web interface.

Part (a) is encapsulated in a new ukgovfinances dataset:

http://knowledgeforge.net/econ/svn/trunk/econdata/ukgovfinances/

Using this data we have made a (small) start on the web interface:

http://www.openeconomics.net/wdmmg/

The Open Knowledge Foundation (which I’m involved in) is co-organizing with MySociety and OPSI, a Workshop on Finding and Re-using Public (Sector) Information.

The event takes place this Saturday (1st of November) at the London Knowledge Lab near Holborn in London. Full details in this OKFN blog post and you can sign up the wiki page:

http://okfn.org/wiki/PublicInformation