miscellaneous factZ – The online home of Rufus Pollock

Last week I was in Oxford to give a talk at the IP Seminar on “How Long Should Copyright Last?”. I have now posted the slides from the talk online.

In addition to covering the basic outline of the optimal term calculation, I was also able to give some results from the recent research on the public domain (see slide 58 onwards).

The other day I posted up some work-in-progress on the subject of patterns of knowledge production.

That material is still in a fairly preliminary state. However, my decision to release it it in this form was a conscious decision and part of an ongoing attempt on my part to practice a more open “release early, release often” approach to research.

In doing this I’m drawing direct inspiration from the open source and open notebook (science) communities and seeking to engage in what might be termed open notebook social science!

I think most researchers (including myself) feel a reluctance to put out material that isn’t at a reasonable level of maturity. While there are some good reasons for this, I think the main motivations are less positive, and are primarily to do with fear: be it of criticism or that your ideas are “taken” by others. While such fears can have some basis, it seems to me the benefits of an open approach — in terms of visibility, dissemination, and potential for collaboration — significantly outweigh any of the associated risks.

Over the last year, I’ve already been making some effort to move in this direction but from this point on I’m aiming to do this more thoroughly and methodically. A first step in this will be to put all the “patterns” and data online.

I’m posting up some work-in-progress entitled Exploring Patterns of Knowledge Production (link to full pdf). Below I’ve excerpted the introduction plus list of motivational questions. Comments (and critique) very welcome!

Introduction

In what follows the term ‘knowledge’ is here used broadly to signify all forms of information production including those involved in technological innovation, cultural creativity and academic advance.

Today, thanks to rapid advances in IT, we have available substantial datasets pertaining both to the extent and the structure of knowledge production across disciplines, space and time.

Especially recent is the availability of good ’structural’ data — that is data on the linkages and relationships of different pieces of knowledge, for example as provided by citation information. This new material allows us to explore the “patterns of knowledge production” in deeper and richer ways than ever previously possible and often using entirely new methods.

For example, it has long been accepted that innovation and creativity are cumulative processes, in which new ideas build upon old. However, other than anecdotal and case-study material provided by historians of ideas and sociologists of science there has been little data with which to study this issue — and almost none of a comprehensive kind that would make possible a systematic examination.

However, the recent availability of comprehensive databases containing ‘citation’ information have allowed us to begin really examining the extent to which new work builds upon old — be it a new technology as represented by a patent or a new idea in academia as represented by a paper, builds upon old.

Similar opportunities present themselves in relation to identifying the creation of new fields of research or technology, and tracing their evolution over time. Here the existence of extensive “structural information” as presented, for example, by citation databases, enables new systematic approaches — for example, can new fields be identified (or perhaps defined) as points in ‘knowledge space’ far away from the existing loci of effort? or, alternatively, by the nature of its connections to the existing body of work?

Structural information of this kind can also be used in charting other changes in the life-cycle of knowledge creation. For example, to offer a specific conjecture, a field entering decline, though still exhibiting a similar level of output (papers etc) and even citations to a field in rude health, may display a citation structure which is markedly different — for example, more clustered within the field itself. Thus, by using this additional structural information we may be able to gain insights not available with simpler approaches.

At the same time, structure must also play a central role in any attempt to estimate knowledge related ‘output’ measures. This is of course not true for other forms of ‘output’, for example that of corn of steel, where we have relatively well-defined objective measures available: tonnes of such-and-such a quality.

But knowledge is different: the most obvious metrics, such as number of patents or papers produced, seem entirely inadequate: one particular innovation or paper may be ‘worth’ as much as a hundred or a thousand others.

The issue here is that, compared to corn or steel, knowledge is extremely inhomogeneous, or put slightly differently, quality (or significance) differs very substantially across the individual pieces of knowledge (papers, patents etc).

Thus, any serious attempt to measure the progress of knowledge must must find some way to do this quality-adjustment and structural information seems essential to this.

What specific questions might we explore with such datasets?

The following is a (non-exhaustive) list of the kinds of questions one might explore using these new datasets:

• Can we use structure to infer information about quality of individual items? Clearly the answer is yes, for example by using a citation-based metric where a work’s value is estimated based on its citation by others.
• Can we then use this information together with more global structure of the production network to gain a better idea of total (quality-adjusted) output. This would allow one to chart progress, or the lack of it, over time?
• Can we use structural information to investigate the life-cycle of fields? For example, can we see fields ‘dying out’ or the onset of diminishing returns? Can we see new fields coming into existence and their initial growth patterns?
• What about productivity per capita and its variation across the population? It is likely that one would need to focus here within a discipline as it would be difficult to directly compare across disciplines, at least when using quality adjusted productivity.
• Do the structures of knowledge production vary over time and across disciplines and does this have implications for their productivity? Can we compare the structure of evolution in technology or economics with that in ‘natural’ evolution and, if not, what are the primary differences?
• How do other (observable) attributes related to the producers of knowledge (their collaboration with others, their geographical location) affect the structures we observe and the associated outcomes (output, productivity) already discussed above?
• Do different policies (for example openness vs. closedness — weak vs. strong IP) have implications for the structure of production and hence for output and productivity?
• Is knowledge production (in a particular area) ergodic or path-dependent? Crudely: do we always end up in the same place or do small shocks have large long-term effects?

Last week I was at the ATRIP Conference to give an invited talk on “How Long Should Copyright Last?”, based on my paper: Forever Minus a Day? Calculating the Optimal Term of Copyright.

Slide are here, and you can find the text of the accompanying introduction below (I plan to write up the full exposition as a short essay — but that is to come).

Most ATRIP participants were lawyers not economists, so this was an opportunity to do a more non-technical presentation (so no equations!). As with most economics, the fundamentals of calculating copyright term are simple: it is just a demand curve plus “welfare analysis” (a fancy name for adding up social benefits and costs), and shorn of “obfuscating” algebra these matters should be understandable by anyone.

How Long Should Copyright Last: Introduction

Before I begin it is important to note that in considering copyright and its term, we must leave to one side the questions of attribution and integrity — their existence and term can and should be considered quite separately from the ‘economic’ rights that form the core of copyright as it operates today.

This small caveat done, I beg your indulgence for a brief historical excursion. In particular, I ask you to cast your mind back a century and a half and more to the Houses of Parliament in the February of 1841.

[[Picture of Serjeant Talfourd]]

As many of you will be aware Serjeant Talfourd had, by this point, been doggedly pursuing a new copyright act for four years — since 1837. Originally wide in scope the Bill had been narrowed and the attention of both supporters and critics alike had come to focus on a single feature of that Act: the proposed extension in the term of protection. Specifically Talfourd’s Act proposed changing the then rule of 28 years or life (whichever being the longer) to life plus sixty — remarkably close to the life plus 70 of today.

[[Picture of Macaulay]]

By February 1841 Talfourd’s Bill had failed no less than 4 times. On its fifth attempt it had reached a second reading and on the fifth of February it came before the House. After a brief introduction by Talfourd — mindful that this was not the first time the matter had been discussed — Thomas Babbington Macaulay rose to speak. In a masterly disquisition, both in content and rhetoric, Macauley set out his opposition to the Bill, and did so so tellingly that the motion was defeated. Talfourd, who lost his seat at the next election, and therefore only saw his Bill pass in the hands of another — and in much reduced form — remained forever embittered by Macaulay’s intervention — coming so late and so decisively in the process.

To read Macaulay’s speech, and, for that matter, the views expressed on all sides in that debate, is to be struck by how little has changed.

[[Valenti Picture]]

When Jack Valenti and Mary Bono are found in recent times calling for a term of ‘Forever Minus a Day’ one hears the echoes of Serjeant Talfourd all those years ago, just as one can hear echoes of those who oppose extensions today of the likes of Henry Warburton, a radical politician and vehement opponent of Talfourd, who claimed the extension was “a robbery upon the public” and that copyright ought to be fixed, “only on such a term of years as would prove a sufficient inducement for authors to write good books”.

And the analogy is telling in other ways. Though Talfourd’s Bill was beaten back by a swell of opposition year after year eventually it was passed — albeit in reduced form and by Lord Mahon — with this success attributable to a persistence made possible not, primarily, by the size, but by the concentration of the interests who sought its passage. Like Fabius Cunctator the proponents of extension, sustained by deep reservoirs of emotional and financial commitment, can afford to wait, able to return, as necessary, again and again, until an opportune moment presents itself for the attainment of their purposes — for the opposition to extension, though broad is ’shallow’ and therefore more easily dissipated by distraction and division.

[[Philosophical Differences]]

Even more striking are the similarity in the issues that occupy centre stage in this debate. First, the fundamental ‘philosophical’ question — which colours all of discussion — of whether we confer copyright because it is a natural right — which should therefore last forever — or for ‘utilitarian’ purposes, that is the public good — in which case it almost certainly should not. Second, descending from these lofty heights of principle, what is the actual effect is copyright? In particular, does it operate to raise price and restrict access — that is: is it a monopoly?; and what specifically are the benefits that accrue to the producers of copyrightable works, and what costs to the public and others who wish to use and reuse them.

I think it is clear that economists — or any group for that matter — have no great claim to authority on answering this first question of principle, for it seems, ultimately, one of opinion. That said, I would note two points which must raise grave doubts as to the existence of any fundamental natural right from which copyright might spring.

First, term limited in all jurisdictions. Second, the breadth of copyright’s application both in subject matter, quality and ownership. For can we truly convince ourselves that “eternal expressions of the human spirit”, worthy of exclusivity for all time, subsist in an advert for toothpaste; or convince ourselves of the special status of the creator when so much copyright today, perhaps even the majority, is immediately, and indeed often automatically, assigned from the ‘creator’ to a corporation.

However, it is not my intention to enter into this debate any further here. Rather, in the interests of ‘full disclosure’ I wish only to make clear my views — and those of economists generally — on the matter, namely that copyright is not a natural right but is created and maintained for the purpose of promoting and securing the public good, no more, no less. (These are views which can come as no surprise given the nature of this talk — an analysis of term only makes sense if its basis is a utilitarian one!)

[[My Views]]

Furthermore, let me also make clear, right at the outset, my view, and one again I think shared by almost all economists, that copyright is a monopoly. This is not to say that copyright is bad — far from it. But to deny that copyright is a monopoly is to obscure its basic nature and operation — an obscuration that has, furthermore and unfortunately, been most common and attractive to those pursuing copyright’s enlargement.

[[The Big M Word]]

And what is the general tendency of monopoly — to echo Macaulay once again? It is indeed to raise prices and limit access. Now, of course, we may debate the precise extent of these effects, but there can be no denying that the very purpose of copyright’s existence is to confer on a single entity — the copyright holder — the power to control the dissemination, and hence the price, of all instances of a particular good — i.e. all copies of a given work.

This is the very definition of a monopoly and the fact that there may exist other goods, other works, which compete with that one makes no difference — a monopoly of apples is no less a monopoly because one does not control oranges. Of course, the existence and proximity of substitutes will alter the affect of the monopoly, but one must be cautious here: close substitutes may limit the negative effects of the copyright monopoly but they will, for the very same reasons, also limit the gains (those increased revenues for copyright-holder).

Returning then to Macaulay whose expression of the matter I cannot better:

[[Macaulay Again]]

“It is good that authors be remunerated; and the least exceptionable way of remunerating them is by a monopoly. Yet monopoly is evil. For the sake of the good we must submit to the evil, but the evil ought not to last a day longer than is necessary for securing the good.”

Our task then is to answer the implicit question: how long should copyright last (so as to not be a day longer than is needed)? More specifically what are the degrees of benefit and harm created by copyright’s monopoly and at what level should term be set to achieve the most advantageous balance of the two?

Just came across this, now year-old, story of Qualcomm’s submarining efforts (and subsequent cover-up) in relation to the JVT (joint video team) standardization effort for H.264/MPEG-4 AVC codec.

Qualcomm participated in the JVT (joint video team) standardization effort for H.264/MPEG-4 AVC (a video codec) which was concluded in 2003. However, they (intentionally) did not declare relevant patents (though obliged to do so) and subsequently sued Broadcom for infringement of these patents in 2005. At the resulting trial they then concealed extensive documentary material relating to their involvement in the standards process. However this fraud was discovered (in dramatic fashion) and they lost their suit with their patents being voided — though only in relation to the standard. From the Court of Appeals Judgement (United States Court of Appeals for the Federal Circuit Judgement 2007-1545, 2008-1162 pp.3-4, emphasis added):

Plaintiff Qualcomm is a member of the American National Standards Institute (”ANSI”), which is the United States representative member body in the ISO/IEC, and was an active dues-paying member for many years prior to 2001. It is also a member of the ITU-T and a participant in the JVT. Qualcomm did not disclose the ‘104 and ‘767 Patents to the JVT prior to release of the H.264 standard in May 2003.

On October 14, 2005, Qualcomm filed the present lawsuit against Broadcom in the United States District Court for the Southern District of California, claiming that Broadcom infringed the ‘104 and ‘767 Patents by making products compliant with the H.264 video compression standard. A jury trial was held from January 9, 2007, to January 26, 2007. The jury returned a unanimous verdict as to non-infringement and validity, finding that (1) Broadcom does not infringe the ‘104 and ‘767 Patents; and (2) the ‘104 and ‘767 Patents were not shown to be invalid. The jury also returned a unanimous advisory verdict as to the equitable issues, finding by clear and convincing evidence that (1) the ‘104 Patent is unenforceable due to inequitable conduct; and (2) the ‘104 and ‘767 Patents are unenforceable due to waiver.

On March 21, 2007, the district court entered an order (1) finding in favor of Qualcomm and against Broadcom on Broadcom’s counterclaim of inequitable conduct as to the ‘104 Patent; (2) finding in favor of Broadcom and against Qualcomm on Broadcom’s affirmative defense of waiver as to the ‘104 and ‘767 Patents; and (3) setting a hearing on an Order to Show Cause as to the appropriate remedy for Qualcomm’s waiver. The district court’s conclusion that Qualcomm waived its rights to assert the ‘104 and ‘767 Patents was based on Qualcomm’s conduct before the JVT.

Throughout discovery, motions practice, trial, and even post-trial, Qualcomm adamantly maintained that it did not participate in the JVT during development of the H.264 standard. Despite numerous requests for production and interrogatories requesting documents relating to Qualcomm’s JVT participation prior to adoption of the H.264 standard, Qualcomm repeatedly represented to the court that it had no such documents or emails. On January 24, 2007, however, one of the last days of trial, a Qualcomm witness testified that she had emails that Qualcomm previously claimed did not exist. Later that day, Qualcomm produced twenty-one emails belonging to that witness. As the district court later discovered, these emails were just the “tip of the iceberg,” as over two hundred thousand more pages of emails and electronic documents were produced post-trial. Remedy Order at 1245. The district court later determined that these documents and emails “indisputably demonstrate that Qualcomm participated in the JVT from as early as January 2002, that Qualcomm witnesses . . . and other engineers were all aware of and a part of this participation, and that Qualcomm knowingly attempted in trial to continue the concealment of evidence.”

There’s an interesting 6 month fellowship at OPSI for work on economics of public sector information being funded by ESRC and National Archives. Deadline for applications is 6th August:

Valuing information: an economic analysis of public sector information and its re-use

Length of Fellowship: Six months

Proposed start date: Autumn 2009

Applications to be submitted as soon as possible (and by 6 August)

Location of Fellowship: The National Archives’ sites (Central London and Kew)

As part of its Placement Fellowship Scheme, the Economic and Social Research Council (ESRC) and The National Archives welcome applications from academic economists interested in working in a research capacity in the Office of Public Sector Information (OPSI). OPSI is part of The National Archives, a member of the Ministry of Justice family, working to set standards, deliver access and encourage the re-use of PSI.

The Placement Fellowship Scheme encourages social science researchers to spend time within a partner organisation to undertake policy relevant research and to develop the research skills of partner employees. The Fellowship will be jointly funded by the ESRC and OPSI while the Fellow remains employed by his or her institution.

See the document below for further details on the Placement Fellowship: http://www.nationalarchives.gov.uk/documents/esrc-placement-fellowship-june-09.pdf

Current methods of disseminating scholarly information focus on the use of journals who retain exclusive rights in the material they publish. Recently there has been increasing dissatisfaction with this model, with suggestions for alternative approaches such as “Open Access”.

Together with a colleague (Omar Al-Ubaydli) I’ve been working to explore the reasons for the development of the traditional journal model, why it is no longer efficient and how it could be improved upon. We’re particularly interested in going beyond the basic question of distribution (access) to that of filtering, i.e. the process of matching information with the scholars who want it.

With the volume of information production ever growing — and attention ever more scarce — filtering is becoming crucial. Digital technology offers us some radically new possibilities. In particular, distribution and filtering can be separated, in turn, allowing filtering to be decentralized and distributed — a model which promises dramatic increases in transparency, innovation and efficiency.

Below is an overview of our analysis with the full version of the current paper here: http://rufuspollock.org/economics/papers/scholars_and_journals.pdf

Overview

It is crucial to the progress of any domain of scholarship that those engaged therein are able to communicate their discoveries and activities to others. As such a variety of systems and institutions have been developed in order to support ’scholarly communication’ in one form or another ranging from personal letters to physical meetings. In recent times, the growth of scholarship, combined with its increasing geographical dispersion, have resulted in the centrality of the written word and its dissemination via ‘journals’. In this paper we consider the purposes of any system of scholarly communication and consider the current academic journal system in light of them. This examination highlights several deficiencies and also suggest various possible improvements.

When thinking about the possible mechanisms of scholarly communication it is useful to specify in more detail the criteria against which they should be measured. That is, to put it more succinctly, what do we want a good mechanism for scholarly communication to do? In particular, when we say communicate we must ask ourselves what, to whom, in what form, etc etc. For it is clear that when we talk of communication we usually mean more than the simple transmission of a piece of information. In fact, today, with so much scholarship available, the challenge may often not lie in the transmission from the author to the reader but in the matching of authors and readers — the decision of ‘what to read’. This growing focus on choice is a natural one in a world where time and attention are limited and the amount of scholarship available is ever increasing. As such it suggests that there are at least two distinct functions performed by a system of scholarly communication:

• Distribution — getting information from authors to readers (and back again)
• Selection (filtering) — deciding what to distribute and to whom

In appreciating this distinction it is illuminating to consider how practice has changed over time. Originally communication between scholars, at least in written form, primarily took the form of letters between the individuals involved. As such, the two activities of distribution and filtering would be almost completely identical. Then, as the number of authors and readers grew this became infeasible and dedicated journals would be created which would then disseminate to their particular readers a selection of what was submitted to them. Thus, what was once a direct peer-to-peer relationship became mediated by a new institutional form: the academic journal — though of course journals were often run by the very readers and authors who used them. Finally, today, thanks to digitization and the Internet peer-to-peer is once again a possibility though with important differences: unlike in the past, where a letter writer chooses the recipient, the modern peer-to-peer approach more resembles journals in that the author and reader act independently — the author uploads or publishes his/her work to a repository entirely separately from the reader finding, downloading and reading it. This last discussion suggests breaking down our original two categories a little further:

• ‘Making available’ — publishing material
• Discovery — finding out what is available
• Choice — choosing from what is available
• Reading — getting access to the material (in the form required)

Here, the first and fourth item would come under the ‘distribution’ heading while the second and third would come under ’selection’. In addition we should mention two other functions performed by such a system, both of which relate to selection: a) improvement of work via peer-review (distinct from filtering process itself); b) ‘quality signalling’ whereby the selection of work helps signal the quality of its creators which in turn is important for the purpose of resource allocation (jobs, grants etc) within the scholarly community.

With these added to the list we now have a good number of separate goals which a scholarly communication mechanism may seek to satisfy. The next stage is to consider how the current system, largely based on academic journals, fares in respect of them.

Goals, Instruments and the Current Journal System

It is well known that in order to fully address a given number of (independent) goals one needs an equal number of instruments. For example, if one is seeking to address both congestion and pollution in relation to road-traffic, a single instrument such as petrol taxes, will be insufficient.

Here too there are multiple independent goals, most notably distribution and selection (matching). These are clearly distinct goals and require distinct instruments for their achievement but journals are but a single instrument which combine distribution and filtering in one mechanism.

Originally, the restrictions of reproduction and distribution technologies, meant they were the best instrument available. Today, with the advent of the computer and the Internet, this is no longer true: distribution (the uploading and downloading) can be done by almost anyone and quite separately from recommendations and rating of that material.

As such, the traditional journal system is becoming a serious constraint, particularly in its closed access form. There are two distinct aspects of this constraint. First, on the distribution side, journals delay and restrict access as a result of higher prices arising either from simple monopoly control or the costs of the (inefficient) selection mechanism the traditional model necessitates. Second, on the selection side, the forced combination of selection and distribution and the associated monopoly control of content greatly limit the efficiency (and utility) of the selection and filtering processes used to match authors and readers together.

Unfortunately, the two-sided nature of the journal market (based on expectations), combined with the current evaluation structure of academia, continue to lock society into this inefficient restriction. Open-access journals provides are an important part of improving the current situation. However, as we discuss below, they are only a first step: in order to reap the full benefits of new technology we must move away from the traditional ‘journal’ model to a system that allow for full separation between the distribution and selection operations.

The Technological Origins of Modern Inefficiency

At this point it is worth considering in a little more detail why restricted-access journals originally came about. The answer lies in the nature of the technology available in earlier periods to manage distribution (printing and transmission). When many journals were originally started the cost of transmitting information was very high and journals acted as a club good by which the costs of reproduction and distribution could be (efficiently) shared (the efficiency arising here from economies of scale).

At the same time, given the limited ‘bandwidth’ it was natural for journals to take on some filtering role in order to economize on the scarce distribution capacity. In this situation, dissemination is limited and with only one instrument available (journals), it is natural to tie dissemination and filtering together (with filtering in many ways secondary). Once filtering is being done it is natural for journals to ‘tie’ material to the journal explicitly via copyright — though at an early stage given the scale economies of journals this explicit tying was not actually necessary and was probably done for simple legal convenience.

With the advent of digital communications, in particular the Internet, bandwidth is no longer scarce. What is now scarce is attention. In this setup the importance of a journal is not its role in efficiently sharing reproduction and distribution costs but its role as a filtering mechanism. However, there is now a problem: when distribution is central it is natural to ‘add-in’ filtering, it is not natural, or necessary, to tie distribution to filtering when filtering is central. In fact it seems clear that distribution and filtering can be done entirely separately (there are potentially lots of ways for you to download my paper quite separate from getting it from a journal — and lots of ways to do matching and filtering other than by journal editors and reviewers). The Open Access movement can be seen as largely about achieving this separation: with open access there is no longer a connection between access/distribution (which would be free) and the filtering mechanism (the choice of which articles go in a particular journal).

That said the ‘Open Access’ movement still has a large focus on journals — albeit open-access ones. This, in our view, is a mistake. Technology has also affected possibilities for filtering. In particular it is no longer clear why the centralized mechanism of official peer-review and journals is superior to alternative decentralized options. The last decade, has witnessed widespread, and often successful, experimentation with distributed voting and evaluation mechanisms (for example Slashdot’s story-ratings and Google’s link-based site rankings).

Thus, to be more radical, it makes sense not only to remove centralized control of distribution but also centralized control of filtering. A more distributed (market-like?) filtering mechanism would permit the same freedom (and same status) for reviewing and recommendation as it does in the production of scholarly information. At the same time it would deliver greater transparency and, by permitting ‘free-entry’ in filtering, would permit greater specialization, greater diversity, increased participation and the increasing efficiency flowing from greater competition.

As such, the gains from going ‘open’ are not simply wider access, but a reduction in the time and energy scholars spend finding and processing research information. Significantly, this second item, which is less frequently mentioned in discussions of ‘Open Access‘, may well be the most significant.

This follows up my previous post. Here we are going to calculation public domain numbers based directly on authorial birth/death date information rather than on guesstimated weightings. We’re going to focus on the Cambridge University Library (CUL) data we used previously.

Table 1: PD Relevant Information Availability

Table 1 presents a summary of how much relevant information is available for items (books) of particular vintages in the CUL catalogue — we only show data from 1870 to 1950 on the presumption that (almost) all pre-1870 publications are PD (their authors would have had to live for more than 70 years post-publication for this not to be the case) and almost all publications post 1950 are in copyright today (their authors would have to have died before 1940 for this not to be the case).

As the table shows, at best only just over 40% of items have a recorded authorial death date and extending to include birth dates only raises this proportion to, at best, the mid mid-to-low fifties. Taking account of items which lack any associated author, raises these figures somewhat further to around 60%, though we should note that the reason for the lack of an associated author is not clear — is it because they are genuinely anonymous or simply because the information has not been recorded? Thus, even for the earliest items listed a large proportion of items (50% or more) lack the necessary information for direct computation of public domain status.

At the same time, we can take some heart, and some interesting facts, from this table. First, a reasonable proportion, amounting to many thousands of items, did have associated death dates. Second, at least for older items, the majority of those with any date had a death date (95% for 1870-1880 and still at over 70% for 1920-1930). Third, and this is a more general observation, proportions were surprisingly constant over time. For example, the proportion of ‘anonymous’ items lies in a narrow band between 10% and 13% for the entire periods. Similarly the proportion of items with any date information ranged only from 45% to 56%. At the same time, and reassuringly, though the proportion with death dates is relatively constant for the oldest periods, in the more recent ones it falls substantially; as one would expect given that some of the authors from those more recent eras are still alive.

Table 2: PD Status by Decade. ‘?’ indicates items where PD status could not be computed. Prop(ortion) 1 equals total PD divided by total for which status could be computed (sum of total PD and Not PD). Prop(ortion) 2 equals total PD divided by number of items for which any author date was known (’Any Date’ in previous table).

Table 2 reports the results of direct computation of PD status based on the information available. Note that, in doing these computations, we have augmented the basic life plus 70 rule with the additional assumptions that a) all items published in 1870 or before are PD b) no author is older than 100 (so if a birth date is more 170 years ago the item is PD) c) every author lives at least until 30 (so that any work published by an author born less than a 100 years ago is automatically not PD).

As is to be expected, for the majority of the periods, the availability of PD status (either PD or Not PD) closely tracks the availability of death date information — the total for which PD status can be determined (the sum of PD and Not PD) almost exactly equals the total for which death date information is available. It is only in the last period 1940-1950 that the birth date appears to make any contribution. More interesting, is how the number PD and Not PD vary over time, especially relative to each other (and as a proportion of the records for which any date is available).

These two proportions/ratios are recorded in the last two columns which record, respectively: 1) the PD total relative to the number of items for which any status could be computed (i.e. the sum of PD and Not PD) 2) the PD total relative to the total number of items for which any date information is available. These ratios change dramatically over the periods shown: starting in the 1870-1880 period in the high 90%s by the 1940s they are down to 20% or below.

Table 3: Suggested PD Proportions

The key question for us is how to extrapolate these PD proportions to the full set of records — i.e. from the set of records for which there is the necessary birth/death date information to that where there is not. The simplest, and most obvious, approach is to assume that the proportions are identical and therefore that the PD proportions calculated on the partial dataset apply to the whole. However, there are some obvious deficiencies in this approach.

In particular, our ability to compute a PD status is largely linked to the existence of a death date and it is likely that the presence of this information is itself correlated with authorial age — after all a death date can only exist once that person has died! This correlation, and the bias it gives rise to, is probably small in the early periods — the authors of any pre 1930 work are almost certainly no longer alive today. However, for the later periods, the bias may be more substantial — it is in these last two periods (1930-1940 and 1940-1950) that there is a significant reduction in the number of records with a death date and (relatedly) a significant increase in the number of records for whom the PD status is unknown.

Thus, in converting the partial PD proportions to full PD proportions it seems sensible to revise down somewhat the partial figures with the revision being greater in later periods. Moreover, we have a lower bound for any downwards revision provided by the total PD as a proportion of all records — which even in the 1940-1950 period stood at 6%. In light of these considerations Table 3 gives fairly conservative figures for PD proportions that when estimating PD size based on publication dates. Interestingly, even with out conservative assumptions, these proportions are rather higher than those used in our previous analysis.

This post continues the work begun in this earlier post on “Estimating Information Production and the Size of the Public Domain”. Update: 2009-07-17 there is now a follow-up post.

Having already obtained estimates of the number of items (publications) produced each year based on library catalogue data our next step is to convert this into an estimate of the “size” of the public domain. (NB: as already discussed, “size” could mean several different things. Here, at least to start with, we’re going to take the simplest and crudest approach and equate size with number of publications/items.)

The natural, and most obvious, approach here is to go through our 1 million+ items and compute their public domain status (as discussed in this earlier post). Unfortunately, as detailed there, this is problematic because we often have insufficient information in library catalogues with which to compute PD status with certainty — in particular, author death dates are frequently absent. Thus, it will be necessary to fall back on some approximate method.

For example, we can use base PD status on simple publication dates: if a book was published, say, 140 years ago it is very likely it is in the public domain — for it to be in copyright its author must have lived more than 70 years after the book came out (remember copyright lasts for life plus 70 years in the EU)! Conversely, any publication less than 70 years old is almost certainly not in the public domain. For periods in between we can assume some proportion of publications are PD starting close to zero for more recent items and rising towards one for older ones. A calculation along those lines is provided in the following table:

Number of UK Public Domain Publications (Based on Cambridge University Library Catalogue Data)

So, based on the assumptions regarding PD proportions given in the table, there are somewhat over 600 thousand PD books according to the holdings of Cambridge University Library (of which just over half, approx 390k are from before 1870). The British Library dataset is approx 4x as big as Cambridge University Library and the numbers scale up roughly proportionately giving a total of over 2.4 million items.

Of course this is a fairly crude approach based purely on publication date and it be improved in a variety of ways, most notably by using the authorial birth date information which is usually present in catalogue data (we can also use death date information where present). This will be the subject of the next post. (2009-07-17 the post is up here).

Here we’re going to look at using library catalogue data as a source for estimating information production (over time) and the size of the public domain.

Library Catalogues

Cultural institutions, primarily libraries, have long compiled records of the material they hold in the form of catalogues. Furthermore, most countries have had one or more libraries (usually the national library) whose task included an archival component and, hence, whose collections should be relatively comprehensive, at least as regards published material.

The catalogues of those libraries then provide an invaluable resource for charting, in the form of publications, levels of information production over time (subject, of course, to the obvious caveats about coverage and the relationship of general “information production” to publications).

Furthermore, library catalogue entries record (almost) the right sort of information for computing public domain status, in particular a given record usually has a) a publication date b) unambiguously identified author(s) with birth date(s) (though unfortunately not death date). Thus, we can also use this catalogue data to estimate the size of the public domain — size being equated here to the total number of items currently in the public domain.

Results

To illustrate, here are some results based on the catalogue of Cambridge University Library which is one of the UK’s “copyright libraries” (i.e. they have a right to obtain, though not an obligation to hold, one copy of every book published in the UK). This first plot shows the numbers of publications per year (as determined by their publication date) up until 1960 (when the dataset ends) based on the publication date recorded in the catalogue.

A major concern when basing an analysis on these kinds of trends is is that fluctuations over time derive not from changes in underlying production and publication rates but changes in acquisition policies of the library concerned. To check for this, we present a second plot which shows the same information but derived from the British Library’s catalogue. Reassuringly, though there are differences, the basic patterns look remarkably similar.

Number of items (books etc) Per Year in the Cambridge University Library Catalogue (1600-1960).

Number of items (books etc) Per Year in the British Library Catalogue (1600-1960).

What do we learn from these graphs?

• In total there were over a million “Items” in this dataset (and parsing, cleaning, loading and analyzing this data took on the order of days — while the preparation work to develop and perfect these algorithms took weeks if not months)
• The main trend is a fairly consistent, and approximately exponential, increase in the number of publications (items) per year. At the start of our time period in 1600 we have around 400 items a year in the catalogue while by 1960 the number is over 16000.
• This is a forty-fold increase and corresponds to an annual growth rate of approx 0.8%. Assuming “growth” began only around the time of the industrial revolution (~ 1750) when output was around 1000 (10-year moving average) gives a fairly similar growth rate of around 0.89%.
• There are some fairly noticeable fluctuations around this basic trend:
  1. There appears to be a burst in publications in the decade or decade and a half before 1800. One can conjecture several, more or less intriguing, reasons for this: the cultural impact of the French revolution (esp. on radicalism), the effect of loosening copyright laws after Donaldson v. Beckett, etc. However, without substantial additional work, for example to examine the content of the publications in that period these must remain little more than conjectures.
  2. The two world wars appear dramatically in our dataset as sharp dips: the pre-1914 level of around 7k+ falls by over a third during the war to around 4.5k and then rises rapidly again to reach, and pass, 7k per year in the early 20s. Similarly, the late 1930s level of around 9.5k per year drops sharply upon the outbreak of war reaching a low of 5350 in 1942 (a drop of 45%), and then rebounding rapidly at the war’s end: from 5.9k in 1945 to 8k in 1946, 9k in 1947 and 11k in 1948!

To do next (but in separate entries — this post is already rather long!):

• Estimates for the the size of the public domain: how many of those catalogue items are in the public domain
• Distinguishing Publications (”Items”) from “Works” — i.e. production of new material versus the reissuance of old (see previous post for more on this).

Colophon: Background to this Research

I’m working on a EU funded project on the Public Domain in Europe, with particular focus on the size and value of the public domain. This involves getting large datasets about cultural material and trying to answer questions like: How many of these items are in the public domain? What’s the difference in price and availability of public domain versus non public domain items?

I’ve also been involved for several years in Public Domain Works, a project to create a database of works which were in the public domain.

Colophon: Data and Code

All the code used in parsing, loading and analysis is open and available from the Public Domain Works mercurial repository. Unfortunately, the library catalogue data is not: library catalogue data, at least in the UK, appears to be largely proprietary and the raw data kindly made available to us for the purposes of this research by the British Library and Cambridge University Library was provided only on a strictly confidential basis.