miscellaneous factZ – The online home of Rufus Pollock

Continues the series of post related to analyzing catalogue data, here are some stats on author “significance” as measured by the number of book entries (’items’) for that author in the Cambridge University Library catalogue from 1400-1960 (there being 1m+ such entries).

I’ve termed this measure “significance” (with intentional quotes) as it co-mingles a variety of factors:

• Prolificness — how many distinct works an author produced (since usually each work will get an item)
• Popularity — this influences how many times the same work gets reissued as a new ‘item’ and the library decision to keep the item
• Merit — as for popularity

The following table shows the top 50 authors by “significance”. Some of the authors aren’t real people but entities such as “Great Britain. Parliament” and for our purposes can be ignored. What’s most striking to me is how closely the listing correlates with the standard literary canon. Other features of note:

• Shakespeare is number 1 (2)
• Classics (latin/greek) authors are well-represented with Cicero at number 2 (4), Horace at 5 (9) followed Homer, Euripides, Ovid, Plato, Aeschylus, Xenophon, Sophocles, Aristophanes and Euclid.
• Surprise entries (from a contemporary perspective): Hannah More, Oliver Goldsmith, Gilbert Burnet (perhaps accounted by his prolificity).
• Also surprising is limited entries from 19th century UK with only Scott (26), Dickens (28) and Byron (41)

Top 50 authors based on CUL Catalogue 1400-1960

The other thing we could look at is the overall distribution of titles per author (and how it varies with rank — a classic “is it a power law” question). Below are the histogram (NB log scale for counts) together with a plot of rank against count (which equates, v. crudely, to a transposed plot of the tail of the histogram …). In both cases it looks (!) like a power-law is a reasonable fit given the (approximate) linearity but this should be backed up with a proper K-S test.

Histogram of items-per-author distribution (log-log)

Rank versus no. of items (log-log)

TODO

• K-S tests
• Extend data to present day
• Check against other catalogue data
• Look at occurrence of people in title names
• Look at when items appear over time

Colophon

Code to generate table and graphs in the open Public Domain Works repository, specifically method ‘person_work_and_item_counts’ in this file: http://knowledgeforge.net/pdw/hg/file/tip/contrib/stats.py

Last week I was in Oxford to give a talk at the IP Seminar on “How Long Should Copyright Last?”. I have now posted the slides from the talk online.

In addition to covering the basic outline of the optimal term calculation, I was also able to give some results from the recent research on the public domain (see slide 58 onwards).

The other day I posted up some work-in-progress on the subject of patterns of knowledge production.

That material is still in a fairly preliminary state. However, my decision to release it it in this form was a conscious decision and part of an ongoing attempt on my part to practice a more open “release early, release often” approach to research.

In doing this I’m drawing direct inspiration from the open source and open notebook (science) communities and seeking to engage in what might be termed open notebook social science!

I think most researchers (including myself) feel a reluctance to put out material that isn’t at a reasonable level of maturity. While there are some good reasons for this, I think the main motivations are less positive, and are primarily to do with fear: be it of criticism or that your ideas are “taken” by others. While such fears can have some basis, it seems to me the benefits of an open approach — in terms of visibility, dissemination, and potential for collaboration — significantly outweigh any of the associated risks.

Over the last year, I’ve already been making some effort to move in this direction but from this point on I’m aiming to do this more thoroughly and methodically. A first step in this will be to put all the “patterns” and data online.

I’m posting up some work-in-progress entitled Exploring Patterns of Knowledge Production (link to full pdf). Below I’ve excerpted the introduction plus list of motivational questions. Comments (and critique) very welcome!

Introduction

In what follows the term ‘knowledge’ is here used broadly to signify all forms of information production including those involved in technological innovation, cultural creativity and academic advance.

Today, thanks to rapid advances in IT, we have available substantial datasets pertaining both to the extent and the structure of knowledge production across disciplines, space and time.

Especially recent is the availability of good ’structural’ data — that is data on the linkages and relationships of different pieces of knowledge, for example as provided by citation information. This new material allows us to explore the “patterns of knowledge production” in deeper and richer ways than ever previously possible and often using entirely new methods.

For example, it has long been accepted that innovation and creativity are cumulative processes, in which new ideas build upon old. However, other than anecdotal and case-study material provided by historians of ideas and sociologists of science there has been little data with which to study this issue — and almost none of a comprehensive kind that would make possible a systematic examination.

However, the recent availability of comprehensive databases containing ‘citation’ information have allowed us to begin really examining the extent to which new work builds upon old — be it a new technology as represented by a patent or a new idea in academia as represented by a paper, builds upon old.

Similar opportunities present themselves in relation to identifying the creation of new fields of research or technology, and tracing their evolution over time. Here the existence of extensive “structural information” as presented, for example, by citation databases, enables new systematic approaches — for example, can new fields be identified (or perhaps defined) as points in ‘knowledge space’ far away from the existing loci of effort? or, alternatively, by the nature of its connections to the existing body of work?

Structural information of this kind can also be used in charting other changes in the life-cycle of knowledge creation. For example, to offer a specific conjecture, a field entering decline, though still exhibiting a similar level of output (papers etc) and even citations to a field in rude health, may display a citation structure which is markedly different — for example, more clustered within the field itself. Thus, by using this additional structural information we may be able to gain insights not available with simpler approaches.

At the same time, structure must also play a central role in any attempt to estimate knowledge related ‘output’ measures. This is of course not true for other forms of ‘output’, for example that of corn of steel, where we have relatively well-defined objective measures available: tonnes of such-and-such a quality.

But knowledge is different: the most obvious metrics, such as number of patents or papers produced, seem entirely inadequate: one particular innovation or paper may be ‘worth’ as much as a hundred or a thousand others.

The issue here is that, compared to corn or steel, knowledge is extremely inhomogeneous, or put slightly differently, quality (or significance) differs very substantially across the individual pieces of knowledge (papers, patents etc).

Thus, any serious attempt to measure the progress of knowledge must must find some way to do this quality-adjustment and structural information seems essential to this.

What specific questions might we explore with such datasets?

The following is a (non-exhaustive) list of the kinds of questions one might explore using these new datasets:

• Can we use structure to infer information about quality of individual items? Clearly the answer is yes, for example by using a citation-based metric where a work’s value is estimated based on its citation by others.
• Can we then use this information together with more global structure of the production network to gain a better idea of total (quality-adjusted) output. This would allow one to chart progress, or the lack of it, over time?
• Can we use structural information to investigate the life-cycle of fields? For example, can we see fields ‘dying out’ or the onset of diminishing returns? Can we see new fields coming into existence and their initial growth patterns?
• What about productivity per capita and its variation across the population? It is likely that one would need to focus here within a discipline as it would be difficult to directly compare across disciplines, at least when using quality adjusted productivity.
• Do the structures of knowledge production vary over time and across disciplines and does this have implications for their productivity? Can we compare the structure of evolution in technology or economics with that in ‘natural’ evolution and, if not, what are the primary differences?
• How do other (observable) attributes related to the producers of knowledge (their collaboration with others, their geographical location) affect the structures we observe and the associated outcomes (output, productivity) already discussed above?
• Do different policies (for example openness vs. closedness — weak vs. strong IP) have implications for the structure of production and hence for output and productivity?
• Is knowledge production (in a particular area) ergodic or path-dependent? Crudely: do we always end up in the same place or do small shocks have large long-term effects?

I wanted to do a reverse proxy to wordpress.com in order to integrate an existing wordpress.com blog into an existing site. This turned out to be a little trickier than I’d thought due to wordpress.com’s usage of gzip deflation of their output.

Figured this out thanks to sound advice here and here:

  ...

  <Proxy *>
        Allow from .mysite.com
  </Proxy>
  ProxyPass / myblog.wordpress.com
  ProxyPassReverse / http://myblog.wordpress.com/
  ProxyHTMLURLMap http://myblog.wordpress.com/ /
  <Location />
    SetOutputFilter proxy-html
    # get rid of Content-Encoding at wordpress end
    # To use this you'll need to do (on debian) a2enmod headers
    RequestHeader unset  Accept-Encoding

    # Alternative method: inflate then deflate again ... (requires more effort at our end)
    # Could NOT get this to work (though suggested by both reference sites!)
    # SetOutputFilter INFLATE;proxy-html;DEFLATE
  </Location>

  ....

Last week I was at the ATRIP Conference to give an invited talk on “How Long Should Copyright Last?”, based on my paper: Forever Minus a Day? Calculating the Optimal Term of Copyright.

Slide are here, and you can find the text of the accompanying introduction below (I plan to write up the full exposition as a short essay — but that is to come).

Most ATRIP participants were lawyers not economists, so this was an opportunity to do a more non-technical presentation (so no equations!). As with most economics, the fundamentals of calculating copyright term are simple: it is just a demand curve plus “welfare analysis” (a fancy name for adding up social benefits and costs), and shorn of “obfuscating” algebra these matters should be understandable by anyone.

How Long Should Copyright Last: Introduction

Before I begin it is important to note that in considering copyright and its term, we must leave to one side the questions of attribution and integrity — their existence and term can and should be considered quite separately from the ‘economic’ rights that form the core of copyright as it operates today.

This small caveat done, I beg your indulgence for a brief historical excursion. In particular, I ask you to cast your mind back a century and a half and more to the Houses of Parliament in the February of 1841.

[[Picture of Serjeant Talfourd]]

As many of you will be aware Serjeant Talfourd had, by this point, been doggedly pursuing a new copyright act for four years — since 1837. Originally wide in scope the Bill had been narrowed and the attention of both supporters and critics alike had come to focus on a single feature of that Act: the proposed extension in the term of protection. Specifically Talfourd’s Act proposed changing the then rule of 28 years or life (whichever being the longer) to life plus sixty — remarkably close to the life plus 70 of today.

[[Picture of Macaulay]]

By February 1841 Talfourd’s Bill had failed no less than 4 times. On its fifth attempt it had reached a second reading and on the fifth of February it came before the House. After a brief introduction by Talfourd — mindful that this was not the first time the matter had been discussed — Thomas Babbington Macaulay rose to speak. In a masterly disquisition, both in content and rhetoric, Macauley set out his opposition to the Bill, and did so so tellingly that the motion was defeated. Talfourd, who lost his seat at the next election, and therefore only saw his Bill pass in the hands of another — and in much reduced form — remained forever embittered by Macaulay’s intervention — coming so late and so decisively in the process.

To read Macaulay’s speech, and, for that matter, the views expressed on all sides in that debate, is to be struck by how little has changed.

[[Valenti Picture]]

When Jack Valenti and Mary Bono are found in recent times calling for a term of ‘Forever Minus a Day’ one hears the echoes of Serjeant Talfourd all those years ago, just as one can hear echoes of those who oppose extensions today of the likes of Henry Warburton, a radical politician and vehement opponent of Talfourd, who claimed the extension was “a robbery upon the public” and that copyright ought to be fixed, “only on such a term of years as would prove a sufficient inducement for authors to write good books”.

And the analogy is telling in other ways. Though Talfourd’s Bill was beaten back by a swell of opposition year after year eventually it was passed — albeit in reduced form and by Lord Mahon — with this success attributable to a persistence made possible not, primarily, by the size, but by the concentration of the interests who sought its passage. Like Fabius Cunctator the proponents of extension, sustained by deep reservoirs of emotional and financial commitment, can afford to wait, able to return, as necessary, again and again, until an opportune moment presents itself for the attainment of their purposes — for the opposition to extension, though broad is ’shallow’ and therefore more easily dissipated by distraction and division.

[[Philosophical Differences]]

Even more striking are the similarity in the issues that occupy centre stage in this debate. First, the fundamental ‘philosophical’ question — which colours all of discussion — of whether we confer copyright because it is a natural right — which should therefore last forever — or for ‘utilitarian’ purposes, that is the public good — in which case it almost certainly should not. Second, descending from these lofty heights of principle, what is the actual effect is copyright? In particular, does it operate to raise price and restrict access — that is: is it a monopoly?; and what specifically are the benefits that accrue to the producers of copyrightable works, and what costs to the public and others who wish to use and reuse them.

I think it is clear that economists — or any group for that matter — have no great claim to authority on answering this first question of principle, for it seems, ultimately, one of opinion. That said, I would note two points which must raise grave doubts as to the existence of any fundamental natural right from which copyright might spring.

First, term limited in all jurisdictions. Second, the breadth of copyright’s application both in subject matter, quality and ownership. For can we truly convince ourselves that “eternal expressions of the human spirit”, worthy of exclusivity for all time, subsist in an advert for toothpaste; or convince ourselves of the special status of the creator when so much copyright today, perhaps even the majority, is immediately, and indeed often automatically, assigned from the ‘creator’ to a corporation.

However, it is not my intention to enter into this debate any further here. Rather, in the interests of ‘full disclosure’ I wish only to make clear my views — and those of economists generally — on the matter, namely that copyright is not a natural right but is created and maintained for the purpose of promoting and securing the public good, no more, no less. (These are views which can come as no surprise given the nature of this talk — an analysis of term only makes sense if its basis is a utilitarian one!)

[[My Views]]

Furthermore, let me also make clear, right at the outset, my view, and one again I think shared by almost all economists, that copyright is a monopoly. This is not to say that copyright is bad — far from it. But to deny that copyright is a monopoly is to obscure its basic nature and operation — an obscuration that has, furthermore and unfortunately, been most common and attractive to those pursuing copyright’s enlargement.

[[The Big M Word]]

And what is the general tendency of monopoly — to echo Macaulay once again? It is indeed to raise prices and limit access. Now, of course, we may debate the precise extent of these effects, but there can be no denying that the very purpose of copyright’s existence is to confer on a single entity — the copyright holder — the power to control the dissemination, and hence the price, of all instances of a particular good — i.e. all copies of a given work.

This is the very definition of a monopoly and the fact that there may exist other goods, other works, which compete with that one makes no difference — a monopoly of apples is no less a monopoly because one does not control oranges. Of course, the existence and proximity of substitutes will alter the affect of the monopoly, but one must be cautious here: close substitutes may limit the negative effects of the copyright monopoly but they will, for the very same reasons, also limit the gains (those increased revenues for copyright-holder).

Returning then to Macaulay whose expression of the matter I cannot better:

[[Macaulay Again]]

“It is good that authors be remunerated; and the least exceptionable way of remunerating them is by a monopoly. Yet monopoly is evil. For the sake of the good we must submit to the evil, but the evil ought not to last a day longer than is necessary for securing the good.”

Our task then is to answer the implicit question: how long should copyright last (so as to not be a day longer than is needed)? More specifically what are the degrees of benefit and harm created by copyright’s monopoly and at what level should term be set to achieve the most advantageous balance of the two?

7/10 (9/10 for genre). Very cleverly done, with a real emotional engagement in the protagonists predicament. This lives up to its reputation as being (well) above average.

Just came across this, now year-old, story of Qualcomm’s submarining efforts (and subsequent cover-up) in relation to the JVT (joint video team) standardization effort for H.264/MPEG-4 AVC codec.

Qualcomm participated in the JVT (joint video team) standardization effort for H.264/MPEG-4 AVC (a video codec) which was concluded in 2003. However, they (intentionally) did not declare relevant patents (though obliged to do so) and subsequently sued Broadcom for infringement of these patents in 2005. At the resulting trial they then concealed extensive documentary material relating to their involvement in the standards process. However this fraud was discovered (in dramatic fashion) and they lost their suit with their patents being voided — though only in relation to the standard. From the Court of Appeals Judgement (United States Court of Appeals for the Federal Circuit Judgement 2007-1545, 2008-1162 pp.3-4, emphasis added):

Plaintiff Qualcomm is a member of the American National Standards Institute (”ANSI”), which is the United States representative member body in the ISO/IEC, and was an active dues-paying member for many years prior to 2001. It is also a member of the ITU-T and a participant in the JVT. Qualcomm did not disclose the ‘104 and ‘767 Patents to the JVT prior to release of the H.264 standard in May 2003.

On October 14, 2005, Qualcomm filed the present lawsuit against Broadcom in the United States District Court for the Southern District of California, claiming that Broadcom infringed the ‘104 and ‘767 Patents by making products compliant with the H.264 video compression standard. A jury trial was held from January 9, 2007, to January 26, 2007. The jury returned a unanimous verdict as to non-infringement and validity, finding that (1) Broadcom does not infringe the ‘104 and ‘767 Patents; and (2) the ‘104 and ‘767 Patents were not shown to be invalid. The jury also returned a unanimous advisory verdict as to the equitable issues, finding by clear and convincing evidence that (1) the ‘104 Patent is unenforceable due to inequitable conduct; and (2) the ‘104 and ‘767 Patents are unenforceable due to waiver.

On March 21, 2007, the district court entered an order (1) finding in favor of Qualcomm and against Broadcom on Broadcom’s counterclaim of inequitable conduct as to the ‘104 Patent; (2) finding in favor of Broadcom and against Qualcomm on Broadcom’s affirmative defense of waiver as to the ‘104 and ‘767 Patents; and (3) setting a hearing on an Order to Show Cause as to the appropriate remedy for Qualcomm’s waiver. The district court’s conclusion that Qualcomm waived its rights to assert the ‘104 and ‘767 Patents was based on Qualcomm’s conduct before the JVT.

Throughout discovery, motions practice, trial, and even post-trial, Qualcomm adamantly maintained that it did not participate in the JVT during development of the H.264 standard. Despite numerous requests for production and interrogatories requesting documents relating to Qualcomm’s JVT participation prior to adoption of the H.264 standard, Qualcomm repeatedly represented to the court that it had no such documents or emails. On January 24, 2007, however, one of the last days of trial, a Qualcomm witness testified that she had emails that Qualcomm previously claimed did not exist. Later that day, Qualcomm produced twenty-one emails belonging to that witness. As the district court later discovered, these emails were just the “tip of the iceberg,” as over two hundred thousand more pages of emails and electronic documents were produced post-trial. Remedy Order at 1245. The district court later determined that these documents and emails “indisputably demonstrate that Qualcomm participated in the JVT from as early as January 2002, that Qualcomm witnesses . . . and other engineers were all aware of and a part of this participation, and that Qualcomm knowingly attempted in trial to continue the concealment of evidence.”

Instructions on using sqlalchemy migrate with Pylons, especially to convert an existing pylons project to use sqlalchemy migrate

This is based off several excellent sources including this guide and these threads.

One important point to note is that you are likely to end up with two versions of your model tables: one in yourapp/model and one in yourapp/migration/versions/*.py with the former representing your tables at HEAD and the second containing upgrade/downgrade scripts whose final result is HEAD. This duplication is a bit annoying and I discuss how it can be avoided below.

1. Install sqlalchemy migrate for your project e.g.

  pip -E {your-virtualenv} install sqlalchemy-migrate
  # or
  easy_install sqlalchemy-migrate

NB: latest version of migrate are only compatible with sqlalchemy >= 0.5 (for previous version of sqlalchemy you need migrate <= 0.4.5)

2. Create the migrate repository (i.e. store for upgrade scripts …).

In your project directory

  migrate create myapp/migration/ "MyApp"

Now create a temporary helper script:

  migrate manage dbmanage.py --repository=myapp/migration/ --url={your-sqlalchemy-db-uri}

3. Set up db version control

  python dbmanage.py version_control

Check the current version (should be 0)

  python dbmanage.py version

4. Create a migration script for your existing db

  python dbmanage.py script "Add existing tables"

This will create a script in myapp/migration/versions/001addexisting_tables.py

Copy into that file the definition for all your existing tables (and other database stuff such as constraints) and then create those tables in the upgrade() function (and delete them in downgrade()).

That’s it! (in theory)

Additional Issues

1. Duplication of model/db code

You now have two places for model/db code:

1. Your migration scripts
2. Your actual model

This doesn’t have to be a problem but it is an obvious way for bugs to creep especially when some people start by creating their DB from the model code and others from the migration scripts.

Warning: this method will not work if do stuff in your table creation that is not persisted into the actual DB sql (e.g. column default values based on a function, custom db types …).

One way to avoid the duplication is to have all table creation and alteration confined to your migration scripts and then have your model tables set up directly from the DB using the autoload=True option. The one disadvantage of this is you can’t see the complete DB set up in one places as tables construction may be spread over several migrate scripts. One solution to this is provided by the experimental ‘create_model’ command which dumps the current DB model in the required sqlalchemy table code.

More discussion in this migrate-users thread

Bringing the Migration DB up to date

If you do set up your DB (from new) directly from your model code rather than the migration scripts then this requires that you set up the migration stuff and update the migrate version to the correct number. (I note there is an experimental updatedbto_model command which is supposed to do this for you). You can do this as follows (assuming your migrate repository is at YOURAPP:

      from migrate.versioning.api import version_control, version
      import YOURAPP.migration.versions
      v = version(YOURAPP.migration.__path__[0])
      # log.info( "Setting current version to '%s'" % v )
      # url is your sqlalchemy db url 
      version_control(url, YOURAPP.migration.__path__[0], v)

Extras

• Should wrap upgrade/downgrade in transactions. I found one way to do this here but testing indicated this didn’t work for me (rollback was not working properly when there was an error)

There’s an interesting 6 month fellowship at OPSI for work on economics of public sector information being funded by ESRC and National Archives. Deadline for applications is 6th August:

Valuing information: an economic analysis of public sector information and its re-use

Length of Fellowship: Six months

Proposed start date: Autumn 2009

Applications to be submitted as soon as possible (and by 6 August)

Location of Fellowship: The National Archives’ sites (Central London and Kew)

As part of its Placement Fellowship Scheme, the Economic and Social Research Council (ESRC) and The National Archives welcome applications from academic economists interested in working in a research capacity in the Office of Public Sector Information (OPSI). OPSI is part of The National Archives, a member of the Ministry of Justice family, working to set standards, deliver access and encourage the re-use of PSI.

The Placement Fellowship Scheme encourages social science researchers to spend time within a partner organisation to undertake policy relevant research and to develop the research skills of partner employees. The Fellowship will be jointly funded by the ESRC and OPSI while the Fellow remains employed by his or her institution.

See the document below for further details on the Placement Fellowship: http://www.nationalarchives.gov.uk/documents/esrc-placement-fellowship-june-09.pdf