Category Archives: Musings

The Elusive Disappearance of Community

From Laslett ‘Phillipe Ariès and “La Famille”‘ p.83 (quoted in Eisenstein, p.131):

The actual reality, the tangible quality of community life in earlier towns or villages … is puzzling … and only too susceptible to sentimentalisation. People seem to want to believe that there was a time when every one belonged to an active, supportive local society, providing a palpable framework for everyday life. But we find that the phenomenon itself and its passing — if that is what, in fact happened– perpetually elude our grasp.

The Knowledge Commons is Different

I was looking again recently at “Understanding the Knowledge Commons” which I had perused previously.

While reading the introductory chapter by Hess and Ostrom I came across:

People started to notice behaviors and conditions on the web-congestion, free riding, conflict, overuse, and “pollution” — that had long been identified with other types of commons. They began to notice that this new conduit of distributing information was neither a private nor strictly a public resource.

I think they are absolutely right to consider the analogies of “knowledge commons” with traditional commons. However, and at the same time, I think it essential to emphasize that “knowledge commons” are also fundamentally different.

The key difference here is in the nature of the underlying good that makes up the commons: in traditional cases the good is some physical resource — seas, rivers, land — to which usage is shared (either de facto or de jure), while in the knowledge case, well, it’s knowledge!

Now physical resources are by their nature ‘rival’ (or ‘subtractable’ as the authors put it), that is your usage and my usage are substitutes — your usage reduces the amount available for me to use and, when we are close to capacity, is strictly rival — either I use it or you use it. Knowledge, however, is a classic example of a non-rival resource: when you learn something from me I’ve lost nothing but you’ve gained something.

This means, for example, that the classic ‘tragedy’ of the commons where overuse leads to destruction of the resource is simply not possible for a knowledge commons — in fact, knowledge is like some magical food from a fairytale where the more its used the more of it there is!

The more useful ‘commons’ analogy for knowledge is not in relation to use but to production and the ‘free-rider’ problems that can arise where something must be done by a team or community. The issue here is that a separation appears between your effort (private) and the resulting outcome (shared) which may lead to an under-supply of effort and ‘free-riding’ on the efforts of others (if there are ten people on guard duty late at night, one can probably take a nap endangering the city but if all ten of them do it then it could be disastrous).


1. Before any misunderstanding arises I should make clear that the authors also acknowledge the role of rival/non-rival distinction — Ostrom, in fact, was one of the ‘coiners’ of the term rivalry. However, the article’s overall focus is on the analogies with the traditional commons.

2. Jamie Boyle has talked about the “second enclosure movement”. Though interesting to make this analogy I think references to the original enclosure movement is unfortunate for two reasons. First, it reinforces the mistaken analogy between knowledge and physical goods. Second, the evidence that the original enclosure movement was bad isn’t very compelling (in fact, it probably delivered net benefits).

Algorithm Speed and the Challenge of Large Datasets

In doing research for the EU Public Domain project (as here and here) we are often handling large datasets, for example one national library’s list of pre-1960 books stretched to over 4 million items. In such a situation, an algorithm’s speed (and space) can really matter. To illustrate, consider our ‘loading’ algorithm — i.e. the algorithm to load MARC records into the DB, which had the following steps:

  1. Do a simple load: i.e. for each catalogue entry create a new Item and new Persons for any authors listed
  2. “Consolidate” all the duplicate Persons, i.e. a Person who is really the same but for whom we create duplicate DB entries in part 1 (we can do this because MARC cataloguers try to uniquely identify authors based on name + birth date + death date).
  3. [Not discussed here] Consolidate “items” to “works” (associate multiple items (i.e. distinct catalogue entries) of, say, a Christmas Carol, to a single “work”)

The first part of this worked great: on a 1 million record load we averaged between 8s and 25s (depending on hardware, DB backend etc) per thousand records with speed fairly constant throughout (so that’s between 2.5 and 7.5h to load the whole lot). Unfortunately, at the consolidate stage we ran into problems: for a 1 million item DB there were several 100 thousand consolidations and we were averaging only 900s per 1000 consolidations! (This also scaled significantly with DB size: a 35k records DB averaged 55s per 1000). This would mean a full run would require several days! Even worse, because of the form of the algorithm (all the consolidation for a given person were done as a batch) we ran into memory issues on big datasets with some machines.

To address this we switched to performing “consolidation” on load, i.e. when creating each Item for a catalogue entry we’d search for existing authors who matched the information we had on that record. Unfortunately this had a huge impact on the load: time grew superlinearly and had already reached 300s per 1000 records at the 100k mark having started at 40 — Figure 1 plots this relationship. By extrapolation, 1M records would take 100 hours plus — almost a week!

At this point we went back to the original approach and tried optimizing the consolidation, first by switching to pure sql and then by adding some indexes on join tables (I’d always thought that foreign keys were auto indexed but it turned out not to be the case!). The first of these changes solved the memory issues, while the second resolved the speed problems providing a speedup of more than 30x (30s per 1000 rather 900s) and reduced the processing time from several days to a few hours.

Many more examples of this kind of issue could be provided. However, this one already serves to illustrate the two main points:

  • With large datasets speed really matters
  • Even with optimization algorithms can take a substantial time to run

Both of these have a significant impact on the speed, and form, of the development process. First, because one has to spend time optimizing and profiling — which like all experimentation is time-consuming. Second because longer run-times directly impact the rate at which results are obtained and development can proceed — often bugs or improvements only become obvious once one has run on a large dataset, plus any change to an algorithm that alters output requires that it be rerun.


Figure 1: Load time when doing consolidation on load

Talk by Frederick Scherer: Deregulatory Roots of the Current Financial Crisis

Last Thursday I attended a talk by Frederick Scherer at the [Judge] entitled: “Deregulatory Roots of the Current Financial Crisis”. Below are some sketchy notes.


Macro story:

  • Huge current account deficit for last 10-15 years
    • Expansionary Fed policy has permitted this to happen while interest rates are low
  • Median real income has not risen since the mid-1970s
    • Cheap money mean personal savings have dropped consistently: 1970s ~ 7%, 2000s ~ 1%
  • Basically overconsumption

Micro story:

  • Back in the old days, banking was very dull — three threes story, “One reason I never worked in the financial industry: it was very dull when I got my MBA in 1958”
  • S&L story of 1980s: inflation squeeze + Reagan deregulation
    • FMs: Fannie Mae, Freddie Mac get more prominent
    • [Ed]: main focus here was on pressure for S&L to find better returns without much mention of the thoughtlessness of Reagan deregulatory approach (deposits still insured but S&L can now invest in anything) and the fraud and waste it engendered — see “Big Money Crime: Fraud and Politics in the Savings and Loan Crisis” by Kitty Calavita, Henry N. Pontell, and Robert Tillman
  • In 1920s there were $2 billion of securitized mortgages (securitazation before the 1980s!)
  • Market vs. bank finance for mortgages: market more than bank by mid-1980s [ed: I think — graph hard to read]
  • To start with: FMs pretty tough when giving mortgages, but with new securitizers and lots of cheap money, standards dropped => moral hazard for issuers [ed: not quite sure why this is moral hazard — securitizers aren’t the ones who should care, it’s the buyers who should care]
  • Even if issuers don’t care, buyers of securitized mortgages should care and they depended on ratings agencies (Moodys, S&P etc)
  • Unfortunately, ratings agencies had serious conflicts of interest as they were paid to do ratings by firms issuing the securities! Result: ratings weren’t done well
  • Worse: people ignored systemic risk in the housing market and therefore made far too low assessment of risk of these securities [ed: ignoring systemic risks implies underestimating correlations — especially for negative changes — between different mortgage types (geographic, owner-type etc). Interesting here to go back and read the quarterly statement from FM in summer 2008 which claims exactly this underestimate.]
  • Banks over-leveraged for the classic reason (it raises your profits if things are good — but you can get wiped out if things are bad)
    • This made banks very profitable: by mid 2000s financial corporations accounted for 30% of all US corporate profits
    • Huge and (unjustified relative to other sectors) wage levels. Fascinating evidence here provide by correlating wage premia to deregulation: fig 6 from Philippson and Reshi shows dramatic association of wage premium (corrected for observable skills) with (de)regulation. Wage premium goes from ~1.6 in 1920s to <1.1 in 1960s and 70s and then back up to 1.6/1.7 in mid 2000s
  • Credit default swaps and default insurance: not entirely new but doubled every year from 2001 to the present ($919 billion in 2001 to $62.2 trillion in 2007)
    • Much of the time CDS issued without any holding of the underlying asset
    • There was discussion on regulating CDSes in 1990s (blue-ribbon panel reported in 1998) but due to shenanigans in the house and senate led by Phil Graham (husband of Wendy Graham who was head of Commodity Futures … Board), CDSes were entirely deregulated via act tacked onto Health-Education-Appropriations bill in 2001.

It goes bad:

  • Housing bubble breaks in 2007 or even 2006
    • Notices of default starts trending upwards in mid 2006
  • [ran out of time]

What is to be done:

  • Need simple, clear rules
    • A regulator cannot monitor everything day-to-day
  • Outlaw Credit Default Swaps
  • Anyone who issues CDOs must “keep skin in the game”
  • Leverage ratios. Perhaps? Hard to regulate.
  • Deal with too big to fail by making it hard for “giants to form” and breaking up existing over-large conglomerates
  • We need to remember history!

Own Comments

This was an excellent presentation though, as was intended, it was more a summary of existing material than a presentation of anything “new”.

Not sure I was convinced by the “remember history” logic. It is always easy to be wise after the event and say “Oh look how similar this all was to 1929”. However, not only is this unconvincing analytically — it is really hard to fit trends in advance with any precision (every business cycle is different), but before the event there are always plenty of people (and lobbyists) arguing that everything is fine and we shouldn’t interfere. Summary: Awareness of history is all very well but it does not provide anything like the precision to support pre-emptive action. As such it is not really clear what “awareness of history” buys us.

More convincing to me (and one could argue this still has some “awareness of history in it) are actions like the following:

  1. Worry about incentives in general and the principal-agent problem in particular. Try to ensure long-termism and prevent overly short-term and high-powered contracts (which essentially end up looking like an call option).

    Since incentives can be hard to regulate directly one may need to work via legislation that affects the general structure of the industry (e.g. Glass-Stegall).

    Summary: banking should be a reasonably dull profession with skill-adjusted wage rates similar to other sectors of the economy. If things get too exciting it is an indicator that incentives are out of line and things are likely to go wrong (quite apart from the inefficiency of having all those smart people pricing derivatives rather than doing something else!)

  2. Be cautious regarding financial innovation especially where new products are complex. New products have little “track record” on which to base assessments of their benefits and risks and complexity makes this worse.

    In particular, complexity worsens the principal-agent problem for “regulators” both within and outside firms (how can I decide what bonus you deserve if I don’t understand the riskiness and payoff structure of the products you’ve sold?). Valuation of many financial products such as derivatives depend heavily — and subtly — on assumptions regarding the distribution of returns of underlying assets (stocks, bonds etc).

    If it is not clear what innovation — and complexity — are buying us we should steer clear, or at least be very cautious. As Scherer pointed out (in response to a question), there is little evidence that the explosion in variety and complexity of financial products since the 80s has actually done anything to make finance more efficient, e.g. by reducing the cost of capital to firms. Of course, it is very difficult to assess the benefits of innovation in any industry, let alone finance, but the basic point that 1940s through 1970s (dull banking) saw as much “growth” in the real economy as the 1980s-2000s (exciting banking) should make us think twice about how much complexity and innovation we need in financial products.

Finally, and on a more theoretical note, I’d also like to have seen more discussion about exactly why standard backward recursion/rational market logic fails here and what implications do the answers have for markets and their regulation. In particular, one would like to know doesn’t knowledge of a bubbles existence in period T lead to its unwinding (and hence by backward recursion to its unwinding in period T-1, and then T-2 etc until the bubble never existed). There are various answers to this in the literature based on things like herding, presence of noise investors, uncertainty about termination, but it would be good to have a summary, especially as regards welfare implications (are bubbles good?), and what policy interventions different theories prescribe.

Discounting and Self-Control

I’m posting up an essay on “Discounting and Self-Control” (pdf). The essay, which I haven’t really touched for over a year, is still in its early stages but having lacked the time to do much on it over the last year, and going on the motto of “release early, release often”, I’m posting it up as a form of alpha version.

… then must you speak
Of one that loved not wisely, but too well;
Of one not easily jealous, but, being wrought,
Perplex’d in the extreme; of one whose hand,
Like the base Judean, threw a pearl away
Richer than all his tribe; …

Othello, The Moor of Venice


An agent’s intertemporal choices depend on a variety of factors, most prominently, their valuation of future payoffs as encapsulated in a discount function. However, it is also clear that factors such as self-control may also play an important role, and given the similarity of impact, a confouding one. We explore the literature on this issue as well as examining what occurs when those with higher time-preference (whether arising from discounting or self-control) also enjoy their consumption more.


The exercise of will, especially in the form of self-control, has long been recognized as central to human existence, experience, and morality. Over the last few decades there has been increasing interest in the issue from a scientific perspective. At the same time, it has also long been appreciated that humans (and other animals) make trade-offs between the present and the future — as well as between different points in the future, and that events taking place closer to the present are given greater weight than those which are more distant. Traditionally, at least in economics, this type of behaviour has been subsumed under the heading of discounting.

Both of these factors, self-control and discounting, affect behaviour, and choices, in relation to outcomes which do not (all) take place in the present. However they are distinct. Specifically, consider a very simple case of two outcomes A and B where B occurs after A (for example, A might be one ice cream today and B an ice cream and a doughnut tomorrow). Self-control issues arise where one prefers B over A but is unable to execute on this preference and therefore actually takes (‘chooses’) A. By contrast, in the discounting case A is actually preferred over B and therefore is chosen (freely) by the decision maker.

It would seem important to keep these two aspects of decision making clearly separated. While lack of ‘self-control’ is usually seen as disadvantageous and a reason for adopting various ‘commitment strategies’ — for example, by opting to remove various items from the choice set (having no cigarettes in the house) — the simple preference for the present over the future incorporated in the discounting model would seem to generate no such difficulties.

However, empirically it may prove rather difficult to do so. As shown by the simple example above the same observed ‘choice’ for A (one ice cream today) over B (ice cream plus doughnut tomorrow) can be the result of two very different processes. Thus if we only observe choices, and not the underlying preferences and/or the process by which the choice is arrived at, it may be impossible to distinguish the two.

It is perhaps for this reason that these distinct aspects are sometimes conflated. Consider, for example, Mischel et al 1989 which is entitled “Delay of Gratification in Children” and summarizes much of Mischel of pioneering work on this area. Mischel’s approach is clearly more oriented along the self-control aspect, and this is borne out in the types of experiments conducted (more on this below). Nevertheless they state (p.934) “The obtained concurrent associations [between treatments and delay] are extensive, indicating that such preferences reflect a meaningful dimension of individual differences, and point to some of the many determinants and correlates of decisions to delay (18).” Here the orientation towards self-control has become a general “decision to delay” and this is borne out by the associated footnote (18) which references related literature in other disciplines and is worth quoting in its entirety:

[… see full essay for more]

Ordnance Survey and Google: Why All the Fuss?

Lots of people have been up in arms about a letter sent out by Ordnance Survey about the “Use of Google Maps for display and promotion purposes”. With titles like “Are the Show Us A Better Way winners safe from Ordnance Survey?” (Guardian), “Home Secretary’s crime maps not allowed say Ordnance Survey” ( or “The mapping mess – Google v OS” ( these seemed to indicate some particularly unreasonable behaviour by OS.

However, after actually reading the original OS letter I’m far from convinced. In essence OS say:

  • If you have created the data yourself you can do whatever you like with it including plotting it on Google maps.
  • However, if you have derived the data from an OS map then you can’t. You can’t because a) as derived data OS have rights in it b) plotting it on a Google map according to Google’s to T&C gives Google a perpetual, royalty-free license to that data. Since, unsurprisingly (and not unreasonably) OS don’t want to give Google such a license they don’t want you plotting it on a Google map.
  • What the implications of this are then depend heavily on:
    1. Whether Google will change their licensing conditions (at least for their free service)
    2. What derived data is

Much of the discussion centred on the last of these items: what is derived data? OS state:

Simply put, Ordnance Survey derived data is any data created using Ordnance Survey base data. For example, if you capture a polygon or a point or any other feature using any Ordnance Survey data, either in its data form or as a background context to the polygon/point/other feature capture, this would constitute derived data.

It should also be borne in mind that data from other suppliers may be based on Ordnance Survey material, and thus the above considerations may still apply. We therefore recommend that you verify whether any third-party mapping you use may have been created in some way from Ordnance Survey data before displaying it on Google Maps.

NOTE: Again, the answer to this question is based on our understanding of which of Google’s standard terms and conditions we believe would apply. In the event that Google is prepared to offer you terms and conditions which do not involve you purporting to grant Google a licence of Ordnance Survey base or derived data, we would have no objection to your hosting such data on top of Google Maps in this scenario.

My understanding of this is that if you extract the geodata from an OS map (i.e. polygon, points, features) by some extraction method (such as tracing) then that’s derived data and OS can control what you do. This is pretty standard: if I copy text from a book by typing it out longhand I’m still infringing copyright.

However, this does not mean if I’m using OS maps as a base-layer and, for example, by clicking at some particular point I generate a lat-long (say to indicate where I live, or where a crime happened) then that lat-long is ‘derived’ data.

Now, of course, this could be a fine line: if I happened to click on a bunch of points, say to indicate a walk I went on, and these also showed the route of road there could be debate as to whether I’m infringing the OS rights in the feature or not.

Nevertheless, the basic principle (as I understand it) is clear: geodata created when using OS tools and maps is always yours unless it is directly replicating the underlying OS data. If this interpretation is correct then this whole debate is a bit of a storm in a teacup and projects such as crime-mapping or providing a loofinder aren’t at any risk from OS’s licensing terms.

Firefox versus IE: Browser Market Shares over Time

It would be interesting to chart over time the progress of open-source, standards compliant, Mozilla-type web browsers (e.g. Firefox) versus Microsoft’s Internet Explorer. As is often the case in other areas, it is not easy to get good (open) data over a reasonable time period. The graph below shows browser market share as measured by the browser usage of visitors to the W3Schools website (data source on Open Economics plus the code to extract original data into this usable form).


Browser Market Share (NB: Firefox was released in Nov 2004 but not listed separately by W3Schools until 2005)

Given the source, and therefore the bias towards more technically savvy users, these figures probably overstate Firefox’s market share somewhat, though the overall trend is probably largely correct. What we see is a steady and continuing increase in Mozilla (Firefox) market share ever since Firefox’s launch in Autumn 2004 and a concomitant decline in market share of IE (the little dip for Firefox at the end appears to be directly attributable to the launch of Google’s chrome). What is particularly interesting is that, at least for W3Schools users, we are almost at the point where there are as many people using Firefox as IE. This is significant for several reasons.

First, because of its level of usage it will no longer be possible for websites to only ‘work in IE’ but instead will always have to work in Firefox as well. This is both good for Firefox and for the standards-compliant browsers more generally (while, of course, Firefox itself is not perfectly standards compliant it has traditionally been much better than IE).

Second, it is an (unusual) example of a case where dominance has not been maintained. Generally a firm with established dominance in a given area is able to maintain — witness the robustness of Microsoft’s established dominance in other areas. By contrast, in this market, as the graph shows, Firefox has almost drawn level with IE and may soon surpass it if the trend of the last few years continues.

Buddhist Economics

The human problem of ‘scarce resources and unlimited wants’ is oft-posited as a primary motivation for studying economics. As this phrase makes clear, ‘wants’ (‘preferences’ to use the more usual terminology) are a central part of what we study, and the existence, and stability, of those ‘wants/preferences’ therefore merit serious consideration.1

Few of us have difficulty accepting the fundamental nature of our desire for food and shelter. However, many of us might have greater difficulties assigning the same fundamentality to the desire for a particular brand of designer perfume or a digital music player. In fact, it is unclear to what extent one can want what one has never known (or conceived of), and thus, while it is not difficult to imagine any human desiring food and shelter — especially when they are absent, it is hard to imagine a stone-age nomad, say, even being able to conceive of designer perfume or iPods (let alone feel their lack).

It is also telling that so many of the consumer goods, especially those away from the necessity end of the spectrum, appear to require active promotion to the public. Of course it is true, as economists are particularly fond of pointing out, that advertising has an informational component — simply letting you know about the existence and attributes of products. However, it is also hard to deny that advertising also has a substantial ‘persuasive’ component, operating either to create preferences or alter existing ones.

If so this has important implications. In particular, it strongly suggests that our wants aren’t simply given but are, at least to some extent, formed by our experience and choices.2 This raises some deep and important questions for economists to answer — questions with a major bearing on the state and direction of many modern societies. It also has some direct connections with one of the oldest, and most philosophical, of the world’s religious traditions: Buddhism. Central to Buddhist teaching are the Four Noble Truths. Succintly put these are, in order:3

  1. (Dukka — The Nature of Suffering) “Birth is suffering, aging is suffering, sickness is suffering, death is suffering, association with the unpleasant is suffering, dissociation from the pleasant is suffering, not to receive what one desires is suffering – in brief the five aggregates subject to grasping are suffering.”
  2. (Samudaya — The Origin of Suffering) “It is this craving (thirst) which produces re-becoming (rebirth) accompanied by passionate greed, and finding fresh delight now here, and now there, namely craving for sense pleasure, craving for existence and craving for non-existence (self-annihilation).”
  3. (Nirodha — The Cessation of Suffering) “It is the complete cessation of that very craving, giving it up, relinquishing it, liberating oneself from it, and detaching oneself from it.'”
  4. (Marga — The Path to Cessation of Suffering) “It is the Noble Eightfold Path, and nothing else, namely: right understanding, right thought, right speech, right action, right livelihood, right effort, right mindfulness and right concentration.”

Why is this teaching relevant here? First, observe a commonality: both economics and Buddhism takes unsatisfied ‘wants’ (or ‘cravings’) as a source of unhappiness. But how do go about solving this problem? Here economics and Buddhism part ways, and rather dramatically, with the Four Noble Truths presenting a path to the achievement of well-being which is almost diametrically opposite to that advocated by economics.

Specifically, the ‘economics’ approach, is based on taking preferences as given and focusing on generating the goods to satisfy them. By contrast, Buddhism sees ‘wants’ as ultimately unsatisfiable, and instead proposes that the way to well-being is not to satisfy them but to relinquish them — while some ‘cravings’ can be temporarily satisfied more will always be generated, moreover there some fundamental desires, such as the wish not to die, cannot be addressed in the material world.4

Put starkly: economic thought directs our energy and efforts to satisfying our wants / cravings, taking them as given, while Buddhism directs those self-same energies to altering our wants, and views most attempts to satisfy them by obtaining ever more ‘things’ as inevitably doomed to failure — in fact, actively counter-productive as more ‘cravings’ are generated by the very process of satisfaction (and attempting to satisfy them takes up our time and energy).

  1. It is interesting how the term ‘preference’ is studiously neutral, and almost anodyne in comparison with a term such as ‘want’, ‘desire’ or even ‘need’, each of which is a potential synonym. One might imagine, and this is simply conjecture, that the term was intentionally adopted in order to remove any overtone of judgement. After all, across most culture and over much of human history, the formation and satisfaction of ‘preferences’ has been a process laden with ethical, and religious, significance. 

  2. In economics jargon: preference are endogenous (i.e. determined within the system) rather than exogenous (fixed externally — e.g. by ‘nature’). The study of endogenous preferences is certainly not new. See for example the review of Bowles (1998) or the early incorporation of changeable preferences into the ‘traditional’ framework by Becker and Stigler (1977). 

  3. These translations of the Dhammacakkappavattana Sutta are taken from 

  4. At least, one might argue, not yet: one day, perhaps technology will even give us the ability to live forever either directly or via upload in silico. However, one senses that even then we will find new cravings and, even, to find our so long desired eternal life itself insufferable.