Grey dawn, you welcome not my spirit to the day

Grey dawn, you welcome not my spirit to the day.
Locked deep in winter’s embrace, the depths of January
Are moribund of hope, and I can but think on Spring
To keep from despair and an endless sojourn in the soft arms of sleep.

The day does not begin but seeps in, in sluggish batches from the East.
The watery light of a half-begotten sun
Has barely strength enough to banish night and makes us only think
Ever of indoors, indoors!

Why weighs my spirit so this season’s lack?
There is good to take in it I’m sure, yet here,
Stood here, this Janus’d morn, with heaven swathed in grey
I cannot find it, and must survive with heavy heart
             these bleak mid-winter days.

Enlightened [TV Series]

I have nearly finished the first series of Enlightened, a TV Series created by Laura Dern and Mike White. The series is extraordinary – even in a world where TV series have become over the last ten years a leading entertainment and art form.

It is not an easy or “fun” series, which probably accounts for its cancellation after just two seasons – I’m sort of amazed it got made in the first place – I imagine Laura Dern had something to do with it.

In fact, it is often profoundly sad – and darkly funny – as we watch the small tragedies and ironies that attend upon Amy (Laura Dern) and those around her. Amy herself is a great tragi-comic creation who remains all too human and un-enlightened despite her initial “enlightenment” at the meditation retreat at the start of episode one.

The best way to describe the series is to imagine it is what Raymond Carver might have produced had he switched from writing short story miniatures of the small desolations and tragedies of suburban America and made TV instead.

Wanted – Data Curators to Maintain Key Datasets in High-Quality, Easy-to-Use and Open Form

Wanted: volunteers to join a team of “Data Curators” maintaining “core” datasets (like GDP or ISO-codes) in high-quality, easy-to-use and open form.

  • What is the project about: Collecting and maintaining important and commonly-used (“core”) datasets in high-quality, standardized and easy-to-use form - in particular, as up-to-date, well-structured Data Packages.
    The “Core Datasets” effort is part of the broader Frictionless Data initiative.
  • What would you be doing: identifying and locating core (public) datasets, cleaning and standardizing the data and making sure the results are kept up to date and easy to use
  • Who can participate: anyone can contribute. Details on the skills needed are below.
  • Get involved: read more below or jump straight to the sign-up section.

What is the Core Datasets effort?

Summary: Collect and maintain important and commonly-used (“core”) datasets in high-quality, reliable and easy-to-use form (as Data Packages).

Core = important and commonly-used datasets e.g. reference data (country codes) and indicators (inflation, GDP)

Curate = take existing data and provide it in high-quality, reliable, and easy-to-use form (standardized, structured, open)

What Roles and Skills are Needed

We need a variety of roles from identifying new “core” datasets to packaging the data to performing quality control (checking metadata etc).

Core Skills - at least one of these skills will be needed:

  • Data Wrangling Experience. Many of our source datasets are not complex (just an Excel file or similar) and can be “wrangled” in a Spreadsheet program. What we therefore recommend is at least one of:
    • Experience with a Spreadsheet application such as Excel or (preferably) Google Docs including use of formulas and (desirably) macros (you should at least know how you could quickly convert a cell containing ‘2014’ to ‘2014-01-01’ across 1000 rows)
    • Coding for data processing (especially scraping) in one or more of python, javascript, bash
  • Data sleuthing - the ability to dig up data on the web (specific desirable skills: you know how to search by filetype in google, you know where the developer tools are in chrome or firefox, you know how to find the URL a form posts to)

Desirable Skills (the more the better!):

  • Data vs Metadata: know difference between data and metadata
  • Familiarity with Git (and Github)
  • Familiarity with a command line (preferably bash)
  • Know what JSON is
  • Mac or Unix is your default operating system (will make access to relevant tools that much easier)
  • Knowledge of Web APIs and/or HTML
  • Use of curl or similar command line tool for accessing Web APIs or web pages
  • Scraping using a command line tool or (even better) by coding yourself
  • Know what a Data Package and a Tabular Data Package are
  • Know what a text editor is (e.g. notepad, textmate, vim, emacs, …) and know how to use it (useful for both working with data and for editing Data Package metadata)

Get Involved - Sign Up Now!

We are looking for volunteer contributors to form a “curation team”.

  • Time commitment: Members of the team commit to at least 8-16h per month (though this will be an average - if you are especially busy with other things one month and do less that is fine)
  • Schedule: There is no schedule so you can contribute at any time that is good for you - evenings, weekeneds, lunch-times etc
  • Location: all activity will be carried out online so you can be based anywhere in the world
  • Skills: see above

To register your interest fill in the following form. Any questions, please get in touch directly.

Want to Dive Straight In?

Can’t wait to get started as a Data Curator? You can dive straight in and start packaging the already-selected (but not packaged) core datasets. Full instructions here:

Thank You to Our Outgoing CEO

This is a joint blog post by Open Knowledge CEO Laura James and Open Knowledge Founder and President Rufus Pollock.

In September we announced that Laura James, our CEO, is moving on from Open Knowledge and we are hiring a new Executive Director.

From Rufus: I want to express my deep appreciation for everything that Laura has done. She has made an immense contribution to Open Knowledge over the last 3 years and has been central to all we have achieved. As a leader, she has helped take us through a period of incredible growth and change and I wish her every success on her future endeavours. I am delighted that Laura will be continuing to advise and support Open Knowledge, including joining our Advisory Council. I am deeply thankful for everything she has done to support both Open Knowledge and me personally during her time with us.

From Laura: It’s been an honour and a pleasure to work with and support Open Knowledge, and to have the opportunity to work with so many brilliant people and amazing projects around the world. It’s bittersweet to be moving on from such a wonderful organisation, but I know that I am leaving it in great hands, with a smart and dedicated management team and a new leader joining shortly. Open Knowledge will continue to develop and thrive as the catalyst at the heart of the global movement around freeing data and information, ensuring knowledge creates power for the many, not the few.

Amazon Twitch Acquisition – Paying 70x Sales

Just an aside from reading the recent Amazon 10-Q. In Note 4 on acquisitions they state:

On September 25, 2014, we acquired Twitch Interactive, Inc. (“Twitch”) for approximately $842 million in cash, as adjusted for the assumption of options and other items. During the nine months ended September 30, 2014, we acquired certain other companies for an aggregate purchase price of $20 million. Acquisition activity for the nine months ended September 30, 2013 was not material. We acquired Twitch because of its community and the live streaming experience it provides. The primary reasons for our other 2014 acquisitions were to acquire technologies and know-how to enable Amazon to serve customers more effectively.

and then in th pro-forma add:

The acquired companies were consolidated into our financial statements starting on their respective acquisition dates. The aggregate net sales and operating loss of the companies acquired was $12 million and $3 million for the nine months ended September 30, 2014.

This means that Amazon acquired Twitch for approximately 70x sales! (Earnings multiple is negative since Twitch was losing money it would appear).

A Data Revolution that Works for All of Us

Many of today’s global challenges are not new. Economic inequality, the unfettered power of corporations and markets, the need to cooperate to address global problems and the unsatisfactory levels of accountability in democratic governance – these were as much problems a century ago as they remain today.

What has changed, however – and most markedly – is the role that new forms of information and information technology could potentially play in responding to these challenges.

What’s going on?

The incredible advances in digital technology mean we have an unprecedented ability to create, share and access information. Furthermore, these technologies are increasingly not just the preserve of the rich, but are available to everyone – including the world’s poorest. As a result, we are living in a (veritable) data revolution – never before has so much data – public and personal – been collected, analysed and shared.

However, the benefits of this revolution are far from being shared equally.

On the one hand, some governments and corporations are already using this data to greatly increase their ability to understand – and shape – the world around them. Others, however, including much of civil society, lack the necessary access and capabilities to truly take advantage of this opportunity. Faced with this information inequality, what can we do?

How can we enable people to hold governments and corporations to account for the decisions they make, the money they spend and the contracts they sign? How can we unleash the potential for this information to be used for good – from accelerating research to tackling climate change? And, finally, how can we make sure that personal data collected by governments and corporations is used to empower rather than exploit us?

So how should we respond?

Fundamentally, we need to make sure that the data revolution works for all of us. We believe that key to achieving this is to put “open” at the heart of the digital age. We need an open data revolution.

We must ensure that essential public-interest data is open, freely available to everyone. Conversely, we must ensure that data about me – whether collected by governments, corporations or others – is controlled by and accessible to me. And finally, we have to empower individuals and communities – especially the most disadvantaged – with the capabilities to turn data into the knowledge and insight that can drive the change they seek.

In this rapidly changing information age – where the rules of the game are still up for grabs – we must be active, seizing the opportunities we have, if we are to ensure that the knowledge society we create is an open knowledge society, benefiting the many not the few, built on principles of collaboration not control, sharing not monopoly, and empowerment not exploitation.

Announcing a Leadership Update at Open Knowledge

Today I would like to share some important organisational news. After 3 years with Open Knowledge, Laura James, our CEO, has decided to move on to new challenges. As a result of this change we will be seeking to recruit a new senior executive to lead Open Knowledge as it continues to evolve and grow.

As many of you know, Laura James joined us to support the organisation as we scaled up, and stepped up to the CEO role in 2013. It has always been her intention to return to her roots in engineering at an appropriate juncture, and we have been fortunate to have had Laura with us for so long – she will be sorely missed.

Laura has made an immense contribution and we have been privileged to have her on board – I’d like to extend my deep personal thanks to her for all she has done. Laura has played a central role in our evolution as we’ve grown from a team of half-a-dozen to more than forty. Thanks to her commitment and skill we’ve navigated many of the tough challenges that accompany “growing-up” as an organisation.

There will be no change in my role (as President and founder) and I will be here both to continue to help lead the organisation and to work closely with the new appointment going forward. Laura will remain in post, continuing to manage and lead the organisation, assisting with the recruitment and bringing the new senior executive on board.

For a decade, Open Knowledge has been a leader in its field, working at the forefront of efforts to open up information around the world and and see it used to empower citizens and organisations to drive change. Both the community and original non-profit have grown – and continue to grow – very rapidly, and the space in which we work continues to develop at an incredible pace with many exciting new opportunities and activities.

We have a fantastic future ahead of us and I’m very excited as we prepare Open Knowledge to make its next decade even more successful than its first.

We will keep everyone informed in the coming weeks as our plans develop, and there will also be opportunities for the Open Knowledge community to discuss. In the meantime, please don’t hesitate to get in touch with me if you have any questions.

A Data API for Data Packages in Seconds Using CKAN and its DataStore

dpm the command-line ‘data package manager’ now supports pushing (Tabular) Data Packages straight into a CKAN instance (including pushing all the data into the CKAN DataStore):

dpm ckan {ckan-instance-url}

This allows you, in seconds, to get a fully-featured web data API – including JSON and SQL-based query APIs:

dpm ckan demo

View fullsize

Once you have a nice web data API like this we can very easily create data-driven applications and visualizations. As a simple demonstration, there’s the CKAN Data Explorer (example with IMF data - see below).

Where Can I Find a CKAN instance to Upload to?

If you’re looking for a CKAN site to upload your Data Packages to we recommend the DataHub which is community-run and free. To upload to the DataHub you’ll want to.

  1. Configure the DataHub CKAN instance in your .dpmrc

    url =
    apikey = your-api-key
  2. Upload your Data Package

    dpm ckan datahub --owner_org=your-organization

    You have to set the owner organization as all datasts on the DataHub need an owner organization.

One I Did Earlier

Here’s a live example of one “I did earlier”:

Context: a big motivation (personally) for doing this is that I’d like to see a nice web data API available for the “Core” Data Packages we’re creating as part of the Frictionless Data effort. If you’re interested in helping, get in touch.

Labs newsletter: 5 June, 2014

Welcome back to the OKFN Labs! Members of the Labs have been building tools, visualizations, and even new data protocols—as well as setting up conferences and events. Read on to learn more.

If you’d like to suggest a piece of news for next month’s newsletter, leave a comment on its GitHub issue.


Thomas Levine has been working on an innovative new approach to searching tabular data, commasearch.

Unlike a normal search engine, where you submit words and get pages of words back, with commasearch, you submit spreadsheets and get spreadsheets in return.

What does that mean, and how does it work? Check out Thomas’s excellent blog post “Pagerank for Spreadsheets” to learn more.

GitHub diffs for CSV files

Submitted by Paul Fitzpatrick.

GitHub has added CSV viewing support in their web interface, which is fantastic, but it still doesn’t handle changes well. If you use Chrome, and want lovely diffs, check out James Smith’s CSVHub extension (blogpost and screenshot). The diffs are produced using the daff library, available in javascript, ruby, php, and python3.

Textus Wordpress plugin

Update from Iain Emsley.

The Open Literature project to provide a Wordpress plugin back-end for the Textus viewer has made new progress.

This project’s goal was to keep the existing Textus frontend—which has been split off as its own project by Rufus Pollock—and replace the backend with a Wordpress plugin, to make it easier to deploy. A version of this plugin backend is now available.

The new plugin acts as a stand-alone module that can be enabled and disabled as required by the administrative user. It creates a new Wordpress post type called “Textus” which is available as part of the menu, giving the user a place to upload text and annotation files using the Media uploader.

If you are interested in the project, check out its issues and discussion on the Open Humanities list.

Data protocols: updates

Data Protocols, the Labs’s set of lightweight standards and patterns for open data, has had a couple of interesting developments.

The JSON Table Schema protocol has just added support for constraints (i.e. validation), thanks to Leigh Dodds. This adds a constraints attribute containing requirements on the content of fields. See the full list of valid constraints on the JSON Table Schema site.

The Data Package Manager tool for Data Packages is shaping up nicely: the install and init commands have now been implemented. You can see an animated GIF of the former in the issue thread.

AnnotatorJS: new home

Annotator is “an open-source JavaScript library to easily add annotation functionality to any webpage”.

The project now lives on its own domain at Check it out and see how easy it is to add comments and notes to your pages!


Data makers everywhere will want to check out csv,conf, a fringe event of Open Knowledge Festival 2014 taking place in Berlin on 15 July.

csv,conf is a non-profit community conference that will “bring together data makers/doers/hackers from backgrounds like science, journalism, open government and the wider software industry to share tools and stories”.

Tickets are $75, $50 with an OKFest ticket. If you can make it to Berlin in July and you’re into “advancing the art of data collaboration”, come join in!

Steve Wynn on Impact of QE on Businesses and Consumers

Saw this nugget buried in a recent earnings call of Wynn Resorts Management. This is Steve Wynn responding to a caller question:

Well, we finished our financing recently. The last tranche was a $750,000 — $750 million bond. We sold it at 5.09 with no covenants nonrecourse to the parent. And that brought our total financing for Cotai to $3,850,000,000 at an average cost of 3.3%. Or to put it another way, we rented the $3.85 billion for $125 million.

Now on one hand, as a businessman, I’m thrilled. Never dreamt that we would see anything so tasty and wonderful as that. On the other hand, it’s a reflection of questionable fiscal and monetary policy in the United States that is artificially depressed interest rates because of quantitative easing by the Fed, which is also sort of killing the value of the dollar and the living standard of the working people.

So the good news is, if you’re a high-class borrower with good credit rating, this is one of the most tastiest seasons of all time for 2 reasons. You’re borrowing money at artificially depressed rates. And you’re most likely going to pay them back with 85-cent dollars.

It’s a perfect storm for a businessperson unless you look at the truth of the matter and the impact it has on your customers and your employees. And that’s a much darker story. It doesn’t lend itself to a soundbite, but it’s — for every businessman in America and any economist that has their heads screwed on right, it’s an ominous situation.

But in terms of our moment in history, in commercial history and our projects in Cotai, along with our colleagues in the industry, it’s nirvana. Capital structure now is — these are mostly at the Venetian and the Wynn, things of beauty. They’re lovely, better than you could ever want. I mean, they’ve got everything, low interest rates, long maturities, low covenants. What else do you want? I mean, it’s great.

If you look at it from our point of view, look at it from a consumers’ point of view or a working person’s point of view, who’s paying for all this cheap money? Well, right now, the Fed is. I thought Bernie Madoff went to jail for that. But anyway, that’s my answer about your capital structure.