• Release: http://pypi.python.org/pypi/datapkg
• Docs: http://packages.python.org/datapkg/

There’s a quick getting started section below (also see the docs).

About the release

This release brings substantial improvements to the download functionality of datapkg including support for extending the download system via plugins. The full changelog below has more details and here’s an example of the new download system being used to download material selectively from the COFOG package on CKAN.

# download metadata and all resources from cofog package to current directory
# Resources to retrieve will be selected interactively
download ckan://cofog .

# download all resources
# Note need to quote *
download ckan://name path-on-disk "*"

# download only those resources that have format 'csv' (or 'CSV')
download ckan://name path-on-disk csv

For more details see the documentation of the download command:

datapkg help download

Get started fast

# 1. Install: (requires python and easy_install)
$ easy_install datapkg
# Or, if you don't like easy_install
$ pip install datapkg or even the raw source!

# 2. [optional] Take a look at the manual
$ datapkg man

# 3. Search for something
$ datapkg search ckan:// gold
gold-prices -- Gold Prices in London 1950-2008 (Monthly)

# 4. Get some data
# This will result in a csv file at /tmp/gold-prices/data
$ datapkg download ckan://gold-prices /tmp

Find out more » — including how to create, register and distribute your own ‘data packages’.

Changelog

• ResourceDownloader objects and plugin point (#964)
• Refactor PackageDownloader to use ResourceDownloader and support Resource filtering
• Retrieval options for package resourcs (#405). Support selection of resources to download (on command line or API) via glob style patterns or user interaction.

• Download: http://pypi.python.org/pypi/pywordpress/
• Source code: https://bitbucket.org/rgrp/pywordpress/

Usage

Command line

Check out the commands::

wordpress.py -h 

You will need to create a config with the details (url, login) of the wordpress instance you want to work with::

cp config.ini.tmpl config.ini
# now edit away ...
vim config.ini

Python library

Read the code documentation::

>>> from pywordpress import WordPress
>>> help(WordPress)

1. TexCount.pl
2. untex + wc
3. wc
4. pdf file

wc

$ wc -w file.tex

This is very simple but is pretty inaccurate since wc has no awareness of tex commands or mathematics (which results in overcounting) and does not expand things like bibliographies (which results in undercounting). Overall the result is likely to be a substantial overcount.

Look at the resulting pdf file.

$ pdftotext file.pdf - | egrep -E '\w\w\w+' | iconv -f ISO-8859-15 -t UTF-8 | wc

More sophisticated but in my experience results in grossly overestimated wordcounts due to inability to deal with mathematics and issues with pdftotext (lots of words get broken up that shouldn’t be).

TexCount.pl

Get it from: http://folk.uio.no/einarro/Comp/texwordcount.html

This seemed to be pretty good.

untex + wc

$ untex file.tex | wc

Again likely to overcount for mathematics and fairly limited removal of tex commands (though may undercount due to omission of citation/biblio type stuff).

First let’s make the script that starts openoffice available in a convenient way e.g. by symlinking into ~/bin or /usr/bin:

 $ cd ~/bin
 $ ln -s /Applications/OpenOffice.org\ 2.0.app/Contents/openoffice.org/program/soffice ./

Now you can do stuff like:

 $ soffice -help

You’ll see there are different switches which allow you to start a text document, a spreadsheet etc. One annoyance to note is that if you get soffice to load a file by doing:

$ soffice [options] ${filename}

The application it will use (writer, calc, math …) will depend solely on the extension of the filename and will ignore any options you give it. So e.g. if you do:

$ soffice -writer some.csv

Then this will load in calc even though the -writer option was given. For more details (on this very old bug) see:

http://www.openoffice.org/servlets/ReadMsg?list=allbugs&msgNo=94354

Fortunately this isn’t too much of a problem since the extension mapping is pretty reasonable.

# if you just wanted trunk replace with this
# svndump {start-repo} | svndumpfilter include trunk 
svnadmin dump {start-repo} > my-dump
# copy file to main server and change to web-user
svnadmin load --parent-dir {some-dir-name} {old-repo} < my-dump

Example

Suppose you had the following in a file called example.csv:

"YEAR","PH","RPH","RPH_1","LN_RPH","LN_RPH_1","HH","LN_HH"
1971,7.8523,43.9168,42.9594,3.7822,3.7602,16185,9.691843

1972,10.5047,55.1134,43.9168370988587,4.0093,3.7822,16397,9.704855

Running:

 $ ./csv2ascii.py example.csv

Would result in:

+------+------+------+------+------+------+------+------+
| YEAR |  PH  | RPH  |RPH_1 |LN_RPH|LN_RPH|  HH  |LN_HH |
+------+------+------+------+------+------+------+------+
| 1971 |7.8523|43.916|42.959|3.7822|3.7602|16185 |9.6918|
+------+------+------+------+------+------+------+------+
| 1972 |10.504|55.113|43.916|4.0093|3.7822|16397 |9.7048|
+------+------+------+------+------+------+------+------+

1. Use sed:

     sed -i 's/foo/foo_bar/g'  *.html
   

2. use the old perl hack:

     perl -w -pi~ -e 's/foo/bar/' [files]
   

   Notes: -p: loop, -i edit files in place (backup with extension if supplied), -w enable warnings

3. Install rpl

   • http://freshmeat.net/projects/rpl/
   • port install rpl (mac osx)
   • apt-get install rpl (debian/ubuntu)

Combining either (1) or (2) with find is pretty powerful. E.g. to do a find and replace on all html files in all subdirectories:

     perl -w -pi -e 's/foo/bar/' `find <path> -name '*.html'`

Published in 1999 and though now a little dated this is still a great essay. It’s full of Stephenson’s wit and detailed down-to-earth explanations. My favourite part is the brilliant and hilarious ‘os as car’ metaphor:

The analogy between cars and operating systems is not half bad, and so let me run with it for a moment, as a way of giving an executive summary of our situation today.

Imagine a crossroads where four competing auto dealerships are situated. One of them (Microsoft) is much, much bigger than the others. It started out years ago selling three-speed bicycles (MS-DOS); these were not perfect, but they worked, and when they broke you could easily fix them.

There was a competing bicycle dealership next door (Apple) that one day began selling motorized vehicles–expensive but attractively styled cars with their innards hermetically sealed, so that how they worked was something of a mystery.

The big dealership responded by rushing a moped upgrade kit (the original Windows) onto the market. This was a Rube Goldberg contraption that, when bolted onto a three-speed bicycle, enabled it to keep up, just barely, with Apple-cars. The users had to wear goggles and were always picking bugs out of their teeth while Apple owners sped along in hermetically sealed comfort, sneering out the windows. But the Micro-mopeds were cheap, and easy to fix compared with the Apple-cars, and their market share waxed.

Eventually the big dealership came out with a full-fledged car: a colossal station wagon (Windows 95). It had all the aesthetic appeal of a Soviet worker housing block, it leaked oil and blew gaskets, and it was an enormous success. A little later, they also came out with a hulking off-road vehicle intended for industrial users (Windows NT) which was no more beautiful than the station wagon, and only a little more reliable.

Since then there has been a lot of noise and shouting, but little has changed. The smaller dealership continues to sell sleek Euro-styled sedans and to spend a lot of money on advertising campaigns. They have had GOING OUT OF BUSINESS! signs taped up in their windows for so long that they have gotten all yellow and curly. The big one keeps making bigger and bigger station wagons and ORVs.

On the other side of the road are two competitors that have come along more recently.

One of them (Be, Inc.) is selling fully operational Batmobiles (the BeOS). They are more beautiful and stylish even than the Euro-sedans, better designed, more technologically advanced, and at least as reliable as anything else on the market–and yet cheaper than the others.

With one exception, that is: Linux, which is right next door, and which is not a business at all. It’s a bunch of RVs, yurts, tepees, and geodesic domes set up in a field and organized by consensus. The people who live there are making tanks. These are not old-fashioned, cast-iron Soviet tanks; these are more like the M1 tanks of the U.S. Army, made of space-age materials and jammed with sophisticated technology from one end to the other. But they are better than Army tanks. They’ve been modified in such a way that they never, ever break down, are light and maneuverable enough to use on ordinary streets, and use no more fuel than a subcompact car. These tanks are being cranked out, on the spot, at a terrific pace, and a vast number of them are lined up along the edge of the road with keys in the ignition. Anyone who wants can simply climb into one and drive it away for free.

Customers come to this crossroads in throngs, day and night. Ninety percent of them go straight to the biggest dealership and buy station wagons or off-road vehicles. They do not even look at the other dealerships.

Of the remaining ten percent, most go and buy a sleek Euro-sedan, pausing only to turn up their noses at the philistines going to buy the station wagons and ORVs. If they even notice the people on the opposite side of the road, selling the cheaper, technically superior vehicles, these customers deride them cranks and half-wits.

The Batmobile outlet sells a few vehicles to the occasional car nut who wants a second vehicle to go with his station wagon, but seems to accept, at least for now, that it’s a fringe player.

The group giving away the free tanks only stays alive because it is staffed by volunteers, who are lined up at the edge of the street with bullhorns, trying to draw customers’ attention to this incredible situation. A typical conversation goes something like this:

Hacker with bullhorn: “Save your money! Accept one of our free tanks! It is invulnerable, and can drive across rocks and swamps at ninety miles an hour while getting a hundred miles to the gallon!”

Prospective station wagon buyer: “I know what you say is true…but…er…I don’t know how to maintain a tank!”

Bullhorn: “You don’t know how to maintain a station wagon either!”

Buyer: “But this dealership has mechanics on staff. If something goes wrong with my station wagon, I can take a day off work, bring it here, and pay them to work on it while I sit in the waiting room for hours, listening to elevator music.”

Bullhorn: “But if you accept one of our free tanks we will send volunteers to your house to fix it for free while you sleep!”

Buyer: “Stay away from my house, you freak!”

Bullhorn: “But…”

Buyer: “Can’t you see that everyone is buying station wagons?”

From apt-cache show:

wdiff’ is a front-end to GNU diff. It compares two files, finding which words have been deleted or added to the first in order to create the second. It has many output formats and interacts well with terminals and pagers (notably with less').wdiff’ is particularly useful when two texts differ only by a few words and paragraphs have been refilled.