miscellaneous factZ – The online home of Rufus Pollock

My experiences (with the assistance of Will Waites) of installing 4store On Ubuntu Jaunty.

No packaged versions of code (there is one in fact from Yves Raimond from mid 2009 but now out of date …), so need to get from github.

Recommend using will waites fork which adds useful features like:

• multiple connections
• triple deletion

Note I had to make various fixes to get this to compile on my ubuntu machine. See diff below.

Install standard ubuntu/debian dependencies:

• See 4store wiki
• rasqal needs to be latest version
  • Get it
  • ./configure –prefix=/usr –sysconfdir=/etc –localstatedir=/var
  • make, make install
• Now install

Now to start a DB:

• 4s-backend-setup {db-name}
• 4s-backend {db-name}

Now for the python bindings also created by will waites and which can be found here

• On my Jaunty needed to convert size_t to int everywhere
• Needed to run with latest cython (v0.12) installed via pip/easy_install
• To run tests need backend db called py4s_test (harcoded)

To run multiple backends at once you will probably need to have avahi dev libraries (not sure which!).

Diff for wwaites 4store fork (updated diff as of 2010-04-28)

diff --git a/src/backend/Makefile b/src/backend/Makefile
index 51a957c..e64eb13 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -2,7 +2,7 @@ include ../discovery.mk
 include ../rev.mk
 include ../darwin.mk

-CFLAGS = -Wall -Wstrict-prototypes -Werror -g -std=gnu99 -O2 -I.. -DGIT_REV=\"$(gitrev)\" `pkg-config --cflags raptor glib-2.0`
+CFLAGS = -Wall -Wstrict-prototypes -g -std=gnu99 -O2 -I.. -DGIT_REV=\"$(gitrev)\" `pkg-config --cflags raptor glib-2.0`
 LDFLAGS = $(ldfdarwin) $(ldflinux) -lz `pkg-config --libs raptor glib-2.0 $(avahi)`

 LIB_OBJS = chain.o bucket.o list.o tlist.o rhash.o mhash.o sort.o \
diff --git a/src/common/Makefile b/src/common/Makefile
index 9b33e94..60cd04f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -21,7 +21,7 @@ ifdef dnssd
   mdns_flags = -DUSE_DNS_SD
 endif

-CFLAGS = -std=gnu99 -fno-strict-aliasing -Wall -Werror -Wstrict-prototypes -g -O2 -I../ -DGIT_REV=\"$(gitrev)\" $(mdns_flags) `pkg-config --cflags $(pkgs)`
+CFLAGS = -std=gnu99 -fno-strict-aliasing -Wall -Wstrict-prototypes -g -O2 -I../ -DGIT_REV=\"$(gitrev)\" $(mdns_flags) `pkg-config --cflags $(pkgs)`
 LDFLAGS = $(ldfdarwin) $(lfdlinux)
 LIBS = `pkg-config --libs $(pkgs)`

diff --git a/src/frontend/results.c b/src/frontend/results.c
index 485ac31..162aa3d 100644
--- a/src/frontend/results.c
+++ b/src/frontend/results.c
@@ -381,12 +381,12 @@ fs_value fs_expression_eval(fs_query *q, int row, int block, rasqal_expression *
             return v;
         }

-        case RASQAL_EXPR_SUM:
-        case RASQAL_EXPR_AVG:
-        case RASQAL_EXPR_MIN:
-        case RASQAL_EXPR_MAX:
-        case RASQAL_EXPR_LAST:
-            return fs_value_error(FS_ERROR_INVALID_TYPE, "unsupported aggregate operation");
+        //case RASQAL_EXPR_SUM:
+        //case RASQAL_EXPR_AVG:
+        //case RASQAL_EXPR_MIN:
+        //case RASQAL_EXPR_MAX:
+        //case RASQAL_EXPR_LAST:
+        //    return fs_value_error(FS_ERROR_INVALID_TYPE, "unsupported aggregate operation");

 #endif
 

Diff to wwaites py4s (updated diff as of 2010-04-28)

diff --git a/_py4s.pxd b/_py4s.pxd
index 5251289..0e26250 100644
--- a/_py4s.pxd
+++ b/_py4s.pxd
@@ -110,7 +110,7 @@ cdef extern from "frontend/results.h":

 cdef extern from "frontend/import.h":
        int fs_import_stream_start(fsp_link *link, char *model_uri, char *mimety
-       int fs_import_stream_data(fsp_link *link, unsigned char *data, size_t co
+       int fs_import_stream_data(fsp_link *link, unsigned char *data, int count
        int fs_import_stream_finish(fsp_link *link, int *count, int *errors)

 cdef extern from "frontend/update.h":

Deliverance is a great library that lets you easily re-theme external websites on the fly. Designed as WSGI middleware, it can be easily combined with some proxying to integrate a bunch of websites together

You can use deliverance plus proxying out-of-the-box using the deliverance-proxy command. However, I was interested in using Deliverance as middleware from code. This turned out to be none too trivial to do — all the examples on the internet seemed to focus on using deliverance-proxy or using it in an ini file.

After much wrestling, most notably with odd issues with gzipped (deflated) content I got it working and you can find a demo implementation (see demo.py and README.txt) here:

http://rufuspollock.org/code/deliverance/

I should also mention the following sources which were all of help in my quest:

• http://codespeak.net/svn/z3/deliverance/sandbox/ianb/ploneconf2008/index.txt
• http://www.gawel.org/weblog/en/2008/12/skinning-with-pyquery-and-deliverance
• http://macadames.wordpress.com/2009/05/23/some-deliverance-tips/
• http://www.coactivate.org/projects/deliverance/lists/deliverance-discussion/archive/2009/05/1241444934295/forum_view
• http://www.sixfeetup.com/blog/2009/4/27/deploying-plone-and-zine-together-with-deliverance-using-repoze

Instructions on using sqlalchemy migrate with Pylons, especially to convert an existing pylons project to use sqlalchemy migrate

This is based off several excellent sources including this guide and these threads.

One important point to note is that you are likely to end up with two versions of your model tables: one in yourapp/model and one in yourapp/migration/versions/*.py with the former representing your tables at HEAD and the second containing upgrade/downgrade scripts whose final result is HEAD. This duplication is a bit annoying and I discuss how it can be avoided below.

1. Install sqlalchemy migrate for your project e.g.

  pip -E {your-virtualenv} install sqlalchemy-migrate
  # or
  easy_install sqlalchemy-migrate

NB: latest version of migrate are only compatible with sqlalchemy >= 0.5 (for previous version of sqlalchemy you need migrate <= 0.4.5)

2. Create the migrate repository (i.e. store for upgrade scripts …).

In your project directory

  migrate create myapp/migration/ "MyApp"

Now create a temporary helper script:

  migrate manage dbmanage.py --repository=myapp/migration/ --url={your-sqlalchemy-db-uri}

3. Set up db version control

  python dbmanage.py version_control

Check the current version (should be 0)

  python dbmanage.py version

4. Create a migration script for your existing db

  python dbmanage.py script "Add existing tables"

This will create a script in myapp/migration/versions/001addexisting_tables.py

Copy into that file the definition for all your existing tables (and other database stuff such as constraints) and then create those tables in the upgrade() function (and delete them in downgrade()).

That’s it! (in theory)

Additional Issues

1. Duplication of model/db code

You now have two places for model/db code:

1. Your migration scripts
2. Your actual model

This doesn’t have to be a problem but it is an obvious way for bugs to creep especially when some people start by creating their DB from the model code and others from the migration scripts.

Warning: this method will not work if do stuff in your table creation that is not persisted into the actual DB sql (e.g. column default values based on a function, custom db types …).

One way to avoid the duplication is to have all table creation and alteration confined to your migration scripts and then have your model tables set up directly from the DB using the autoload=True option. The one disadvantage of this is you can’t see the complete DB set up in one places as tables construction may be spread over several migrate scripts. One solution to this is provided by the experimental ‘create_model’ command which dumps the current DB model in the required sqlalchemy table code.

More discussion in this migrate-users thread

Bringing the Migration DB up to date

If you do set up your DB (from new) directly from your model code rather than the migration scripts then this requires that you set up the migration stuff and update the migrate version to the correct number. (I note there is an experimental updatedbto_model command which is supposed to do this for you). You can do this as follows (assuming your migrate repository is at YOURAPP:

      from migrate.versioning.api import version_control, version
      import YOURAPP.migration.versions
      v = version(YOURAPP.migration.__path__[0])
      # log.info( "Setting current version to '%s'" % v )
      # url is your sqlalchemy db url 
      version_control(url, YOURAPP.migration.__path__[0], v)

Extras

• Should wrap upgrade/downgrade in transactions. I found one way to do this here but testing indicated this didn’t work for me (rollback was not working properly when there was an error)

I’ve had occasion recently to frequently work with “dates” that come in a lot of shapes and sizes including:

• Dates in distant past and future including BC/BCE dates
• Dates in a wild variety of formats: Jan 1890, January 1890, 1st Dec 1890, Spring 1890 etc
• Dates of varying precision: e.g. 1890, 1890-01 (i.e. Jan 1890), 1890-01-02
• Imprecise dates: c1890, 1890?, fl 1890 etc

Unfortunately existing support for these in python is fairly weak. I therefore authored a python FlexiDate module (part of a new swiss (army knife) package) which is focused on supporting:

1. Dates outside of Python (or DB) supported period (esp. dates < 0 AD)
2. Imprecise dates (c.1860, 18??, fl. 1534, etc)
3. Normalization of these dates to machine processable versions especially:
   • ISO 8601
   • Dates sortable in the database (in correct date order)

Background

Things we would like:

1. Dates outside of Python (or DB) supported period (esp. dates < 0 AD)
2. Imprecise dates (c.1860, 18??, fl. 1534, etc)
3. Normalization of dates to machine processable versions
4. Sortable in the database (in correct date order)
5. Human readability as dates will be re-edited/viewed by people

Not all of these requirements are satisfiable at once in a simple way.

Be clear about what we want:

1. Storage (and preservation) of “user” dates (both normal and non-normal)
2. Normalization of dates (e.g. to ~ ISO 8601)
3. Integration with database (sortability and serializability)

Solution for 1: Represent dates as strings.

Solution for 2: Have a parser (via an intermediate FlexiDate object).

Solution for 3: convert to a float.

Remark: no string based date format will sort dates correctly based on std string ordering (PF: let x,y be +ve dates and X,Y their string representations then if X -X<-Y (wrong!))

Thus we need to add some other field if we wish dates to be correctly sorted (or not worry about sorting of -ve dates …)

1. For any given date attribute have 2 actual fields:

• user version — the version edited by users
• normalized/parsed version — a version that is usable by machines

2. Store both versions in a single field but with some form of serialization.

1. Convert dates to long ints (unlimited in precision) and put this in a separate field and use that for sorting.

Comments

Initially thought that we should parse before saving into a FlexiDate format but: a) why bother b) when parsing always hard not to be lossy (in particular when converting to iso8601 using e.g. dateutil very difficult to not add info e.g. parsing 1860 can easily give us 1860-01-01 …).

References and Existing Libraries

• Excellent dateutil (which we use0
• http://wiki.python.org/moin/WorkingWithTime
• ISO 8601: http://www.iso.org/iso/dateandtime_format
  • http://code.google.com/p/pyiso8601/source/browse/trunk/iso8601/iso8601.py
• http://www.feedparser.org/docs/date-parsing.html
• http://seehuhn.de/pages/pdate

A new version (v1.2) of my python script for converting markdown to latex is now done. markdown2latex (renamed from mkdn2latex) has been extensively refactored to become a proper python-markdown extension. This means it can be used seemlessly alongside plain markdown conversion, as well as independently whether as a module or, in its classic form, from the command line.

In addition for ease of installation it has also been turned into a proper python package and registered on pypi so you can just do:

$ easy_install markdown2latex

Alternatively you can still get it straight from the repository at:

http://knowledgeforge.net/okftext/svn/trunk/python/markdown2latex/

An (ongoing) summary of my experience with some of the utilities available for plotting from a python perspective.

Last updated: 2008-03-06

Ploticus

• (+) Fast, powerful, mature, well-documented
• (-) Not python based

C-based rather than python-based but fast and powerful. There is a (fairly crude) set of python bindings available here: http://www.srcc.lsu.edu/~davids/ploticus_module.html. Alternatively one can just call the ploticus command from a python script.

Matplotlib

• (+) Fairly powerful, mature, well-documented, nice pure python API
• (-) A little slow; requires a backend to be installed (so installation on a server is a problem)
• Could support object-orientation better

PyChart

http://home.gna.org/pychart/

• (+) Pure python, quite simple to use, good documentation
• (-) Not quite as nice looking or as powerful as e.g. ploticus

Biggles

http://biggles.sourceforge.net/

• last updated: 2004-03-08
• looks fine but does not seem to be actively developed any longer

Example

See: http://home.gna.org/pychart/examples/index.html. This is the bar/line example from there:

from pychart import *
theme.get_options()

data = [(10, 20, 30), (20, 65, 33),
    (30, 55, 30), (40, 45, 51),
    (50, 25, 27), (60, 75, 30)]

ar = area.T(size = (150,120),
            y_grid_interval=10,
            x_axis=axis.X(label="X label", label_offset=(0,-7)),
            y_axis=axis.Y(label="Y label"),
            legend = legend.T(), y_range = (0, None))

ar.add_plot(bar_plot.T(label="foo", data=data),
            line_plot.T(label="bar", data=data, ycol=2))
ar.draw()

import readline
readline.write_history_file('my_history.py')

I’ve been thinking about how to have a versioned domain model similar to the way we have versioned filesystems (e.g. subversion) for over two years. Over the last few months whatever bits of free time I’ve had have gone into developing a prototype built on top of sqlobject and I’ve now got a rough and ready (but fully functional) library:

http://project.knowledgeforge.net/ckan/svn/vdm/branches/sqlobj/

A demo of how it is used is best shown by the tests:

http://project.knowledgeforge.net/ckan/svn/vdm/branches/sqlobj/vdm/dm_test.py

Why be tied to SQLObject: obviously being so directly tied to sqlobject is not such a great thing but I intentionally chose to build on it because so many people will already be writing their domain models using SQLObject.

I’ve updated mkdn2latex the python script which converts markdown to latex (see also the original release announcement). Changes include:

• Support for markdown code blocks and html pre/code blocks generally using latex verbatim
• Verified compatibility with markdown 1.6
• A few minor bugfixes

Following my release of the markdown to latex script I’ve had a few enquiries from people asking about integrating mathematics with markdown generally (e.g. for web output as well as for output to latex). I’d already been using mathematics in markdown and then processing to html before I wrote the mkdn2latex script and in a world where one didn’t need to produce nice pdfs for conferences and journals it would be my preferred format. Anyway here’s a summary of the ways in which you can add mathematics support to basic markdown:

Mathematics in Markdown Howto

There are two possible options for pure web output with mathematics using markdown:

1. Add asciimathml/latexmathml support into the html files in which the markdown output will be inserted (these are javascript files to convert latex like mathematics to mathml on the fly see 1 and 2 — note that i recommend latexmathml as it is closer to latex).

2. Convert to latex and then convert to html use latex2html or similar

For pure html work I’ve used approach (1) up until now. This requires no change to your markdown processor only that you link to the right asciimathml/latexmathml javascript in the resulting html document (you can see an example in this simple wrapper around the basic markdown script)

In both cases you will want to insert math sections into your source markdown file. My convention is that any maths whether in paragraph or out should be enclosed in double dollars as in: \$\$ …. \$\$ (note that the \ should not be there but because latexmathml script is being used on this blog we need to escape one of the $ so that the text actually displays — as opposed to being render as mathematics). This is slightly different from the standard asciimathml/latexmathml conventions which just use a single $). I’ve made the necessary modifications (very minor) to asciimathml and latexmathml and you can find them at:

http://knowledgeforge.net/okftext/svn/trunk/js/

(look in the src subdirectories)

To summarize:

1. Create your markdown documents as normal.

2. To add mathematics just add it as for latex but using $\$ as delimiters. (If you plan to use javascript approach read up on those scripts to see what parts of latex they support). For example this would be fine (again ignore the backslashes):

    A simple markdown file, $$x$$, with some mathematics:
   
   
    \$\$ x^{2} + y^{2} = z^{2} \$\$
   
   
    A new paragraph after a block of mathematics ...
   

3. Then:

   1. EITHER convert to markdown as usual but then insert link to modified latexmathml.js in your html documents (or if using original latexmathml just convert $$ to $ everywhere)
   2. OR convert markdown to latex using my script and then use latex2html