miscellaneous factZ – The online home of Rufus Pollock

My experiences (with the assistance of Will Waites) of installing 4store On Ubuntu Jaunty.

No packaged versions of code (there is one in fact from Yves Raimond from mid 2009 but now out of date …), so need to get from github.

Recommend using will waites fork which adds useful features like:

• multiple connections
• triple deletion

Note I had to make various fixes to get this to compile on my ubuntu machine. See diff below.

Install standard ubuntu/debian dependencies:

• See 4store wiki
• rasqal needs to be latest version
  • Get it
  • ./configure –prefix=/usr –sysconfdir=/etc –localstatedir=/var
  • make, make install
• Now install

Now to start a DB:

• 4s-backend-setup {db-name}
• 4s-backend {db-name}

Now for the python bindings also created by will waites and which can be found here

• On my Jaunty needed to convert size_t to int everywhere
• Needed to run with latest cython (v0.12) installed via pip/easy_install
• To run tests need backend db called py4s_test (harcoded)

To run multiple backends at once you will probably need to have avahi dev libraries (not sure which!).

Diff for wwaites 4store fork (updated diff as of 2010-04-28)

diff --git a/src/backend/Makefile b/src/backend/Makefile
index 51a957c..e64eb13 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -2,7 +2,7 @@ include ../discovery.mk
 include ../rev.mk
 include ../darwin.mk

-CFLAGS = -Wall -Wstrict-prototypes -Werror -g -std=gnu99 -O2 -I.. -DGIT_REV=\"$(gitrev)\" `pkg-config --cflags raptor glib-2.0`
+CFLAGS = -Wall -Wstrict-prototypes -g -std=gnu99 -O2 -I.. -DGIT_REV=\"$(gitrev)\" `pkg-config --cflags raptor glib-2.0`
 LDFLAGS = $(ldfdarwin) $(ldflinux) -lz `pkg-config --libs raptor glib-2.0 $(avahi)`

 LIB_OBJS = chain.o bucket.o list.o tlist.o rhash.o mhash.o sort.o \
diff --git a/src/common/Makefile b/src/common/Makefile
index 9b33e94..60cd04f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -21,7 +21,7 @@ ifdef dnssd
   mdns_flags = -DUSE_DNS_SD
 endif

-CFLAGS = -std=gnu99 -fno-strict-aliasing -Wall -Werror -Wstrict-prototypes -g -O2 -I../ -DGIT_REV=\"$(gitrev)\" $(mdns_flags) `pkg-config --cflags $(pkgs)`
+CFLAGS = -std=gnu99 -fno-strict-aliasing -Wall -Wstrict-prototypes -g -O2 -I../ -DGIT_REV=\"$(gitrev)\" $(mdns_flags) `pkg-config --cflags $(pkgs)`
 LDFLAGS = $(ldfdarwin) $(lfdlinux)
 LIBS = `pkg-config --libs $(pkgs)`

diff --git a/src/frontend/results.c b/src/frontend/results.c
index 485ac31..162aa3d 100644
--- a/src/frontend/results.c
+++ b/src/frontend/results.c
@@ -381,12 +381,12 @@ fs_value fs_expression_eval(fs_query *q, int row, int block, rasqal_expression *
             return v;
         }

-        case RASQAL_EXPR_SUM:
-        case RASQAL_EXPR_AVG:
-        case RASQAL_EXPR_MIN:
-        case RASQAL_EXPR_MAX:
-        case RASQAL_EXPR_LAST:
-            return fs_value_error(FS_ERROR_INVALID_TYPE, "unsupported aggregate operation");
+        //case RASQAL_EXPR_SUM:
+        //case RASQAL_EXPR_AVG:
+        //case RASQAL_EXPR_MIN:
+        //case RASQAL_EXPR_MAX:
+        //case RASQAL_EXPR_LAST:
+        //    return fs_value_error(FS_ERROR_INVALID_TYPE, "unsupported aggregate operation");

 #endif
 

Diff to wwaites py4s (updated diff as of 2010-04-28)

diff --git a/_py4s.pxd b/_py4s.pxd
index 5251289..0e26250 100644
--- a/_py4s.pxd
+++ b/_py4s.pxd
@@ -110,7 +110,7 @@ cdef extern from "frontend/results.h":

 cdef extern from "frontend/import.h":
        int fs_import_stream_start(fsp_link *link, char *model_uri, char *mimety
-       int fs_import_stream_data(fsp_link *link, unsigned char *data, size_t co
+       int fs_import_stream_data(fsp_link *link, unsigned char *data, int count
        int fs_import_stream_finish(fsp_link *link, int *count, int *errors)

 cdef extern from "frontend/update.h":

I’ve had occasion recently to frequently work with “dates” that come in a lot of shapes and sizes including:

• Dates in distant past and future including BC/BCE dates
• Dates in a wild variety of formats: Jan 1890, January 1890, 1st Dec 1890, Spring 1890 etc
• Dates of varying precision: e.g. 1890, 1890-01 (i.e. Jan 1890), 1890-01-02
• Imprecise dates: c1890, 1890?, fl 1890 etc

Unfortunately existing support for these in python is fairly weak. I therefore authored a python FlexiDate module (part of a new swiss (army knife) package) which is focused on supporting:

1. Dates outside of Python (or DB) supported period (esp. dates < 0 AD)
2. Imprecise dates (c.1860, 18??, fl. 1534, etc)
3. Normalization of these dates to machine processable versions especially:
   • ISO 8601
   • Dates sortable in the database (in correct date order)

Background

Things we would like:

1. Dates outside of Python (or DB) supported period (esp. dates < 0 AD)
2. Imprecise dates (c.1860, 18??, fl. 1534, etc)
3. Normalization of dates to machine processable versions
4. Sortable in the database (in correct date order)
5. Human readability as dates will be re-edited/viewed by people

Not all of these requirements are satisfiable at once in a simple way.

Be clear about what we want:

1. Storage (and preservation) of “user” dates (both normal and non-normal)
2. Normalization of dates (e.g. to ~ ISO 8601)
3. Integration with database (sortability and serializability)

Solution for 1: Represent dates as strings.

Solution for 2: Have a parser (via an intermediate FlexiDate object).

Solution for 3: convert to a float.

Remark: no string based date format will sort dates correctly based on std string ordering (PF: let x,y be +ve dates and X,Y their string representations then if X -X<-Y (wrong!))

Thus we need to add some other field if we wish dates to be correctly sorted (or not worry about sorting of -ve dates …)

1. For any given date attribute have 2 actual fields:

• user version — the version edited by users
• normalized/parsed version — a version that is usable by machines

2. Store both versions in a single field but with some form of serialization.

1. Convert dates to long ints (unlimited in precision) and put this in a separate field and use that for sorting.

Comments

Initially thought that we should parse before saving into a FlexiDate format but: a) why bother b) when parsing always hard not to be lossy (in particular when converting to iso8601 using e.g. dateutil very difficult to not add info e.g. parsing 1860 can easily give us 1860-01-01 …).

References and Existing Libraries

• Excellent dateutil (which we use0
• http://wiki.python.org/moin/WorkingWithTime
• ISO 8601: http://www.iso.org/iso/dateandtime_format
  • http://code.google.com/p/pyiso8601/source/browse/trunk/iso8601/iso8601.py
• http://www.feedparser.org/docs/date-parsing.html
• http://seehuhn.de/pages/pdate

7.5/10. Spare, stark and beautifully written but hampered by a plot (and characters) that seem but weakly thought out.

7/10. Good but not more than that. It has the promise of developing into something more, something epic, but obviously, was cut short before that promise could be realized — or shown to be misplaced.

7/10. Extensive in its imagination but losing some of its power by this very fact as the stories start to blur and something of its early intensity is lost as we head into the closing sections. Overall a very dark (and probably correct) vision of the immigrant experience in the US: all prejudice, death and broken families — at least as I can remember, there was not one happy relationship or family recorded in these 380 odd pages.

Interesting, disarmingly honest, but not, ultimately, entirely convincing that ’self-organized criticality’ is the key to “How Nature Works”.

7/10. Having now finished the final volume of Skidelsky’s trilogy it is clear that the first volume was the best. This is not necessarily a reflection on Skidelsky’s efforts but on the nature of the subject matter — the first section of Keynes’ life, with its natural intertwining of life, friends and work, is the most suited to the biographical form. Here instead, by the nature of Keynes own activities he is forced to confine attention almost entirely to the work, and to work that was almost entirely of a bureaucratic or diplomatic nature.

It can be difficult in such to circumstances to sustain interest over long narrations of a particular policy debate within the British Government or the progress of a particular negotiation with the United States (which formed the main part of Keynes activities). The form of the book (a biography) in these circumstances exacerbates the problem. As biography one needs to keep things ‘personal’ focusing on Keynes’ personal experience together with the sketches of the personalities he encountered. This often may result in the underlying issues getting lost. If on the other hand one takes a more analytical, historical, approach in which the issues under discussion are made central with appropriate background supplied and analysis provided then one is rapidly leaving the realm of biography for that of (economic) history. Not only is this departing from the book’s ‘core mission’ but also may make things rather dry for the non-specialist. To my mind this tension is not adequately resolved, and, just as with Vol. 2, in my view, a more detailed historical/analytical treatment would have been better — along the lines of the masterly section in the book’s concluding chapter where Skidelsky summarizes Keynes (Economic) legacy and its impact on post-war posterity (conclusion: ‘Keynesianism’ was of little importance).

Comments on previous volumes:

• http://www.rufuspollock.org/2006/10/30/john-maynard-keynes-hopes-betrayed-1883-1920-by-robert-skidelsky/
• http://www.rufuspollock.org/2006/12/05/john-maynard-keynes-the-economist-as-saviour-1920-1937-by-robert-skidelsky/

7/10. Well written and fascinating, particularly in its clear demonstration of the way the French just ‘gave up’ (both generally in the inter-war period and in 1940 itself). I would have preferred more analytical clarity regarding exactly when things went wrong and why — at some moments Horne seems to be suggesting that a sufficiently active response by the French in the first few days (between the 12th and the 14th of May) might have made a decisive difference in reversing the tide, at others that the Germans superiority in weapons, tactics and men (quality, not necessarily quantity) meant that France was doomed from the start. The relative success of the few British salleys against the Germans make me incline more towards the former possibility. I also think this view may be warranted by the concerns evinced so frequently by those within the German General Staff (and Hitler himself) about the vulnerability of their flanks, as well as the huge convoys through the Ardennes, in the first few decisive days of the battle. If this is the case, it shows that what is today considered one of the greatest and most brilliant military victories of all time might well have ended up as another failed Schlieffen plan.

5.5/10. Disappointing though perhaps because of the high expectations engendered by the book’s reputation. To my mind, the book has not dated well and the general insights regarding working practices set out in the afterword seem debatable (the Machine referred to in the title is not the computer but the organizationthat built it).

6/10. Interesting though, as usual with these things, the book is long on personality, anecdote and rumour, and a bit too short on factual details of what exactly happened (roughly LTCM made ever more leveraged bets on ever less liquid and well understood products and got mashed up when a crisis came along and everyone ran for the door).

Interesting side points that are alleged (but not confirmed) include Goldman Sachs using their position as potential buyer/rescuer to gain privileged access to LTCM’s position and then front-running them (i.e. selling those positions out from under LTCM thereby helping Goldman and harming LTCM).