miscellaneous factZ – The online home of Rufus Pollock

Deliverance is a great library that lets you easily re-theme external websites on the fly. Designed as WSGI middleware, it can be easily combined with some proxying to integrate a bunch of websites together

You can use deliverance plus proxying out-of-the-box using the deliverance-proxy command. However, I was interested in using Deliverance as middleware from code. This turned out to be none too trivial to do — all the examples on the internet seemed to focus on using deliverance-proxy or using it in an ini file.

After much wrestling, most notably with odd issues with gzipped (deflated) content I got it working and you can find a demo implementation (see demo.py and README.txt) here:

http://rufuspollock.org/code/deliverance/

I should also mention the following sources which were all of help in my quest:

• http://codespeak.net/svn/z3/deliverance/sandbox/ianb/ploneconf2008/index.txt
• http://www.gawel.org/weblog/en/2008/12/skinning-with-pyquery-and-deliverance
• http://macadames.wordpress.com/2009/05/23/some-deliverance-tips/
• http://www.coactivate.org/projects/deliverance/lists/deliverance-discussion/archive/2009/05/1241444934295/forum_view
• http://www.sixfeetup.com/blog/2009/4/27/deploying-plone-and-zine-together-with-deliverance-using-repoze

I wanted to do a reverse proxy to wordpress.com in order to integrate an existing wordpress.com blog into an existing site. This turned out to be a little trickier than I’d thought due to wordpress.com’s usage of gzip deflation of their output.

Figured this out thanks to sound advice here and here:

  ...

  <Proxy *>
        Allow from .mysite.com
  </Proxy>
  ProxyPass / myblog.wordpress.com
  ProxyPassReverse / http://myblog.wordpress.com/
  ProxyHTMLURLMap http://myblog.wordpress.com/ /
  <Location />
    SetOutputFilter proxy-html
    # get rid of Content-Encoding at wordpress end
    # To use this you'll need to do (on debian) a2enmod headers
    RequestHeader unset  Accept-Encoding

    # Alternative method: inflate then deflate again ... (requires more effort at our end)
    # Could NOT get this to work (though suggested by both reference sites!)
    # SetOutputFilter INFLATE;proxy-html;DEFLATE
  </Location>

  ....

I’ve had occasion recently to frequently work with “dates” that come in a lot of shapes and sizes including:

• Dates in distant past and future including BC/BCE dates
• Dates in a wild variety of formats: Jan 1890, January 1890, 1st Dec 1890, Spring 1890 etc
• Dates of varying precision: e.g. 1890, 1890-01 (i.e. Jan 1890), 1890-01-02
• Imprecise dates: c1890, 1890?, fl 1890 etc

Unfortunately existing support for these in python is fairly weak. I therefore authored a python FlexiDate module (part of a new swiss (army knife) package) which is focused on supporting:

1. Dates outside of Python (or DB) supported period (esp. dates < 0 AD)
2. Imprecise dates (c.1860, 18??, fl. 1534, etc)
3. Normalization of these dates to machine processable versions especially:
   • ISO 8601
   • Dates sortable in the database (in correct date order)

Background

Things we would like:

1. Dates outside of Python (or DB) supported period (esp. dates < 0 AD)
2. Imprecise dates (c.1860, 18??, fl. 1534, etc)
3. Normalization of dates to machine processable versions
4. Sortable in the database (in correct date order)
5. Human readability as dates will be re-edited/viewed by people

Not all of these requirements are satisfiable at once in a simple way.

Be clear about what we want:

1. Storage (and preservation) of “user” dates (both normal and non-normal)
2. Normalization of dates (e.g. to ~ ISO 8601)
3. Integration with database (sortability and serializability)

Solution for 1: Represent dates as strings.

Solution for 2: Have a parser (via an intermediate FlexiDate object).

Solution for 3: convert to a float.

Remark: no string based date format will sort dates correctly based on std string ordering (PF: let x,y be +ve dates and X,Y their string representations then if X -X<-Y (wrong!))

Thus we need to add some other field if we wish dates to be correctly sorted (or not worry about sorting of -ve dates …)

1. For any given date attribute have 2 actual fields:

• user version — the version edited by users
• normalized/parsed version — a version that is usable by machines

2. Store both versions in a single field but with some form of serialization.

1. Convert dates to long ints (unlimited in precision) and put this in a separate field and use that for sorting.

Comments

Initially thought that we should parse before saving into a FlexiDate format but: a) why bother b) when parsing always hard not to be lossy (in particular when converting to iso8601 using e.g. dateutil very difficult to not add info e.g. parsing 1860 can easily give us 1860-01-01 …).

References and Existing Libraries

• Excellent dateutil (which we use0
• http://wiki.python.org/moin/WorkingWithTime
• ISO 8601: http://www.iso.org/iso/dateandtime_format
  • http://code.google.com/p/pyiso8601/source/browse/trunk/iso8601/iso8601.py
• http://www.feedparser.org/docs/date-parsing.html
• http://seehuhn.de/pages/pdate