This is a general library for implementing a web crawler. 

The crawler works by retrieving an HTML page and transforming the HTML
(content + presentation) into content using XSLT stylesheets. Using a convention
for links in the converted content, it becomes possible to build a generic interface on the retrieved pages for navigating through the content. 

A configuration file determines how a certain page must be retrieved and transformed.