(no commit message)

[utils] / crawler / basic / ABOUT.txt

diff --git a/crawler/basic/ABOUT.txt b/crawler/basic/ABOUT.txt
new file mode 100644 (file)
index 0000000..b61c613
--- /dev/null
+++ b/crawler/basic/ABOUT.txt
@@ -0,0 +1,9 @@
+This is a general library for implementing a web crawler. 
+
+The crawler works by retrieving an HTML page and transforming the HTML
+(content + presentation) into content using XSLT stylesheets. Using a convention
+for links in the converted content, it becomes possible to build a generic interface on the retrieved pages for navigating through the content. 
+
+A configuration file determines how a certain page must be retrieved and transformed. 
+
+