Automatic Recording for KiSS Hard Disk Recorders

+ Changelog + +

+ 24 August 2006 +

The crawler now uses desktop login for crawling. Also, it is much more efficient since + it no longer needs to crawl the individual programs. This is because the channel page + includes descriptions of programs in javascript popups which can be used by the crawler. + The result is a significant reduction of the load on the KiSS EPG site. Also, the delay + between requests has been increased to further reduce load on the KiSS EPG site.
+ The crawler now crawls programs for tomorrow instead of for today. +
+ The web based crawler is configured to run only between 7pm and 12pm. It used to run at + 5am. +

+ +

+ 13-20 August 2006 +

+ There were several changes to the login procedure, requiring modifications to the crawler. +

The crawler now uses the 'Referer' header field correctly at login.
KiSS now uses hidden form fields in their login process which are now also handled correctly by the + crawler.

@@ -61,13 +103,20 @@

The easy way to start is the - binary version. + standalone program binary version + or using the web + application.

The latest source can be obtained from subversion with the URL https://wamblee.org/svn/public/utils. The subversion repository allows read-only access to anyone.

+ The application was developed and tested on SuSE linux 9.1 with JBoss 4.0.2 application + server (only required for the web application). It requires at least a Java Virtual Machine + 1.5 or greater to run. +

@@ -81,9 +130,14 @@ tailored to the KiSS electronic programme guide.

programs.xml: containing a description of which programs must be recorded and which programs are interesting.

org.wamblee.crawler.properties: Containing a configuration of - how to notify users of results.

org.wamblee.crawler.properties: Containing a configuration

+ For the standalone program, all configuration files are in the conf directory. + For the web application, the properties files is located in the WEB-INF/classes + directory of the web application, and crawler.xml and programs.xml + are located outside of the web application at a location configured in the properties file. +

@@ -203,7 +257,7 @@ Installing and running the crawler

- Binary distribution + Standalone application

In the binary distribution, execute the run script for your operating system @@ -212,6 +266,21 @@

+ Web application +

+ After deploying the web application, navigate to the + application in your browser (e.g. + http://localhost:8080/wamblee-crawler-kissweb). + The screen should show an overview of the last time it ran (if + it ran before) as well as a button to run the crawler immediately. + Also, the result of the last run can be viewed. + The crawler will run automatically every morning at 5 AM local time, + and will retry at 1 hour intervals in case of failure to retrieve + programme information. +

Source distribution

@@ -225,12 +294,18 @@

General usage

- The crawler, as it is now, is s standalone program which is - intended to be run from a command-line. When it runs, it - retrieves the programs for today. As a result, it is advisable + When the crawler runs, it + retrieves the programs for tomorrow. As a result, it is advisable to run the program at an early point of the day as a scheduled - task (e.g. cron on unix). + task (e.g. cron on unix). For the web application this is + preconfigured at 5AM.

+ + If you deploy the web application today, it will run automatically + on the next (!) day. This even holds if you deploy the application + before the normal scheduled time. + +

Modifying the program to allow it to investigate tomorrow's programs instead is easy as well but not yet implemented. @@ -245,7 +320,7 @@

The best example is in the distribution itself. It is my personal - programs.xml file. + programs.xml file.