<header>
<title>Automatic Recording for KiSS Hard Disk Recorders</title>
</header>
- <body>
+ <body>
+ <warning>
+ KiSS makes regular updates to their site that sometimes require adaptations
+ to the crawler. If it stops working, check out the most recent version here.
+ </warning>
+ <section id="changelog">
+ <title>Changelog</title>
+
+ <section>
+ <title>31 August 2006</title>
+ <ul>
+ <li>Added windows bat file for running the crawler under windows.
+ Very add-hoc, will be generalized. </li>
+ </ul>
+ <title>24 August 2006</title>
+ <ul>
+ <li>The crawler now uses desktop login for crawling. Also, it is much more efficient since
+ it no longer needs to crawl the individual programs. This is because the channel page
+ includes descriptions of programs in javascript popups which can be used by the crawler.
+ The result is a significant reduction of the load on the KiSS EPG site. Also, the delay
+ between requests has been increased to further reduce load on the KiSS EPG site. </li>
+ <li>
+ The crawler now crawls programs for tomorrow instead of for today.
+ </li>
+ <li>
+ The web based crawler is configured to run only between 7pm and 12pm. It used to run at
+ 5am.
+ </li>
+ </ul>
+ </section>
+
+ <section>
+ <title>13-20 August 2006</title>
+ <p>
+ There were several changes to the login procedure, requiring modifications to the crawler.
+ </p>
+ <ul>
+ <li>The crawler now uses the 'Referer' header field correctly at login.</li>
+ <li>KiSS now uses hidden form fields in their login process which are now also handled correctly by the
+ crawler.</li>
+ </ul>
+ </section>
+ </section>
<section id="overview">
<title>Overview</title>
records programs for you or sends notifications about interesting ones.
</p>
<p>
- In its current version, the crawler can be used a standalone program
- only and the preferred way to run it is as a scheduled task.
+ In its current version, the crawler can be used in two ways:
</p>
+ <ul>
+ <li><strong>standalone program</strong>: A standalone program run as a scheduled task.</li>
+ <li><strong>web application</strong>: A web application running on a java
+ application server. With this type of use, the crawler also features an automatic retry
+ mechanism in case of failures, as well as a simple web interface. </li>
+ </ul>
</section>
<section>
</p>
<p>
The easy way to start is the
- <a href="installs/crawler/kiss/kiss-crawler-bin.zip">binary version</a>.
+ <a href="installs/crawler/kiss/kiss-crawler-bin.zip">standalone program binary version</a>
+ or using the <a href="installs/crawler/kissweb/wamblee-crawler-kissweb.war">web
+ application</a>.
</p>
<p>
The latest source can be obtained from subversion with the
URL <code>https://wamblee.org/svn/public/utils</code>. The subversion
repository allows read-only access to anyone.
</p>
+ <p>
+ The application was developed and tested on SuSE linux 9.1 with JBoss 4.0.2 application
+ server (only required for the web application). It requires at least a Java Virtual Machine
+ 1.5 or greater to run.
+ </p>
</section>
<section>
<title>Configuring the crawler</title>
<p>
- The crawler comes with two configuration files, namely
- <code>crawler.xml</code> and <code>programs.xml</code>.
+ The crawler comes with three configuration files:
+ </p>
+ <ul>
+ <li><code>crawler.xml</code>: basic crawler configuration
+ tailored to the KiSS electronic programme guide.</li>
+ <li><code>programs.xml</code>: containing a description of which
+ programs must be recorded and which programs are interesting.</li>
+ <li><code>org.wamblee.crawler.properties</code>: Containing a configuration </li>
+ </ul>
+ <p>
+ For the standalone program, all configuration files are in the <code>conf</code> directory.
+ For the web application, the properties files is located in the <code>WEB-INF/classes</code>
+ directory of the web application, and <code>crawler.xml</code> and <code>programs.xml</code>
+ are located outside of the web application at a location configured in the properties file.
</p>
+
<section>
<title>Crawler configuration <code>crawler.xml</code></title>
Programme Guide.
</p>
</section>
-
- <section>
- <title>Program configuration: <code>programs.xml</code></title>
-
- <p>
- The <code>programs.xml</code> file contains the following
- configuration items:
- </p>
- <ul>
- <li>Notification configuration: Describing how to
- do notification of the results of crawling the site. </li>
- <li>Zero or more configurations of interesting programs. </li>
- </ul>
- <section>
- <title>Notification configuration</title>
- <p>
- Notification is configured in the (surprise, surprise!)
- <code>notification</code> element. This notification element
- is used to configure respectively sender mail address (= reply
- address), recipient address, subject of the email, smtp server
- host and port and optional username and password.
- In addition it contains the names of the stylesheets to
- generate the HTML and Text reports. These stylesheets
- should not be changed.
- </p>
- </section>
-
+
<section>
<title>Program configuration</title>
<p>
</table>
<p>
- It is possible that different programs cannot be recorded at
+ It is possible that different programs cannot be recorded
since they overlap. To deal with such conflicts, it is possible
to specify a priority using the <code>priority</code> element.
Higher values of the priority value mean a higher priority.
</p>
</section>
-
-
+
+ <section>
+ <title>Notification configuration</title>
+ <p>
+ Edit the configuration file <code>org.wamblee.crawler.properties</code>.
+ The properties file is self-explanatory.
+ </p>
</section>
</section>
+
+
+
<section>
<title>Installing and running the crawler</title>
<section>
- <title>Binary distribution</title>
+ <title>Standalone application</title>
<p>
In the binary distribution, execute the
<code>run</code> script for your operating system
</p>
</section>
+ <section>
+ <title>Web application</title>
+ <p>
+ After deploying the web application, navigate to the
+ application in your browser (e.g.
+ <code>http://localhost:8080/wamblee-crawler-kissweb</code>).
+ The screen should show an overview of the last time it ran (if
+ it ran before) as well as a button to run the crawler immediately.
+ Also, the result of the last run can be viewed.
+ The crawler will run automatically every morning at 5 AM local time,
+ and will retry at 1 hour intervals in case of failure to retrieve
+ programme information.
+ </p>
+ </section>
+
<section>
<title>Source distribution</title>
<p>
<section>
<title>General usage</title>
<p>
- The crawler, as it is now, is s standalone program which is
- intended to be run from a command-line. When it runs, it
- retrieves the programs for today. As a result, it is advisable
+ When the crawler runs, it
+ retrieves the programs for tomorrow. As a result, it is advisable
to run the program at an early point of the day as a scheduled
- task (e.g. cron on unix).
+ task (e.g. cron on unix). For the web application this is
+ preconfigured at 5AM.
</p>
+ <note>
+ If you deploy the web application today, it will run automatically
+ on the next (!) day. This even holds if you deploy the application
+ before the normal scheduled time.
+ </note>
+
<p>
Modifying the program to allow it to investigate tomorrow's
programs instead is easy as well but not yet implemented.
<p>
The best example is in the distribution itself. It is my personal
- <code>programs.xml</code> file.
+ <code>programs.xml</code> file.
</p>
</section>
git://wamblee.org / utils / blobdiff