added support for configuring request headers.
[utils] / crawler / ABOUT.txt
1 This directory contains a generic web crawler (basic directory) and several useful implementations build on top of this.
2