increased the max delay to 1 second to make it a bit nicer.
[utils] / crawler / ABOUT.txt
1 This directory contains a generic web crawler (basic directory) and several useful implementations build on top of this.
2