Now crawling in desktop mode should work. It is much more efficient
[utils] / crawler / ABOUT.txt
1 This directory contains a generic web crawler (basic directory) and several useful implementations build on top of this.
2