This directory contains a generic web crawler (basic directory) and several useful implementations build on top of this.