This project Web Spider contains 

WEB-CRAWLER that crawles a site given a base url and outputs links it found to separate files
INDEXER that load pages found by the webrawler and creates a searchable index for it
USER-INTERFACE that can be loaded for all modules, or loaded separately per for the crawler and indexer. The program works with command line interface too

See Report