Open-source crawlers

Full-featured, flexible and extensible. Run on any platform. Crawl what you want, how you want.



Available Crawlers

...

HTTP Collector

Web crawler for the Internet or Intranet.

More Info

...

Filesystem Collector

Crawl data from local disk, FTP, SFTP, WebDAV, HDFS, ...

More Info

Features

Why chose Norconex Collectors? Click here to learn more.

Universal Crawlers
Easy for developers to extend
Commercially supported
Modify document metadata
Easy to run
Embeddable
Ease of maintenance
Resumable upon system failure
Modular design
Cross-platform
Portable
Open Source
Powerful
Easy to use
Good documentation
Event listeners
Logs are meaningful and verbose
Flexible
More Info