Norconex Products
OPEN SOURCE CRAWLERS
Full-featured, flexible and extensible. Run on any platform. Crawl what you want, how you want.
Available Crawlers
HTTP Crawler
Collect content from websites for your search engine or any other data repository. This full-featured collector can run independently or embed it within your own application.
Filesystem Crawler
Norconex Filesystem Crawler is a flexible crawler for collecting, parsing and manipulating data ranging from local hard drives to network locations into various data repositories such as search engines.
Features
- Universal Crawlers
- Easy for developers to extend
- Commercially supported
- Modify document metadata
- Easy to run
- Embeddable
- Ease of maintenance
- Resumable upon system failure
- Modular design
- Cross-platform
- Portable
- Open Source
- Powerful
- Easy to use
- Good documentation
- Event listeners
- Logs are meaningful and verbose
- Flexible