Harvest-NG is a set of tools for building a standards-compliant web crawler. It is implemented in Perl, taking advantage of many of the existing perl tools, and can provide a complete harvesting solution for a web site. The development of Harvest-NG was started as an attempt to preserve the strong features of the Harvest architecture, but to allow more rapid developing and prototyping from a cleaner, better structured codebase.

The current release of Harvest-NG has come a long way towards achieving these goals, and provides a strong foundation upon which to build internet resource discovery systems. These are the two roles of the project, to build both a fast, resilient, and efficient indexing system, and through this, to produce a highly extensible toolkit for building other systems to achieve different tasks.

Harvest-NG isn't yet, a plug-and-play search service. If you want this, you'll need to look at combining it with existing search products, or running it as part of a Harvest system. Some information on making Harvest work with other products is available from this site.

Follow these links for more information about how to get, use, and help develop, harvest-ng.

• About Harvest-NG
• Downloading and Installing Harvest-NG
• Using Harvest-NG
• Getting help
• Information for Developers