These pages contain POD documentation for each of the Harvest-NG classes. This information is automatically generated from information in the source code files, and as such corresponds to the current CVS copy, not necessarily the current distribution version.

• Harvest::ClientServer - provide a multiprocess reaper
• Harvest::Config - configuration file parsing
• Harvest::Controller - interface to the spider controller
• Harvest::Database - Database of all gathered objects
• Harvest::Debug - static debugging control
• Harvest::Object - represents ``objects'' being gathered
• Harvest::Reaper - fetch and summarise objects
• Harvest::SingleThread - run a reaper in a single thread
• Harvest::Version - static encoding of version information

• Harvest::Controller::Bucket - maintain collections of URLs
• Harvest::Controller::FilterManager - maintain the URL filtering system
• Harvest::Controller::GenericFilter Harvest::Controller::Filter::Generic - generic filter class
• Harvest::Controller::GenericFilterNoState - generic filter class which doesn't have persisent state
• Harvest::Controller::RootNode - store all of the information about a RootNode
• Harvest::Controller::SOIFFilter - SOIF data based filtering
• Harvest::Controller::URLFilter - URL filtering
• Harvest::Controller::Workload - a class for maintaining a robots workload

• Harvest::Controller::SOIFFilter::CacheControl - respect the CacheControl header
• Harvest::Controller::SOIFFilter::CompatHack - backwards compatibility headers
• Harvest::Controller::SOIFFilter::DCDot - map Dublin Core metadata
• Harvest::Controller::SOIFFilter::Description - add a description of an object
• Harvest::Controller::SOIFFilter::MD5Visited - keep a track of objects visited
• Harvest::Controller::SOIFFilter::NoArchive - respect the X-No-Archive header
• Harvest::Controller::SOIFFilter::Prune - prune SOIF;
• Harvest::Controller::SOIFFilter::RobotsMeta - handle the robot control META tag
• Harvest::Controller::SOIFFilter::Timing - add time information to objects

• Harvest::Controller::URLFilter::Depth - limit based on depth
• Harvest::Controller::URLFilter::HostLimit - limit on number of hosts to crawl
• Harvest::Controller::URLFilter::HostList - restrict gathering to a list of hosts
• Harvest::Controller::URLFilter::NeverUp - prevent upwards traversal
• Harvest::Controller::URLFilter::RobotsTxt - robots.txt file parsing
• Harvest::Controller::URLFilter::Scheme - limit based on URL scheme
• Harvest::Controller::URLFilter::URLLimit - limit based on number of URLs fetched
• Harvest::Controller::URLFilter::URLregex - limit based on regular expressions
• Harvest::Controller::URLFilter::Visited - keep a track of URLs already visited

• Harvest::Database::DBM - Database of all gathered objects
• Harvest::Database::DirStructure - Database represented as a directory tree
• Harvest::Database::Generic - Parent class for Harvest-NG Databases

• Harvest::Object::Headers - a set of protocol headers
• Harvest::Object::Manage - encapsulate object management information

• Harvest::Reaper::Fetcher - fetch objects
• Harvest::Reaper::HarvestUA - A LWP class for the Harvest user agent
• Harvest::Reaper::Summarise - maintain the summarising system

• Harvest::Reaper::Encoding::Gzip - handle gzip based content encoding

• Harvest::Reaper::Summarise::FTPListing - create links from a FTP directory listing
• Harvest::Reaper::Summarise::Filter - summarise data being fetched for filters
• Harvest::Reaper::Summarise::Generic - generic summariser class
• Harvest::Reaper::Summarise::HTML - summarise an HTML document
• Harvest::Reaper::Summarise::HTTP - create summariser and management entries from HTTP headers
• Harvest::Reaper::Summarise::MD5 - add MD5 hash of data
• Harvest::Reaper::Summarise::NewsHeaders - create summariser entries from News headers
• Harvest::Reaper::Summarise::RPM - summarise a RPM package
• Harvest::Reaper::Summarise::RunProg - run a program to do a summary
• Harvest::Reaper::Summarise::Schema - add schema and location information
• Harvest::Reaper::Summarise::Text - summarise plain text