Harvest::Controller::SOIFFilter::RobotsMeta - handle the robot control META tag | |
| |
DESCRIPTION | |
http://www.w3.org/Search/9605-Indexing-Workshop/ReportOutcomes/Spidering.txt suggests a protocol for controlling robot access by means of META tags within the document. This class implements this form of control.
| |
PARAMETERS | |
None. | |