ClientServer provides a single machine, multi-process reaper.

ClientServer provides a method of running the reaper where there can be multiple fetch actions taking place at once. The restricting factor of gathering is often network speed, especially when slow sites are being gathered from. Running with this option allows slow sites to be gathered at their own pace, whilst gathering from faster sites need not be held up.

The module is implemented using multiple processes communicating via at TCP/IP socket.

These options should be in the top level section of the configuration file

Socket

The port to be used for communication. This should be a TCP port which you know is free on your machine, and must be higher than 1024.

Reapers

The number of reapers to run against a given server. This should be no more than the maximum number of servers you are indexing (as the same server cannot be gathered by more than one reaper consecutively, regardless of the delay