PHPCrawl webcrawler library/framework

Using multiple processes

Since version 0.8 phpcrawl is able to use multiple processes to spider a website. In most cases using more processes simultaneously will speed up the crawling-procedure dramatically.

In order to start phpcrawl in multi-process-mode, simply call the goMultiProcessed()-method instead of the go()-method to start the crawler.

$crawler = new MyCrawler();

// ...

// Start crawling by using 5 processes

However, there are some things to consider when using the multi-process mode:
  • Some PHP-extensions are required to successfully run phpcrawl in multi-process mode (PCNTL-extension, SEMAPHORE-extension, PDO-extension). For more details see the requirements page.
  • The multi-process mode only works on unix/linux-based systems
  • Scripts using phpcrawl with mutliple processes have to be run from the commandline (php CLI)
  • Increasing the number of processes to very high values does't automatically mean that the crawling-process will go off faster! The ideally number of processes depends on a lot of circumstances like the available bandwidth, the local technical environment (CPU), the delivery-rate and data-rate of the server hosting the taget-website and so on.
    Using something between 3 to 10 processes should be good values to start from.