|$regex||string||The rule as a regular-expression|
|bool|| TRUE if the rule was added to the list.|
FALSE if the given regex is not valid.
After receiving the HTTP-header of a followed URL, the crawler check's - based on the given rules - whether the content of that URL
should be received.
If no rule matches with the content-type of the document, the content won't be received.
This rules lets the crawler receive the content/source of pages with the Content-Type "text/html" AND "text/css".
Other pages or files with different content-types (e.g. "image/gif") won't be received (if this is the only rule added to the list).
IMPORTANT: By default, if no rule was added to the list, the crawler receives every content.
Note: To reduce the traffic the crawler will cause, you only should add content-types of pages/files you really want to receive.
But at least you should add the content-type "text/html" to this list, otherwise the crawler can't find any links.