ModSecurity with the OWASP CRS rule set is switched on in ten minutes and switched off three days later — after articles stop saving in the admin area, file uploads break, and a customer cannot place an order because their address contained an apostrophe. The conclusion "the WAF gets in the way of work" suggests itself, and it is wrong: nearly all of those blocks are cured by three or four precise exceptions, and the whole trick is finding them correctly.

Do not turn on blocking straight away

The first week is observation only. In /etc/modsecurity/modsecurity.conf:

SecRuleEngine DetectionOnly

In this mode the WAF logs everything it would have blocked and blocks nothing. A week of real traffic — including your own work in the admin area, image uploads and placing an order — gives a list of genuine false positives rather than hypothetical ones. Switching to On makes sense only once that list has been dealt with.

How CRS works: not one rule but a sum

This is the key to everything that follows. CRS almost never blocks a request with a single rule. Each rule that fires adds anomaly points to the request, and the block happens when the total crosses a threshold. That is why the log shows not one line but several, and why the last of them is the rule with identifier 949110 — the one that adds up the total.

The practical consequence: the exception must be made for the rule that scored the points, not for 949110. Disable the summing rule and you disable the entire set, leaving the WAF as a line in a config file.

The second consequence is the paranoia level. It is one by default, and that is the right choice. Levels 2 and 3 add rules that produce false positives on ordinary sites by design, and are worth enabling only once level one is fully tuned.

Find the guilty rule

Everything you need is in the audit log (/var/log/modsec_audit.log) and in the web server's error log. Search by the time of the block:

sudo grep -o 'id "[0-9]*"' /var/log/modsec_audit.log | sort | uniq -c | sort -rn | head

That is a frequency list of the rules that fired. Then, for a specific identifier, look at what exactly triggered it:

sudo grep -A5 'id "942100"' /var/log/modsec_audit.log | head -40

Three things are needed: the rule identifier, the parameter name (ARGS:content, ARGS:comment) and the request path. The exception is built out of those.

The usual suspects

The list repeats from site to site:

  • 942100 — SQL injection. Fires on text fields with long content: an article body, a product description, a comment. Quotation marks, brackets and words like select look suspicious to the detector inside ordinary prose;
  • 941100 and the 941xxx family — XSS. They arrive together with a visual editor: HTML tags in a field are the entire point of how it works;
  • 920420 — disallowed Content-Type. Breaks APIs and file uploads: the default set of permitted types is narrow, and in older versions application/json was not in it;
  • 913100 — scanner by User-Agent. Catches legitimate tools along with scanners: availability monitoring, curl in your own scripts;
  • 200002, 200004 — request body parsing errors. They usually mean not an attack but an exceeded body size limit, that is, a large file upload.

Three ways to make an exception

In increasing order of bluntness. All of them go into your own file (for example /etc/modsecurity/crs/REQUEST-900-EXCLUSION-RULES.conf) rather than into the CRS files themselves: the rule set gets updated and your edits would vanish with it.

Remove one parameter from under one rule. The most precise option, and the one to aim for:

SecRuleUpdateTargetById 942100 "!ARGS:content"

Disable a rule on one path only. Right when a specific page is noisy — an editor, an import, a review form:

SecRule REQUEST_URI "@beginsWith /admin/post" \
    "id:1001,phase:1,pass,nolog,ctl:ruleRemoveById=942100"

Disable the rule entirely. A last resort and nearly always a sign that the cause was never found:

SecRuleRemoveById 942100

The difference between the first and the third is substantial. In the first case one field — the article body — stops being checked for SQL injection by one rule; in the second and third the whole site stops being checked. The difference in effort is about five minutes.

After the edits, check the configuration and reload gently:

sudo apachectl configtest && sudo systemctl reload apache2
sudo nginx -t && sudo systemctl reload nginx

An order that saves a week

  1. A week in DetectionOnly with normal work on the site, admin area and uploads included.
  2. A frequency list of rules from the audit log. Work down it from the top — the first three or four account for ninety per cent of the noise.
  3. For each: work out which parameter and on which page. Make the exception by parameter, not by rule.
  4. Only now SecRuleEngine On.
  5. Once a month, look in at the blocks: the site changed, so new false positives appeared.

That last point is where it usually falls apart: the audit log is gigabytes of text and nobody will read it by hand. The point of a collected panel is to have the list of rules that fired, the blocked requests and the active rule set in front of you rather than extracted with grep. What that looks like is on the demo page below.