Removing overbearing use of Refer header added in 3.0.0 due to FPs

[csanders-git]

Thanks to @jschleus for the report. This false positive was noted in #547

[dune73]

The Referer is a dangerous header and with all these log dashboards and online log browsers, it becomes more dangerous still.

But of course it's a source of many FPs and I see the need to get rid of these.

May I suggest to add move the Referer to a strict sibling of these two rules in PL2? Most applications have fairly standard Referers 90% of the time (-> internal referers). So if you handle the possible FPs at PL2 for these known offending Referers, you are quite OK. And I think we can expect that effort from somebody running PL2.

[lifeforms]

@dune73 I'm not against adding XSS checks in PL2, but we should probably be a bit smarter about it instead of just adding Referer check as I had done last-minute. These particular rules will fire heavy on a lot of URLs. Since Referer XSS checks were not in CRS2 at all, I would propose that it's not a "must have" for 3.0 and we'd better make it a bit of a research project. Maybe we should try looking for a few little PoCs to get more of an idea of how log viewers will be affected so we can intelligently add some detection.

[dune73]

PL3 as a compromise?

This keeps the check in the rulebase and makes sure we do not forget about it. Without us having to deal with it in the near future.

[lifeforms]

Would not mind it, but on the whole I'm not sure if we're doing users a service with this check. Also even if we enable this rule the XSS protection is likely not very complete. We really have to dive into the problem more deeply. To remind ourselves to revisit this issue I created #606.

[dune73]

OK. You are probably right and my desire to get stuff off the table is too drastic.

Ready to merge.