incorporating new Suricata rules (and removing old ones) without restarting the Suricata containers

[mmguero]

A user asked about the ability to pick up new custom suricata rules without stopping/starting the suricata container. Mainly, the ability to have rules files placed in ./suricata/rules/ get incorporated into the new suricata.yaml config file and rules deleted from there to be remove.

This documentation suggests a restart of the live-suricata process via supervisord, but I'm suspicious if that's actually correct because I don't see how that would regenerate the config file.

I gave this command:

docker compose exec -u 1000 suricata-live bash -c '/usr/local/bin/suricata_config_populate.py --suricata /usr/bin/suricata-offline -vv && kill -USR2 $(pidof suricata) && echo "Suricata reload signaled"'

as a temporary workaround, but it was reported that old rules were not removed.

So the task is:

• figure out what needs to happen to (if necessary, which I think it is, as I think it enumerates the custom rules files and appends their paths to the yaml file), for a command for new suricata rules to be added to the file and now-missing suricata rules to be removed, and to reload the rules in suricata (probably with that USR2 signal) I've determined this is actually working the way it should. There are some nuances with Kubernetes (see my long comment below) but I don't think it's really much that can be done about it atm. It's doable, you just have to get the files you want into /opt/suricata/rules in the container (and the ones you don't want out) and run the same command to restart it as indicated in the new documentation. The policy manager feature ("policy manager" for Malcolm and Hedgehog Linux (meta-issue) #396) will change how this is done anyway.
• fix the documentation

Closing this bug with just the documentation fix.

[mmguero]

After some debugging with the user I replied with this. Pasting here for tracking purposes.

The custom suricata rules are pull from /opt/suricata/rules, but for a Kubernetes-based deployment like yours, they are actually created as configmaps in /opt/suricata/rules/configmap. Then, during the container startup, these files get rsynced into /opt/suricata/rules (this is standard for all of the malcolm containers). But this sync would only happen on container startup, not during runtime.

So, I'm not sure how you're adding new files. You said the new ones were being seen, right? but the old ones are still there as well? If that's the case I am assuming you're recreating the configmap (actually, I don't see how this would be having the new rules show up if this is what you were doing, due to the sync I talked about above)? or are you doing something else to copy the new rules straight into /opt/suricata/rules directly?

If the case is that you are seeing the new rules show up in the suricata.yaml, but the old ones are being left behind, then what you said about removing what was in the /opt/suricata/rules directory followed by rerunning the script should have worked. observe:

suricata@sgrover:/opt/suricata$ ls /opt/suricata/rules
dns.rules  suspicious.rules

suricata@sgrover:/opt/suricata$ tail /etc/suricata/suricata.yaml 
rule-files:
- suricata.rules
- /opt/suricata/rules-default/OT/malcolm/CVE-2023-28771_Zyxel.rules
- /opt/suricata/rules-default/OT/malcolm/CVE-2023-6448_Unitronics_VisiLogic.rules
- /opt/suricata/rules-default/OT/nsacyber/ELITEWOLF/AllenBradley_RockwellAutomation.rules
- /opt/suricata/rules-default/OT/nsacyber/ELITEWOLF/SchweitzerEngineeringLaboratories.rules
- /opt/suricata/rules-default/OT/nsacyber/ELITEWOLF/Siemens.rules
- /opt/suricata/rules/dns.rules
- /opt/suricata/rules/suspicious.rules

suricata@sgrover:/opt/suricata$ rm -f /opt/suricata/rules/dns.rules 
suricata@sgrover:/opt/suricata$ ls /opt/suricata/rules
suspicious.rules

suricata@sgrover:/opt/suricata$ /usr/local/bin/suricata_config_populate.py --suricata /usr/bin/suricata-offline && kill -USR2 $(pidof suricata) && echo "Suricata reload signaled"
Suricata reload signaled

suricata@sgrover:/opt/suricata$ tail /etc/suricata/suricata.yaml 
  filename: /var/run/suricata/suricata-command.socket
rule-files:
- suricata.rules
- /opt/suricata/rules-default/OT/malcolm/CVE-2023-28771_Zyxel.rules
- /opt/suricata/rules-default/OT/malcolm/CVE-2023-6448_Unitronics_VisiLogic.rules
- /opt/suricata/rules-default/OT/nsacyber/ELITEWOLF/AllenBradley_RockwellAutomation.rules
- /opt/suricata/rules-default/OT/nsacyber/ELITEWOLF/SchweitzerEngineeringLaboratories.rules
- /opt/suricata/rules-default/OT/nsacyber/ELITEWOLF/Siemens.rules
- /opt/suricata/rules/suspicious.rules

Now here's what I'm a little confused about: you talk about rules still being in /var/lib/suricata/rules/suricata.rules, but custom rules don't get written to that file at all in the first place:

suricata@sgrover:/opt/suricata$ tail /etc/suricata/suricata.yaml 
rule-files:
- suricata.rules
- /opt/suricata/rules-default/OT/malcolm/CVE-2023-28771_Zyxel.rules
- /opt/suricata/rules-default/OT/malcolm/CVE-2023-6448_Unitronics_VisiLogic.rules
- /opt/suricata/rules-default/OT/nsacyber/ELITEWOLF/AllenBradley_RockwellAutomation.rules
- /opt/suricata/rules-default/OT/nsacyber/ELITEWOLF/SchweitzerEngineeringLaboratories.rules
- /opt/suricata/rules-default/OT/nsacyber/ELITEWOLF/Siemens.rules
- /opt/suricata/rules/dns.rules
- /opt/suricata/rules/suspicious.rules

suricata@sgrover:/opt/suricata$ cat /opt/suricata/rules/dns.rules
alert dns any any -> any any (msg:"Teleconsole DNS query"; dns_query; content:"teleconsole"; nocase; sid:9000032; rev:1;)

suricata@sgrover:/opt/suricata$ grep -c teleconsole /var/lib/suricata/rules/suricata.rules 
0

See? My teleconsole rule there is referenced by its filename in /etc/suricata/suricata.yaml but it's not in /var/lib/suricata/rules/suricata.rules (it would actually cause a duplicate signature error if it were in both places).

Long story short, whatever is in /opt/suricata/rules/ at the time suricata_config_populate.py is run is what will end up in the rule-files section of /etc/suricata/suricata.yaml. However you do it, as long as you get that /opt/suricata/rules/ directory looking right (add the files you want to add, remove the files you want to remove) then run the config populate script, that's what it'll use. That script doesn't generate /var/lib/suricata/rules/suricata.rules at all.