Site traffic classified as CDN

[patricknugent-datto]

Describe the bug

I have these two rules as part of a chain in my mangle table:

Chain classify (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all  --  any    any     anywhere             anywhere            mark match 0x0/0xf0 ndpi host onet.pl proto all MARK or 0x40
    0     0 MARK       all  --  any    any     anywhere             anywhere            mark match 0x0/0xf0 ndpi proto amazonaws MARK or 0x30

The first rule should match any onet.pl traffic and mark the packet with 0x40 to be dropped later, blocking the site.

The second rule should match amazonaws traffic and mark the packet with 0x30 to allow it.

However with both of these rules, it seems like the traffic to onet.pl is being classified as amazonaws instead of falling under the first rule and the packets are not being dropped. If I remove the second rule then the traffic is classified as onet.pl as expected and the packets are being dropped correctly.

I would expect that the traffic should be matching the first rule and dropping the packets. A similar issue can also be seen on other sites using CDNs (e.g rule 1 matching amazon.com and rule 2 matching amazonaws). Adding a first rule to match the site and then adding a second rule to match the CDN will result in all or most site traffic being classified as the CDN rather than the site. Is this a potential misclassification issue or might there be something else going on? Perhaps there is a better approach for implementing what I'm looking for here?

Expected behavior

Site traffic is classified separately from CDN traffic

Obtained behavior

Site traffic is being classified as CDN traffic

nDPI Environment (please complete the following information):

• OS name: Ubuntu
• OS version: 20.04
• nDPI version or commit hash: 3ff5a9b

How to reproduce the reported bug

Reproducible using ndpiReader?

• The reported bug is reproducible using ndpiReader.

If your bug is reproducible using a pcap, please attach a pcap file (or a valid link to download it)

Example: site_traffic.zip

[vel21ripn]

I received your traffic sample.
I will look into the issue shortly.

[vel21ripn]

Please show the output
iptables -nvxL | grep -i ndpi

[patricknugent-datto]

Sure, here is the output:

24    30672 NDPI       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* add metadata to /proc/net/xt_ndpi/flows */ ndpi ! error NDPI NETFLOW

This is in a custom chain in the filter table. Packets marked 0x40 are sent here from the FORWARD table. This rule will add the metadata to /proc/net/xt_ndpi/flows. The rule after this should send these packets to NFQUEUE to be dropped.

[vel21ripn]

What command did you use to create the rules

    0     0 MARK       all  --  any    any     anywhere             anywhere            mark match 0x0/0xf0 ndpi host onet.pl proto all MARK or 0x40

Where does "proto all" come from?

In order to understand where the error is, you need a test setup and traffic samples or
the ability to perform the necessary requests.
Next, enable iptables tracing (-j TRACE) and debugging in nDPI and see which rules are triggered for each packet.

modprobe xt_ndpi  lib_trace=4 xt_debug=1

for i in ct_start packet ipt_match ct_cache ipt_match dpi_done dpi hostname tls guessed
do echo "$i y" >/proc/net/xt_ndpi/debug; done

[patricknugent-datto]

These are the commands that I used to create those rules:

iptables -t mangle -A classify -m mark --mark 0x0/0xf0 -m ndpi --all --host onet.pl -j MARK --or-mark 0x40
iptables -t mangle -A classify -m mark --mark 0x0/0xf0 -m ndpi --proto amazonaws -j MARK --or-mark 0x30

The "proto all" part comes from "ndpi --all" and in theory it should match any "onet.pl" traffic regardless of the protocol.

Thanks for the debugging tips, I'll also try to debug this a bit more on my end to see what rules are being triggered here.

[vel21ripn]

The "--host" option does not require the "--all" option.
If a hostname was specified for the connection and it matches the hostname in the rule, then the rule will match.
Try retrying on the latest commit b51a117
I'll try to reproduce your situation, but it will take some time.

[patricknugent-datto]

Tried giving the latest commit a go but no luck there so far, still the same behaviour as before.

Sounds good, if I also manage to find anything on my end in the meantime then I'll let you know.