zmq format for ntopng incorrect? (openflow support) #52
Comments
|
Have you looked at the NetFlow spec? That's mostly what I used to map things. If I had to guess, the |
|
the .XXX is actually a IANA PEN specifically for nprobe (see https://www.ntop.org/nprobe/iana-assigned-a-pen-to-ntop/). But it turns out that this isn't actually required for sending flows to ntopng. I ran a few tests and stripped down the amount of fields sent by nprobe to this: I also modified https://github.com/synfinatic/netflow2ng/blob/main/cmd/zmq.go to output exactly the same (note: omitting IPV6 here). This is an example of a zmq message from nprobe: This is an example of a zmq message from netflow2ng: As you can see, the output has the same structure. Still, the nprobe messages result in actual flows in ntopng, and the netflow2ng messages don't result in flows in ntopng, only in zmq messages. By the way, the ntopng log doesn't complain about anything (no errors, no warnings at debug log level). Do you have any pointers on how to proceed here? Did you try to use netflow2ng with ntopng 6.0? |
This comment was marked as outdated.
This comment was marked as outdated.
|
I haven't looked at this at all to be honest... I'm not even sure if I'm using noting 6 or 5.x off the top of my head since I very rarely even look at it anymore. Probably worth looking at the ntopng code and see what it's doing? |
|
Hi! I'm seeing what I think is the same issue with Ntopng 6.1 - seeing collected ZMQ messages, but no collected flows and no traffic. I turned the logging level up to 6 and buried amongst the debug messages, I did see:
I've only had time to quickly review the code for Ntopng, but this statement seems to come from Ntop.cpp where there are a few functions that check if an interface is allowed. I must confess I haven't fully understood them yet. I'm not clear what the issue is yet - whether ZMQ interfaces have become a licensable feature and so are not allowed. I can't find any reference in the GUI to allowing interfaces, or in the documentation (which in turn leads me to think this might be license related but I must stress this is an assumption). It could be a separate issue of course, but it seems rather coincidental that I'm seeing the same counter behaviour as shown in the original post, and it appears that Ntopng is deliberately dropping this traffic because the interface "isn't allowed". The Ntopng documentation does not make is clear that this is/has become a licensable feature. |
|
I have the same problem with ntopng 6.0 (stable) ---> no traffic being recorded ... all 0 Using ntopng with --interface zmq://x.x.x.x:5556 Would love it if this was corrected :-) |
Hi @ToqQrrl , I think the correct syntax to use is '--interface tcp://x.x.x.x:5556' or I may be missing something? Anyway, with the above in my config I am able to collect flows, but if I look at the hosts, the "Throughput" and "Total Bytes" columns are both 0 for all entries. Using netflow2ng 0.0.5 to consume netflow v9 from a opnsense firewall |
|
Hi @gitet … in the documentation it says that we should use zmq:// instead of tcp:// … but both give the same result … I have le flows but all throughput and total bytes are 0 just like you |
|
I haven't had a chance yet to dig deeper, but looks like https://github.com/ntop/ntopng/blob/dev/src/ZMQParserInterface.cpp has changed significantly from 5.6 to 6.0 which most likely explains what is going on here. |
|
Digging further... not sure what your problem is as I can't reproduce it using ntopng v6.0.240305. Can you share your ntopng.conf. Are you sure your device sending the netflow packets is using v9 and not v10/IPFIX? FWIW, looks like v5 used |
|
@synfinatic Thank you for not giving up on this 👍 I'm sending the same netflow data to a raspberrypi running Netflow2ng info: netflow2ng --version
netflow2ng v0.0.5 -- Copyright 2020-2022 Aaron Turner
c819d0568627c4eaad3982bab0c16c2630bc8ef0 (v0.0.5-7-gc819d05) built at 2024-02-13T20:51:39+0100Invoked with Small json sample produced with {"1":0,"10":2,"11":443,"12":"35.186.227.140","130":"192.168.10.254","14":1,"15":"$MY_ROUTER_IP","16":0,"17":0,"2":0,"21":1710428931,"22":1710428625,"29":24,"30":0,"32":0,"4":6,"44":24,"45":0,"5":0,"52":0,"54":0,"58":0,"59":0,"6":27,"60":4,"61":0,"7":56668,"8":"192.168.10.100","80":"00:00:00:00:00:00","81":"00:00:00:00:00:00","88":0,"89":0} My ntopng config: grep -v '^#.*' /etc/ntopng/ntopng.conf
-G=/var/run/ntopng.pid
-i=wlp4s0
-i=tcp://127.0.0.1:5556
-w=3000
-m=192.168.10.0/24
-q
--community
-v=2BTW: I'm using the ntopng binaries provided via https://packages.ntop.org/ |
|
So going back to the beginning... @level20peon tried to show that nprobe is sending different data than netflow2ng. But this is expected. For example, if you look at ntop_flow.h you'll see there are four different ways the MAC address can be specified: #define IN_SRC_MAC 56
#define OUT_DST_MAC 57
#define IN_DST_MAC 80
#define OUT_SRC_MAC 81And that makes sense since packets have a direction (in vs. out) and you have two addresses (src and dst). So the fact that two different netflow messages use different keys for the MAC addresses makes sense and if you look, both nprobe and netflow2ng are sending valid values. That said, nothing I see in the json output of netflow2ng looks wrong or invalid. Digging through the ntopng code, I don't see anything obvious that would cause any issues and my testing has shown it works fine with both ntopng v5 and v6. If you'd like me to investigate further, frankly I need pcaps of the netflow traffic being sent to netflow2ng and more information about how the netflow traffic being generated. @gitet mentioned using OpnSense which I assume is using softflowd? What version? what is the configuration/CLI arguments? If it helps, here is how my pfSense box is running softflowd v1.0.0: Furthermore, you'll get debugging of ZMQ messages with ntopng at |
|
Hi @synfinatic - I dusted off and booted my old asus router which is running softflowd 1.0.0. Flow stats emitted from softflowd is received and ingested perfectly fine by my ntopng, so I may need to look closer at the stuff emitted by my opnsense box and raise an issue with the opnsense team. Btw: Opnsense is not running softflowd, they are instead using I've decided to just run ntopng on the opnsense box directly instead (via plugin). Works like a charm. If you're interested in a pcap containing datagrams from both of my routers, please drop me a message. If not, my part in this has come to and end for now. Sorry for leading you on a wild goose chase - I should not have assumed the problem was related to netflow2ng or ntopng. Thanks! |
|
My problem is exactly the same … opensense … sorry I couldn’t comment earlier. So you guys think the problem comes from flowd on the opensense box ? |
@ToqQrrl , this is my guess, but the logic behind it is not rock solid. I've tested the following variants (all using netflow v9):
|
|
@gitet , no appologies necessary. We all learned something new today. :) Anyways, the only reference for So I'm happy to see about why this is a problem and try to fix it, but someone is going to have to provide a pcap of say 100-200 netflow v9 packets. Just attach it here to this ticket, or if you're super privacy worried about someone seeing your flow data, just email me.... my email address is my github username at gmail. |
|
@synfinatic , happy to hear that :-) Yes, I'm jumping to conclusions - flowd is a collector. I'm not particularly fluent with FreeBSD, but as far as I am able to tell, netflow is generated by a NETGRAPH kernel module ( root@firewall:/etc/rc.d # ngctl list | grep netflow
Name: ksocket_netflow_igc1 Type: ksocket ID: 000000e4 Num hooks: 1
Name: netflow_igc0 Type: netflow ID: 000000e9 Num hooks: 3
Name: ksocket_netflow_igc0 Type: ksocket ID: 000000ef Num hooks: 1
Name: netflow_igc1 Type: netflow ID: 000000de Num hooks: 3
root@firewall:/etc/rc.d # ngctl show netflow_igc0:
Name: netflow_igc0 Type: netflow ID: 000000e9 Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
export9 ksocket_netflow_igc0 ksocket 000000ef inet/dgram/udp
out1 igc0 ether 00000002 upper
iface1 igc0 ether 00000002 lower
root@firewall:/etc/rc.d # ngctl show netflow_igc1:
Name: netflow_igc1 Type: netflow ID: 000000de Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
export9 ksocket_netflow_igc1 ksocket 000000e4 inet/dgram/udp
out2 igc1 ether 00000003 upper
iface2 igc1 ether 00000003 lower I'll generate a pcap for you and post here ASAP. |
|
netflow.pcapng.gz |
|
So looking at the pcaps:
Honestly, the more I read this comment: #52 (comment) the more I think this has something to do with how you've got ntopng configured or something? At this point, i'd suggest you enable |
synfinatic
changed the title
|
Any update on this issue? I see it's still open. I filed an issue #59, which was flagged as a duplicate of this issue. Just to confirm, with the logging set to debug, should I be seeing output on the console? After netflow2ng starts and I see the listening message, I get nothing more. Hate to ask, but perhaps @synfinatic could attach a basic docker-compose.yml that works so we can rule out netflow2ng and then focus on ntopng and pfsense as the issue? |
synfinatic
changed the title
|
@robgarrett I closed your ticket as a duplicate because based on the limited information you provided it appeared to be a duplicate of this. If you are not getting any netflow2ng logs, then you're not sending netflow traffic to it. If you are using pfSense/softflowd then I am incorrect and this is not the same issue, but most likely an issue with your firewall configuration. |
Correct, I do sometimes see flow information in the metrics page at 0.0.0.0:8080, but nothing on the console. I also configured softflowd per your earlier comment, and ran a packet capture on the docker network and confirmed UDP traffic coming into the netflow2d container on port 2055. Would you consider my request to create a quick docker compose? If not, does the yml file I attached to the original #59 issue work? Any help would be greatly appreciated. |
|
@robgarrett your config looks fine. have you tried increasing ntopng logging? |
|
metrics.txt Going back to netflow2ng for a moment. I see the following block in the metrics, does this mean flows are making it to netflow2ng? flow_traffic_summary_size_bytes{local_ip="10.5.0.11",local_port="2055",remote_ip="192.168.1.1",remote_port="48611",type="NetFlow",quantile="0.5"} NaN |
|
@robgarrett Since you're using softflowd, i'll reopen your ticket and we can stop polluting this one. |
|
Also trying to push data to ntopng v6, and I'm getting the same problem where ntopng is showing zmq messages but 0 flows. Running ntopng in debug mode doesn't give any extra useful information, although sometimes: 20/May/2024 15:28:22 [ZMQCollectorInterface.cpp:353] WARNING: Unsupported publisher version: is your nProbe sender outdated? [319][24][53][2][1] I'm not sure what more info to provide but am happy to do so. |
|
@GyroW what software is Mikrotik using to generate netflow data? did this work with ntopng v5? |
|
@synfinatic Mikrotik is using it's own proprietary "traffic flow". (I think). I tried compiling ntopng v5 but I couldn't get it to work, so I don't know if it does. |
|
@GyroW well this ticket is for openflow support, so maybe you should open a new issue? |
|
A search brought me here... Not sure if this is as-expected... |
I am trying to push some data into ntopng 6.0, but it doesn't receive any flows, despite receiving zmq messages:
So I went ahead and compared the output of netflow2ng (0.0.5) with the output of nprobe (10.5) and they seem to be very different:
netflow2ng output
nprobe output
I am using the @NTOPNG@ template on nprobe, which expands to
%IN_SRC_MAC %OUT_DST_MAC %INPUT_SNMP %OUTPUT_SNMP %SRC_VLAN %IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV6_SRC_ADDR %IPV6_DST_ADDR %IP_PROTOCOL_VERSION %PROTOCOL %L7_PROTO %L7_CONFIDENCE %IN_BYTES %IN_PKTS %OUT_BYTES %OUT_PKTS %FIRST_SWITCHED %LAST_SWITCHED %CLIENT_TCP_FLAGS %SERVER_TCP_FLAGS %L7_PROTO_RISK %L7_RISK_SCORE %EXPORTER_IPV4_ADDRESS %DIRECTION %SAMPLING_INTERVAL %TOTAL_FLOWS_EXP %NPROBE_IPV4_ADDRESS %NPROBE_INSTANCE_NAME %FLOW_SOURCE %POST_NAT_SRC_IPV4_ADDR %POST_NAT_DST_IPV4_ADDR %POST_NAPT_SRC_TRANSPORT_PORT %POST_NAPT_DST_TRANSPORT_PORTLooking at https://github.com/cloudflare/goflow/tree/master/decoders/netflow, I can see how I could manipulate the JSON output by incorporating those additional fields into https://github.com/synfinatic/netflow2ng/blob/main/cmd/zmq.go. However, I am not sure how those "combined" fields (i.e. "35632.527") are created / what they mean. Also, I am not sure why for example the MAC addresses use different identifiers on the nprobe string (for example 56 - IPFIX_FIELD_sourceMacAddress, 57 - IPFIX_FIELD_postDestinationMacAddress) compared to the identifiers netflow2ng uses (for example 80 - IPFIX_FIELD_destinationMacAddress, 81 - IPFIX_FIELD_postSourceMacAddress). Same goes for a couple of the other fields - I guess the template used by netflow2ng at the moment is an old version, compatible with previous versions of ntopng!?
So, for now at least, ntopng doesn't receive any flows from netflow2ng. Could you give any pointers on how to incorporate the changes neccessary to support a more current ntopng version?
The text was updated successfully, but these errors were encountered: