* bump libnDPI to c53c82d4823b5a8f856d1375155ac5112b68e8af
* run_tests.sh: improved execution from non-git directories e.g. via `make dist`
* updated JSON schema to be more restrictive
* nDPId: splitted generic get_ip_from_sockaddr into IPv4/IPv6 to prevent compiler warnings on some platforms
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* TCP timeout after FIN/RST: switched back to the value from a35fc1d5ea
* py-flow-info: reset 'guessed' flag after detection/detection-update received
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
- nDPId: fixed invalid IP4/IP6 tuple compare
- nDPIsrvd: fixed caching issue (finally)
- added tiny c example (can be used to check flow manager sanity)
- c-captured: use flow_last_seen timestamp from `struct nDPIsrvd_flow`
- README.md update: added example JSON sequence
- nDPId: added new flow event `update` necessary for correct
timeout handling (and other future use-cases)
- nDPIsrvd.h and nDPIsrvd.py: switched to an instance
(consists of an alias/source tuple) based flow manager
- every flow related event **must** now serialize `alias`, `source`,
`flow_id`, `flow_last_seen` and `flow_idle_time` to make the timeout
handling and verification process work correctly
- nDPIsrvd.h: ability to profile any dynamic memory (de-)allocation
- nDPIsrvd.py: removed PcapPacket class (unused)
- py-flow-dashboard and py-flow-multiprocess: fixed race condition
- py-flow-info: print statusbar with probably useful information
- nDPId/nDPIsrvd.h: switched from packet-flow only timestamps (`pkt_*sec`)
to a generic flow event timestamp `ts_msec`
- nDPId-test: added additional checks
- nDPId: increased ICMP flow timeout
- nDPId: using event based i/o if capturing packets from a device
- nDPIsrvd: fixed memory leak on shutdown if remote descriptors
were still connected
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPId-test: disable #include <syslog.h> if NO_MAIN macro defined
* nDPId-test: mock syslog flags and functions
* gitlab-ci: force -Werror
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* After the first packet was processed, "flow_last_seen" was still 0.
This behaviour is invalid as the first packet may contain l4 payload data e.g. for UDP
and it also breaks nDPId json consistency "flow_first_seen" > 0, but "flow_last_seen" == 0.
* JSON schema: set minimum timestamp value for Epoch timestamps to 24710 for flow_*_seen and
1 for pcap packet ts. Those values are dependant on some manipulated pcap's in libnDPI/tests/pcap.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* more structs are now "compressable"
* fixed missing DAEMON_RECONNECT event
* improved memory profiler
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* c-collectd gives the user control over collectd-exec instance name
* added missing collectd type `flow_l4_icmp_count`
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPIsrvd services usually do not care about layer4 data length,
payload length is quite more essential for further processing
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* fixed inconsistent processing of remaining flows during nDPId shutdown phase
* fixed multiple `detected' flow events
(instead only `detection-update' flow events can occur after a `detected' flow event)
* fixed nDPIsrvd.py invalid message buffer handling
* improved run_tests.sh so only valid pcap capture files are getting processed
(and some more cosmetics + logging)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* added another Python search path and try-catch ModuleNotFoundError again
* run_tests.sh checks for OpenBSD netcat (required for -q)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* generate DAEMON_EVENT_INIT as well as DAEMON_EVENT_SHUTDOWN
* process remaining flows before shutdown (and generate events)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* improved Makefile.old install targets
* splitted nDPIsrvd_parse into nDPIsrvd_parse_line and nDPIsrvd_parse_all for the sake of readability
* minor Python script improvments (check for nDPIsrvd.py on multiple locations, may be superseeded by setuptools in the future)
* some paths needs to be absolute (chdir() during daemonize) and therefor additional checks introduced
* test run script checks and fails if certain files are are missing (PCAP file <=> result output file)
* removed not very useful "internal format error" JSON serialization if a BUG for same exists
* fixed invalid l4 type statistics counters for nDPIsrvd-collectd
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
How? After the detection finished, internal ndpi structs can be free'd as they are not needed anymore.
* Set the amount of max. packets to process via subopt.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* fixed invalid flow event schema type
* added run_tests.sh to generate/diff JSON dumps
* renamed lot's of vars/fns in nDPId.c/nDPIsrvd.c, so nDPId-test.c can include "*.c"
* improved CMake dependency checks
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* nDPIsrvd.h does flow mgmt out of the box
* dissect received JSON strings via callback
* added new JSON key/values for packet-flows (usecTimestamp/L3/L4 info)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fixed ARM32 xcompile warnings; Other GCC versions, other uint64_t's..
* Replaced ridiculous nDPIsrvd_JSON_BYTES with NETWORK_BUFFER_LENGTH_DIGITS.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Making Compare&Fetch mandatory.
* Added some more Compare&Fetch to prevent TSAN complaining about data races.
Fixed possible but more ore less harmless data races during shutdown process.
* Shrink SIGNAL handler to a minimum. SYSV Signal handling and MT-safety is awkward.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* only IPv4 supported for now
* refactored nDPId's internal IP address storage
* use fresh ndpi_free_flow_data() to free nDPI's dynamic allocated data
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Two reasons:
* reduce heap memory allocations
* nDPId flow info struct may be inflated in the future (more bytes to compress)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Switching back to blocking mode works as a quick fix but is not sufficient.
See comments.
* nDPId prints more accurate error messages if command line argument validation failed
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added pcap diff script
* Added \n to JSON string end (useful for debugging and readability)
* Use first host/server name character for hash calculation as well
* Removed error'ing EPOLLHUP handling in nDPIsrvd (connection closing will be detected via read())
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
