Commit Graph

40 Commits

Author SHA1 Message Date
Toni Uhlig
471ea83493 bump libnDPI to e946f49aca13e4447a7d7b2acae6323a4531fb55
* incorporated upstream changes

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2025-01-25 10:07:25 +01:00
Toni Uhlig
ce9752af16 Fixed some SonarCloud issues
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-11-05 13:43:23 +01:00
Toni Uhlig
f7933d0fdb Slightly unified C example's logging
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-11-05 12:48:36 +01:00
Toni Uhlig
25944e2089 Fixed some SonarCloud issues
* fixed dependabot werkzeug (3.0.3 to 3.0.6)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-10-26 11:35:30 +02:00
Toni Uhlig
762e6d36bf Some small fixes.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-08-09 11:09:39 +02:00
Toni Uhlig
717d66b0e7 Fixed missing statistics updating for unknown mapping keys in collectd/influxd.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-02-20 23:16:31 +01:00
Toni Uhlig
feb2583ef6 bump libnDPI to 4543385d107fcc5a7e8632e35d9a60bcc40cb4f4
* incorporated API changes from nDPI

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-02-06 10:34:52 +01:00
Toni Uhlig
a007a907da Fixed invalid flow risk aggregation in collectd/influxd examples.
* CI: build single nDPId executable with `-Wall -Wextra -std=gnu99`
 * fixed missing error events in influxd example
 * added additional test cases for collectd
 * extended grafana dashboard

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-01-06 19:32:47 +01:00
Toni Uhlig
876aef98e1 Improved collectd example.
* similiar behavior to influxd example
 * gauges and counters are now handled properly

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-01-05 11:26:53 +01:00
Toni Uhlig
53d8a28582 Replaced ambiguous naming of "JSON string" to more accurate "JSON message".
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-12-04 13:01:27 +01:00
Toni Uhlig
7368d34d8d c-collectd: Fixed missing escape char.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-12-01 06:43:39 +01:00
Toni Uhlig
5335d84fe5 Add DBUS suspicious flow event notification daemon.
* nDPIsrvd.h: support for closing/resetting a nDPIsrvd_socket (required for a reconnect)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-03 16:07:28 +01:00
Toni Uhlig
dfd0449306 Fix issues detected by SonarCloud.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-05 17:26:06 +02:00
Toni Uhlig
7515c1b072 Fix CI.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-09-10 20:28:11 +02:00
Toni Uhlig
ebb439d959 Tiny improvments.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-02 21:00:39 +02:00
Toni Uhlig
c8ec505b9c bump libnDPI to 8ea0eaa0d0c4a3be05f67ef7fa1d22c2579cf7d1
* added build fix for Gitlab CI
 * added friendly C11 check
 * set required libnDPI versionto 4.7
   (ArchLinux ndpi-git sets version to 4.7, which is not released yet)
 * reduced sklearn-random-forest memory consumption by adjusting min. sample leaf

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-06-12 19:05:33 +02:00
Toni Uhlig
a3d20c17d1 Improved collectd risk processing to be in sync with libnDPI risks.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-01-11 06:28:10 +01:00
Toni Uhlig
c5930e3510 Add collectd statistics diff test.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-12-06 19:51:53 +01:00
Toni Uhlig
c63cbec26d Improved nDPIsrvd-collectd statistics.
* Improved RRD-Graph generation script and static WWW html files.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-10-31 12:45:15 +01:00
lns
49ea4f8474 Small fixes.
Signed-off-by: lns <matzeton@googlemail.com>
2022-10-01 22:37:25 +02:00
Toni Uhlig
b6060b897e c-analysed: improved feature extraction from "analyse" events
* c-captured: update detected risks on "detection-update" events
 * c-collectd: added missing flow breed
 * c-collectd: PUTVAL macros are more flexible now

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-10-01 18:01:56 +02:00
Toni Uhlig
2103ee0811 Refactored client distributor C API.
* Still not perfect, but the code before was not even able to deal with JSON arrays.
   Use common "speaking" function names for all functions in nDPIsrvd.h
 * Provide a more or less generic and easy extendable JSON walk function.
 * Modified C examples to align with the changed C API.
 * c-collectd: Reduced lot's of code duplication by providing mapping tables.
 * nDPId: IAT array requires one slot less (first packet has always an IAT of 0).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-25 00:54:39 +02:00
Toni Uhlig
36f1786bde nDPIsrvd.h: Fixed bug during token parsing/hashing. Do not hash array contents.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-23 00:13:19 +02:00
Toni Uhlig
9a28475bba Improved flown analyse event:
* store packet directions
 * merged direction based IATs
 * merged direction based PKTLENs

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-22 19:07:08 +02:00
Toni Uhlig
08f263e409 nDPId: Reduced flow-updates for TCP flows to 1/4 of the timeout value.
* nDPId: Fixed broken validation tests.
 * nDPId: Removed TICK_RESOLUTION, not required anymore.
 * c-collectd: Improved total layer4 payload calculation/update handling.
 * c-collectd: Updated RRD Graph script according to total layer4 payload changes.
 * py-flow-info.py: Fixed several bugs and syntax errors.
 * Python scripts: Added dirname(argv[0]) as search path for nDPIsrvd.py.
 * nDPIsrvd&nDPId-test: Fixed missing EPOLLERR check.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-19 19:39:49 +02:00
Toni Uhlig
94aa02b298 nDPIsrvd-collectd: Stdout should be unbuffered.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-12 13:32:50 +02:00
Toni Uhlig
dcf78ad3ed Disable timestamp generation in nDPIsrvd-collectd as default.
* collectd's rrdtool write plugin does silently fail with those ones (dunno why)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-04 17:24:10 +02:00
Toni Uhlig
0fd59f060e Split *_l4_payload_len' into *_src_l4_payload_len' and `*_dst_l4_payload_len'.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-08-15 22:55:19 +02:00
Toni Uhlig
2cb2c86cb5 c-collectd: fixed incorrect PUTVAL
* get rid of types.db

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-08-15 16:42:59 +02:00
Toni Uhlig
fc442180da c-collectd: fixed possible undefined behavior
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-14 03:48:57 +02:00
lns
2a8883a96e CMake: do not add /usr/include/ndpi to include dirs if BUILD_NDPI or STATIC_LIBNDPI_INSTALLDIR used.
* c-collectd: fixed memory leak on failure
 * py-flow-info.py: fancy spinners and stats counting improved

Signed-off-by: lns <matzeton@googlemail.com>
2022-06-10 14:34:30 +02:00
lns
c7bf94e9f1 nDPIsrvd.(h|py): Added socket read/recv timeout.
* nDPIsrvd.h: support for O_NONBLOCK nDPIsrvd_socket

Signed-off-by: lns <matzeton@googlemail.com>
2022-04-17 18:56:30 +02:00
Toni Uhlig
c0b7bdacbc Reworked nDPIsrvd.h C-API.
* nDPIsrvd.h: Provide nDPId thread storage.
 * nDPIsrvd.py: Fixed instance cleanup bug.
 * nDPIsrvd.h: Support for instance/thread user data and cleanup callback.
 * nDPIsrvd.h: Most recent flow time stored in thread ht instead of instance ht.
 * nDPId: Moved flow logger out the memory profilier into SIGUSR1 signal handling.
 * nDPId: Added signal fd to be usable within epoll's event handling (live-capture only!)
 * nDPId: Added information about ZLib compressions to daemon status/shutdown events.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-21 15:56:01 +01:00
Toni Uhlig
9e07a57566 Major nDPId extension. Sorry for the huge commit.
- nDPId: fixed invalid IP4/IP6 tuple compare
 - nDPIsrvd: fixed caching issue (finally)
 - added tiny c example (can be used to check flow manager sanity)
 - c-captured: use flow_last_seen timestamp from `struct nDPIsrvd_flow`
 - README.md update: added example JSON sequence
 - nDPId: added new flow event `update` necessary for correct
   timeout handling (and other future use-cases)
 - nDPIsrvd.h and nDPIsrvd.py: switched to an instance
   (consists of an alias/source tuple) based flow manager
 - every flow related event **must** now serialize `alias`, `source`,
   `flow_id`, `flow_last_seen` and `flow_idle_time` to make the timeout
   handling and verification process work correctly
 - nDPIsrvd.h: ability to profile any dynamic memory (de-)allocation
 - nDPIsrvd.py: removed PcapPacket class (unused)
 - py-flow-dashboard and py-flow-multiprocess: fixed race condition
 - py-flow-info: print statusbar with probably useful information
 - nDPId/nDPIsrvd.h: switched from packet-flow only timestamps (`pkt_*sec`)
   to a generic flow event timestamp `ts_msec`
 - nDPId-test: added additional checks
 - nDPId: increased ICMP flow timeout
 - nDPId: using event based i/o if capturing packets from a device
 - nDPIsrvd: fixed memory leak on shutdown if remote descriptors
   were still connected

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-20 00:50:38 +01:00
Toni Uhlig
3a76035570 bump libnDPI to 6b7e5fa8d251f11c1bae16ea892a43a92b098480
* fixed linking issue by using CMake to check if explicit link against libm required
 * make nDPIsrvd collectd exit if parent pid changed, meaning that collectd died somehow
 * nDPId-test restores SIGPIPE to the default handler (termination), so abnormal connection drop's do now have consequences

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-08-03 12:37:59 +02:00
Toni Uhlig
54e0601fec Unified IO buffer mgmt.
* c-collectd gives the user control over collectd-exec instance name
 * added missing collectd type `flow_l4_icmp_count`

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-07 15:04:46 +02:00
Toni Uhlig
0a7ad7a76a nDPId-test: added JSON distribution + JSON parsing (Multithreaded design re-using most of nDPId/nDPIsrvd core)
* improved Makefile.old install targets
 * splitted nDPIsrvd_parse into nDPIsrvd_parse_line and nDPIsrvd_parse_all for the sake of readability
 * minor Python script improvments (check for nDPIsrvd.py on multiple locations, may be superseeded by setuptools in the future)
 * some paths needs to be absolute (chdir() during daemonize) and therefor additional checks introduced
 * test run script checks and fails if certain files are are missing (PCAP file <=> result output file)
 * removed not very useful "internal format error" JSON serialization if a BUG for same exists
 * fixed invalid l4 type statistics counters for nDPIsrvd-collectd

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-04-09 00:18:35 +02:00
Toni Uhlig
1c3ef69faa nDPIsrvd collectd-exec overhaul.
* Install targets updated.
 * Removed nDPIsrvd.h token validation function (done automatically by token_get).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-03-15 14:39:43 +01:00
Toni Uhlig
9a06b97473 Fixed collectd-exec issues.
* Added collectd example config and types.db

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-03-12 17:30:04 +01:00
Toni Uhlig
772b67b767 Added basic collectd-exec example.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-03-10 18:32:09 +01:00