80 Commits

Author SHA1 Message Date
Toni
42c54d3755 Initial tunnel decoding (GRE - Layer4 only atm) (#55)
Initial tunnel decoding (GRE - Layer4 only atm). Fixes #53
 * make finally use of the thread distribution seed
 * Handle GRE/PPP subprotocol the right way
 * Add `-t` command line / config option
 * Removed duplicated and obsolete IP{4,6}_SIZE_SMALLER_THAN_HEADER which is the same as IP{4,6}_PACKET_TOO_SHORT
 * Updated error event schema

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2025-02-25 15:17:16 +01:00
Toni Uhlig
9105b393e1 Fixed some SonarCloud issues
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-11-14 10:21:35 +01:00
Toni Uhlig
5423797267 Added nDPId ndpi_process_packet() LLVM fuzzer
* replaced dumb `dumb_fuzzer.sh`
 * fixed nDPId NULL pointer deref found by fuzzer
 * nDPI: `--enable-debug-build` and `--enable-debug-messages` for non release builds
 * nDPI: do not force `log.level` to `3` anymore, use config value instead

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-10-24 15:45:04 +02:00
Toni Uhlig
7d58703bdb Removed ENABLE_MEMORY_STATUS CMake option as it's now enabled for **all** builds
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-10-22 13:12:59 +02:00
Toni Uhlig
ae36f8df6c Added libnDPI global context init/deinit used for cache mgmt.
* support for adding *.ndpiconf for nDPI config tests
 * all other configs should have the suffix *.conf
 * fixed nDPI malloc/free wrapper set (was already too late set)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-10-21 16:17:01 +02:00
Toni Uhlig
8c5ee1f7bb Added config testing script.
* nDPId-test may now make use of an optional config file as cmd arg

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-10-21 16:10:09 +02:00
Toni
efed6f196e Read and parse configuration files. Fixes #41. (#42)
Read and parse configuration files. Fixes #41.

 * supports nDPId / nDPIsrvd via command line parameter `-f`
 * nDPId: read general/tuning and libnDPI settings
 * support for settings risk domains libnDPI option via config file or via `-R` (Fixes #45, thanks to @UnveilTech)
 * added some documentation in the config file
 * adjusted Systemd and Debian packaging to make use of config files

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-10-16 14:13:55 +02:00
Toni
5e4005162b Add PF_RING support. (#38) 2024-08-19 18:33:18 +02:00
Toni Uhlig
8af37b3770 Fix some SonarCloud complaints.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-02-13 07:21:47 +01:00
Toni Uhlig
71d933b0cd Fixed an event issue.
* a "detection-update" event was thrown even if nothing changed
 * in some cases "not-detected" events were spammed if detection not completed
 * tell `libnDPI` how many packets per flow we want to dissect
 * `nDPId-test` validates total active flows in the right way

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-12-14 15:45:08 +01:00
Toni Uhlig
f5c5bc88a7 Replaced ambiguous naming of "JSON string" to more accurate "JSON message". #2
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-12-04 13:13:05 +01:00
Toni Uhlig
53d8a28582 Replaced ambiguous naming of "JSON string" to more accurate "JSON message".
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-12-04 13:01:27 +01:00
Toni Uhlig
ff77bab398 Warn about unused return values that are quite important.
* CI: ArchLinux build should now instrument `-Werror`
 * CI: Increased OpenWrt build verbosity

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-12-01 06:43:39 +01:00
Toni Uhlig
bb9f02719d Added SonarCloud exclusions for third-party files and files lacking relevance.
* fixed two other "bugs"

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-20 14:51:59 +01:00
Toni Uhlig
82934b7271 Fixed clang-tidy warnings.
* fixed/improved c-captured logging

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-13 13:52:42 +01:00
Toni Uhlig
5d56288a11 Fixed more SonarCloud complaints.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-07 00:02:36 +01:00
Toni Uhlig
84b12cd02c Fixed some SonarCloud complaints.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-06 13:31:59 +01:00
Toni
93498fff02 Apple/BSD port (#30)
* Add MacOS to Github CI builds.
* Fixed libnDPI-4.8 CI build.
* Fixed missing include for `struct sockaddr*`.
* Reworked IPv4 address and netmask retrieval.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-06 13:25:49 +01:00
Toni
1b67927169 Event I/O abstraction layer. (#28)
* Finalize Event I/O abstraction layer.
* Fix possible fd leakage, Gitlab-CI build and error logging.
* Fixed possible uninitialized signalfd variable.
* Fixed possible memory leak.
* Fixed some SonarCloud complaints.
* Fixed nDPId-test nDPIsrvd-arpa-mockup stuck indefinitely.
* Add nDPId / nDPIsrvd command line option to use poll() on Linux instead of the default epoll().

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-06 12:38:15 +01:00
Toni Uhlig
44adfc0b7d Sonarcloud integration
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-05 17:37:42 +02:00
Toni Uhlig
dfd0449306 Fix issues detected by SonarCloud.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-05 17:26:06 +02:00
Toni Uhlig
07f2c2d9cc nDPId-test: ingore event handler failures caused by arpa mockup
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-05 17:25:54 +02:00
Toni Uhlig
73b8c378f2 nDPId event I/O fixes.
* forcibly disable epoll even if available
 * nDPId-test event I/O selftest
 * CI event I/O tests

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-05 12:32:45 +02:00
Toni Uhlig
96b0a8a474 Add event I/O abstraction.
* required to support non-Linux OS e.g. Mac OS X / BSD
 * see Github issue #19

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-09-10 19:11:58 +02:00
Toni Uhlig
cc60e819e8 Fixed invalid base64 encoding in some rare cases.
* nDPId-test may also verify the correct encoding/decoding

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-24 22:51:57 +02:00
Toni Uhlig
0e31829401 nDPId-test: threads should block all unix signals
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-02 23:34:10 +02:00
Toni Uhlig
d9f304e4b0 nDPId-test: print additional startup/init log messages
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-02 22:32:33 +02:00
Toni Uhlig
c57ace2fd3 Correctly handle EINTR while doing I/O..
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-07-18 13:51:00 +02:00
Toni Uhlig
57d8dda350 nDPId-test: Fixed invalid error retval when epoll_wait() returns EINTR.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-07-16 23:23:07 +02:00
Toni Uhlig
2c95b31210 nDPId-test: Reworked I/O handling to prevent some endless loop scenarios. Fixed a race condition in the memory wrapper as well.
* nDPId: Instead of sending too long JSON strings, log an error and some parts.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-12-02 22:11:57 +01:00
Toni Uhlig
64f6abfdbe Unified nDPId/nDPIsrvd command line argument storage.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-11-21 11:26:05 +01:00
Toni Uhlig
ce567ae5b7 Improved the point of time when to append the raw packet base64 data to the serializer.
* nDPId-test: Increased the max-packets-per-flow-to-send from 3 to 5.
   This is quite useful for TCP as the first 3 packets are usually part of the three-way-handshake.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-11-15 06:25:16 +01:00
Toni Uhlig
805aef5de8 Increased network buffer size to 33792 bytes.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-10-30 22:13:07 +01:00
Toni Uhlig
2d14509f04 nDPid-test: add buffer test
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-10-30 20:12:17 +01:00
Toni Uhlig
916d2df6ea nDPId-test: Fixed thread sync/lock issue.
* rarely happens in CI

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-10-22 01:45:14 +02:00
Toni Uhlig
b6060b897e c-analysed: improved feature extraction from "analyse" events
* c-captured: update detected risks on "detection-update" events
 * c-collectd: added missing flow breed
 * c-collectd: PUTVAL macros are more flexible now

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-10-01 18:01:56 +02:00
Toni Uhlig
14f6b87551 Added nDPIsrvd-analysed to generate CSV files from analyse events.
* nDPIsrvd.h: iterate over JSON arrays
 * nDPId: calculate l3 payload packet entropies for analysis

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-30 19:28:49 +02:00
Toni Uhlig
74f71643da nDPId-test: Force collector blocking mode.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-25 16:24:05 +02:00
Toni Uhlig
2103ee0811 Refactored client distributor C API.
* Still not perfect, but the code before was not even able to deal with JSON arrays.
   Use common "speaking" function names for all functions in nDPIsrvd.h
 * Provide a more or less generic and easy extendable JSON walk function.
 * Modified C examples to align with the changed C API.
 * c-collectd: Reduced lot's of code duplication by providing mapping tables.
 * nDPId: IAT array requires one slot less (first packet has always an IAT of 0).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-25 00:54:39 +02:00
Toni Uhlig
36f1786bde nDPIsrvd.h: Fixed bug during token parsing/hashing. Do not hash array contents.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-23 00:13:19 +02:00
Toni Uhlig
9a28475bba Improved flown analyse event:
* store packet directions
 * merged direction based IATs
 * merged direction based PKTLENs

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-22 19:07:08 +02:00
Toni Uhlig
08f263e409 nDPId: Reduced flow-updates for TCP flows to 1/4 of the timeout value.
* nDPId: Fixed broken validation tests.
 * nDPId: Removed TICK_RESOLUTION, not required anymore.
 * c-collectd: Improved total layer4 payload calculation/update handling.
 * c-collectd: Updated RRD Graph script according to total layer4 payload changes.
 * py-flow-info.py: Fixed several bugs and syntax errors.
 * Python scripts: Added dirname(argv[0]) as search path for nDPIsrvd.py.
 * nDPIsrvd&nDPId-test: Fixed missing EPOLLERR check.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-19 19:39:49 +02:00
Toni Uhlig
efaa76e978 Provide thread sync via locking on architectures that do not support Compare&Swap.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-17 18:27:17 +02:00
Toni Uhlig
b3e9af495c Add OpenWrt CI via Github Actions.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-17 10:31:26 +02:00
Toni Uhlig
d4633c1192 New flow event: 'analysis'.
* The goal was to provide a separate event for extracted feature that are not required
   and only useful for a few (e.g. someone who wants do ML).
 * Increased network buffer size to 32kB (8192 * 4).
 * Switched timestamp precision from ms to us for *ALL* timestamps.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-13 22:05:08 +02:00
Toni Uhlig
a992c79ab6 Fixed compilation warnings on linux32 platforms.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-10 22:45:12 +02:00
Toni Uhlig
f9bd7d29ce Bump libnDPI to 37f918322c0a489b5143a987c8f1a44a6f78a6f3 and updated flow json schema file.
* export env vars AR / CMAKE_C_COMPILER_AR and RANLIB / CMAKE_C_COMPILER_RANLIB while building libnDPI
 * nDPId check API version during startup (macro vs. function call) and print a warning if they are different

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-06 14:50:46 +02:00
lns
d646ec5ab4 nDPId: Fixed fcntl() issue; invalid fcntl() set after a blocking-write.
* nDPId: imrpvoed collector socket error messages on connect/write/etc failures
 * reverted `netcat` parts of the README

Signed-off-by: lns <matzeton@googlemail.com>
2022-08-29 15:29:07 +02:00
Toni Uhlig
0fd59f060e Split *_l4_payload_len' into *_src_l4_payload_len' and `*_dst_l4_payload_len'.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-08-15 22:55:19 +02:00
lns
905545487d Split flow_packets_processed' into flow_src_packets_processed' and `flow_dst_packets_processed'.
* no use for `flow_avg_l4_payload_len' -> removed
 * test/run_tests.sh does not fail if git-worktree's are used

Signed-off-by: lns <matzeton@googlemail.com>
2022-08-15 18:36:49 +02:00