31 Commits

Author SHA1 Message Date
Toni Uhlig
54e0601fec Unified IO buffer mgmt.
* c-collectd gives the user control over collectd-exec instance name
 * added missing collectd type `flow_l4_icmp_count`

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-07 15:04:46 +02:00
Toni Uhlig
f713ec702b Added nDPId semantic validation test.
* fixed inconsistent processing of remaining flows during nDPId shutdown phase
 * fixed multiple `detected' flow events
   (instead only `detection-update' flow events can occur after a `detected' flow event)
 * fixed nDPIsrvd.py invalid message buffer handling
 * improved run_tests.sh so only valid pcap capture files are getting processed
   (and some more cosmetics + logging)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-04-14 22:04:42 +02:00
Toni Uhlig
514c427917 Fixed nDPIsrvd.h flow end bug (flow end callback could never be called and caused some memory troubles).
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-04-11 12:34:38 +02:00
Toni Uhlig
131cf5385b Added IPv6 support for -I / -E.
* added another Python search path and try-catch ModuleNotFoundError again
 * run_tests.sh checks for OpenBSD netcat (required for -q)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-04-11 00:16:52 +02:00
Toni Uhlig
4e583cd4de Added JSON schema validation to run_tests.sh
* Python3 scripts are now compatible with versions <3.6
 * improved and prettified run_tests.sh

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-04-09 14:12:48 +02:00
Toni Uhlig
0a7ad7a76a nDPId-test: added JSON distribution + JSON parsing (Multithreaded design re-using most of nDPId/nDPIsrvd core)
* improved Makefile.old install targets
 * splitted nDPIsrvd_parse into nDPIsrvd_parse_line and nDPIsrvd_parse_all for the sake of readability
 * minor Python script improvments (check for nDPIsrvd.py on multiple locations, may be superseeded by setuptools in the future)
 * some paths needs to be absolute (chdir() during daemonize) and therefor additional checks introduced
 * test run script checks and fails if certain files are are missing (PCAP file <=> result output file)
 * removed not very useful "internal format error" JSON serialization if a BUG for same exists
 * fixed invalid l4 type statistics counters for nDPIsrvd-collectd

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-04-09 00:18:35 +02:00
Toni Uhlig
bdc8c5df2a Reduced code duplication. Preps for nDPId-test.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-03-23 14:25:56 +01:00
Toni Uhlig
c68c1750ba Switched to CMake build system.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-03-17 17:41:19 +01:00
Toni Uhlig
1c3ef69faa nDPIsrvd collectd-exec overhaul.
* Install targets updated.
 * Removed nDPIsrvd.h token validation function (done automatically by token_get).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-03-15 14:39:43 +01:00
Toni Uhlig
412d8feba0 Added missing enum strings.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-03-09 17:46:16 +01:00
Toni Uhlig
4699263d65 Fixed possible NULL pointer deref in dependencies/nDPIsrvd.h and examples/c-captured/c-captured.c
* `make install` C examples
 * examples/py-flow-info/flow-info.py supports filtering
 * added breed to examples/py-flow-info/flow-info.py

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-28 11:27:22 +01:00
Toni Uhlig
e0310d7e1d Finalized examples/c-captured to dump packet bytes to PCAP for further analysis.
* Fixed memory holes in nDPId structs.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-24 18:43:26 +01:00
Toni Uhlig
0b5b177c14 Extended nDPIsrvd.h with address parsing.
* nDPId supports looading of custom nDPI protocol/category files
 * extended JSON schemas according to nDPI / nDPId JSON serializing
 * removed memory holes in nDPId
 * extended examples/c-captured

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-23 14:46:47 +01:00
Toni Uhlig
0a6d44dc60 C-Api uses similiar flow key hash calculation as Python-Api.
* Make use of flow id / alias / source which is required for future use cases where multiple nDPId instances (same / different machines) feed one nDPIsrvd.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-19 18:39:14 +01:00
Toni Uhlig
a1805eb891 Added JSON schema files and a Python schema validator.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-17 14:00:10 +01:00
Toni Uhlig
893f437051 Aligned nDPIsrvd.py to nDPIsrvd.h for consistency.
* Simplified Python interface as well.
 * c-captured and flow-undetected-to-pcap.py produce similiar results
 * Removed overloaded nDPIsrvd.py event structures.
 * flow-info.py prints (with a color-hash) additional information e.g. alias/source and midstream

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-16 20:43:00 +01:00
Toni Uhlig
69b4e662c7 nDPIsrvd C API overhaul and massive simplification.
* nDPIsrvd.h does flow mgmt out of the box
 * dissect received JSON strings via callback
 * added new JSON key/values for packet-flows (usecTimestamp/L3/L4 info)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-06 15:41:30 +01:00
Toni Uhlig
257cce1dcc Fixed braindead failure - Increase of NETWORK_BUFFER_MAX_SIZE > 9999 means to also increase NETWORK_BUFFER_LENGTH_DIGITS to 5.
* Fixed ARM32 xcompile warnings; Other GCC versions, other uint64_t's..
 * Replaced ridiculous nDPIsrvd_JSON_BYTES with NETWORK_BUFFER_LENGTH_DIGITS.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-01-27 18:05:13 +01:00
Toni Uhlig
9564b0ce2c Increased JSON buffer size to 12288 (libnDPI serializes more and more information).
* Making Compare&Fetch mandatory.
 * Added some more Compare&Fetch to prevent TSAN complaining about data races.
   Fixed possible but more ore less harmless data races during shutdown process.
 * Shrink SIGNAL handler to a minimum. SYSV Signal handling and MT-safety is awkward.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-01-27 17:19:29 +01:00
Toni Uhlig
102b61175c All colors are beautiful for py-flow-info.
* instance alias is now mandatory (will be required for future use-cases)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-01-26 19:07:16 +01:00
Toni Uhlig
25c23e3d23 Finished (mostly) C guessed/undetected flow dumper.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-01-26 18:31:24 +01:00
Toni Uhlig
b82a535a45 Increased network buffer size for python scripts (and raise exception if capacity reached).
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-01-08 10:54:44 +01:00
Toni Uhlig
613e60ca2a Declared nDPI id and flow structs in nDPId flow info struct.
Two reasons:
 * reduce heap memory allocations
 * nDPId flow info struct may be inflated in the future (more bytes to compress)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-12-30 13:28:32 +01:00
Toni Uhlig
e864787848 Generic nDPIsrvd.h event handling integration and flow tracking with uthash.
* aims to be re-usable for many different apps and use-cases

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-12-06 17:01:06 +01:00
Toni Uhlig
00e5132a80 Merge commit 'c8bf38e5fb717d40635a2a89b22ed71b0de4266b' as 'dependencies/uthash' 2020-12-01 13:33:34 +01:00
Toni Uhlig
32b192df3b Removed all LUA leftover as preparation for UTHASH.
* Added UNIX socket support for nDPIsrvd.h (and c-captured)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-12-01 13:31:07 +01:00
Toni Uhlig
54dd72676d Provide functionality do deal with tokens from JSMN.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-11-17 19:29:10 +01:00
Toni Uhlig
a03e0c8ba8 Fixed inconsistent callback function call if a token parsed.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-11-13 19:55:54 +01:00
Toni Uhlig
aac0e77a2c Add 'dependencies/jsmn/' from commit '053d3cd29200edb1bfd181d917d140c16c1f8834'
git-subtree-dir: dependencies/jsmn
git-subtree-mainline: f79991d70c
git-subtree-split: 053d3cd292
2020-11-13 14:45:57 +01:00
Toni Uhlig
f79991d70c Re-Cloned JSMN as subtree.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-11-13 14:44:39 +01:00
Toni Uhlig
7853c79cd6 Renamed misleading "contrib" subfolder to "dependencies".
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-11-13 14:26:51 +01:00