Commit Graph

11 Commits

Author SHA1 Message Date
Toni Uhlig
92b3c76446 Added Keras based Autoencode (Work-in-Progress!)
* minor fixes

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-07-16 22:06:36 +02:00
Toni Uhlig
08f263e409 nDPId: Reduced flow-updates for TCP flows to 1/4 of the timeout value.
* nDPId: Fixed broken validation tests.
 * nDPId: Removed TICK_RESOLUTION, not required anymore.
 * c-collectd: Improved total layer4 payload calculation/update handling.
 * c-collectd: Updated RRD Graph script according to total layer4 payload changes.
 * py-flow-info.py: Fixed several bugs and syntax errors.
 * Python scripts: Added dirname(argv[0]) as search path for nDPIsrvd.py.
 * nDPIsrvd&nDPId-test: Fixed missing EPOLLERR check.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-19 19:39:49 +02:00
Toni Uhlig
6f1f9e65ea Fixed some pyhton issues with static class members.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-08 14:17:24 +01:00
Toni Uhlig
9db048c9d9 Serialize flow risk score / confidence.
* bump libnDPI to 8b062295cc76a60e3905c054ce37bd17669464d1
 * removed ndpi_id_struct's

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-27 02:53:39 +01:00
Toni Uhlig
9e07a57566 Major nDPId extension. Sorry for the huge commit.
- nDPId: fixed invalid IP4/IP6 tuple compare
 - nDPIsrvd: fixed caching issue (finally)
 - added tiny c example (can be used to check flow manager sanity)
 - c-captured: use flow_last_seen timestamp from `struct nDPIsrvd_flow`
 - README.md update: added example JSON sequence
 - nDPId: added new flow event `update` necessary for correct
   timeout handling (and other future use-cases)
 - nDPIsrvd.h and nDPIsrvd.py: switched to an instance
   (consists of an alias/source tuple) based flow manager
 - every flow related event **must** now serialize `alias`, `source`,
   `flow_id`, `flow_last_seen` and `flow_idle_time` to make the timeout
   handling and verification process work correctly
 - nDPIsrvd.h: ability to profile any dynamic memory (de-)allocation
 - nDPIsrvd.py: removed PcapPacket class (unused)
 - py-flow-dashboard and py-flow-multiprocess: fixed race condition
 - py-flow-info: print statusbar with probably useful information
 - nDPId/nDPIsrvd.h: switched from packet-flow only timestamps (`pkt_*sec`)
   to a generic flow event timestamp `ts_msec`
 - nDPId-test: added additional checks
 - nDPId: increased ICMP flow timeout
 - nDPId: using event based i/o if capturing packets from a device
 - nDPIsrvd: fixed memory leak on shutdown if remote descriptors
   were still connected

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-20 00:50:38 +01:00
Toni Uhlig
f713ec702b Added nDPId semantic validation test.
* fixed inconsistent processing of remaining flows during nDPId shutdown phase
 * fixed multiple `detected' flow events
   (instead only `detection-update' flow events can occur after a `detected' flow event)
 * fixed nDPIsrvd.py invalid message buffer handling
 * improved run_tests.sh so only valid pcap capture files are getting processed
   (and some more cosmetics + logging)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-04-14 22:04:42 +02:00
Toni Uhlig
131cf5385b Added IPv6 support for -I / -E.
* added another Python search path and try-catch ModuleNotFoundError again
 * run_tests.sh checks for OpenBSD netcat (required for -q)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-04-11 00:16:52 +02:00
Toni Uhlig
4e583cd4de Added JSON schema validation to run_tests.sh
* Python3 scripts are now compatible with versions <3.6
 * improved and prettified run_tests.sh

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-04-09 14:12:48 +02:00
Toni Uhlig
0a7ad7a76a nDPId-test: added JSON distribution + JSON parsing (Multithreaded design re-using most of nDPId/nDPIsrvd core)
* improved Makefile.old install targets
 * splitted nDPIsrvd_parse into nDPIsrvd_parse_line and nDPIsrvd_parse_all for the sake of readability
 * minor Python script improvments (check for nDPIsrvd.py on multiple locations, may be superseeded by setuptools in the future)
 * some paths needs to be absolute (chdir() during daemonize) and therefor additional checks introduced
 * test run script checks and fails if certain files are are missing (PCAP file <=> result output file)
 * removed not very useful "internal format error" JSON serialization if a BUG for same exists
 * fixed invalid l4 type statistics counters for nDPIsrvd-collectd

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-04-09 00:18:35 +02:00
Toni Uhlig
0b5b177c14 Extended nDPIsrvd.h with address parsing.
* nDPId supports looading of custom nDPI protocol/category files
 * extended JSON schemas according to nDPI / nDPId JSON serializing
 * removed memory holes in nDPId
 * extended examples/c-captured

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-23 14:46:47 +01:00
Toni Uhlig
a1805eb891 Added JSON schema files and a Python schema validator.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-17 14:00:10 +01:00