Commit Graph

31 Commits

Author SHA1 Message Date
Toni Uhlig
ed1647b944 Disconnect nDPIsrvd clients immediately instead waiting for a failed write().
* nDPIsrvd: Collector/Distributor logging improved
 * nDPIsrvd: Command line option for max remote descriptors
 * nDPId: Stop spamming nDPIsrvd Collector with the same events over and over again
 * nDPId: Refactored some variable names and events

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-13 02:28:10 +01:00
Toni Uhlig
6f1f9e65ea Fixed some pyhton issues with static class members.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-08 14:17:24 +01:00
Toni Uhlig
46f68501d5 Added daemon event: DAEMON_EVENT_STATUS (periodically send's daemon statistics.)
* Improved distributor timeout handling (per-thread).
 * flow-info.py / flow-dash.py: Distinguish between flow risk severities.
 * nDPId: Skip tag switch datalink packet dissection / processing.
 * nDPId: Fixed incorrect value for current active flows.
 * Improved JSON schema's.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-06 17:38:05 +01:00
Toni Uhlig
9db048c9d9 Serialize flow risk score / confidence.
* bump libnDPI to 8b062295cc76a60e3905c054ce37bd17669464d1
 * removed ndpi_id_struct's

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-27 02:53:39 +01:00
Toni Uhlig
cb80c415d8 Improved py-flow-info to provide more optional information about received timestamps.
* py-flow-dashboard: Added color mapping for PieCharts/Graph that make more sense
 * nDPId: Renamed `flow_type' to a more precisely `flow_state'
 * nDPId: Changed the default setting to process only as much packets as libnDPI does

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-05 15:27:13 +01:00
Toni Uhlig
6fd6dff14d Added additional (minimalistic) detection information to flow updates.
This will only affect flows with the state `FT_FINISHED' (detection done).

 * nDPIsrvd.py: force use of JSON schema Draft 7 validator
 * flow-dash.py: gather/use total processed layer4 payload size
 * flow-info.py: added additional event filter
 * flow-info.py: prettified flow events printing whose detection is in progress
 * py-semantic-validation.py: added validation checks for FT_FINISHED
 * updated flow event JSON schema

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-04 01:12:18 +01:00
Toni Uhlig
29a1b13e7a Improved Plotly/Dash example. It is now somehow informative.
* TCP timeout after FIN/RST: switched back to the value from a35fc1d5ea
 * py-flow-info: reset 'guessed' flag after detection/detection-update received

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-25 11:16:41 +01:00
Toni Uhlig
9e07a57566 Major nDPId extension. Sorry for the huge commit.
- nDPId: fixed invalid IP4/IP6 tuple compare
 - nDPIsrvd: fixed caching issue (finally)
 - added tiny c example (can be used to check flow manager sanity)
 - c-captured: use flow_last_seen timestamp from `struct nDPIsrvd_flow`
 - README.md update: added example JSON sequence
 - nDPId: added new flow event `update` necessary for correct
   timeout handling (and other future use-cases)
 - nDPIsrvd.h and nDPIsrvd.py: switched to an instance
   (consists of an alias/source tuple) based flow manager
 - every flow related event **must** now serialize `alias`, `source`,
   `flow_id`, `flow_last_seen` and `flow_idle_time` to make the timeout
   handling and verification process work correctly
 - nDPIsrvd.h: ability to profile any dynamic memory (de-)allocation
 - nDPIsrvd.py: removed PcapPacket class (unused)
 - py-flow-dashboard and py-flow-multiprocess: fixed race condition
 - py-flow-info: print statusbar with probably useful information
 - nDPId/nDPIsrvd.h: switched from packet-flow only timestamps (`pkt_*sec`)
   to a generic flow event timestamp `ts_msec`
 - nDPId-test: added additional checks
 - nDPId: increased ICMP flow timeout
 - nDPId: using event based i/o if capturing packets from a device
 - nDPIsrvd: fixed memory leak on shutdown if remote descriptors
   were still connected

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-20 00:50:38 +01:00
Toni Uhlig
93bff603d0 py-flow-info: Support for --ipwhois, --new and --detection.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-04-30 22:39:11 +02:00
Toni Uhlig
131cf5385b Added IPv6 support for -I / -E.
* added another Python search path and try-catch ModuleNotFoundError again
 * run_tests.sh checks for OpenBSD netcat (required for -q)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-04-11 00:16:52 +02:00
Toni Uhlig
4e583cd4de Added JSON schema validation to run_tests.sh
* Python3 scripts are now compatible with versions <3.6
 * improved and prettified run_tests.sh

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-04-09 14:12:48 +02:00
Toni Uhlig
0a7ad7a76a nDPId-test: added JSON distribution + JSON parsing (Multithreaded design re-using most of nDPId/nDPIsrvd core)
* improved Makefile.old install targets
 * splitted nDPIsrvd_parse into nDPIsrvd_parse_line and nDPIsrvd_parse_all for the sake of readability
 * minor Python script improvments (check for nDPIsrvd.py on multiple locations, may be superseeded by setuptools in the future)
 * some paths needs to be absolute (chdir() during daemonize) and therefor additional checks introduced
 * test run script checks and fails if certain files are are missing (PCAP file <=> result output file)
 * removed not very useful "internal format error" JSON serialization if a BUG for same exists
 * fixed invalid l4 type statistics counters for nDPIsrvd-collectd

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-04-09 00:18:35 +02:00
Toni Uhlig
4699263d65 Fixed possible NULL pointer deref in dependencies/nDPIsrvd.h and examples/c-captured/c-captured.c
* `make install` C examples
 * examples/py-flow-info/flow-info.py supports filtering
 * added breed to examples/py-flow-info/flow-info.py

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-28 11:27:22 +01:00
Toni Uhlig
0b5b177c14 Extended nDPIsrvd.h with address parsing.
* nDPId supports looading of custom nDPI protocol/category files
 * extended JSON schemas according to nDPI / nDPId JSON serializing
 * removed memory holes in nDPId
 * extended examples/c-captured

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-23 14:46:47 +01:00
Toni Uhlig
893f437051 Aligned nDPIsrvd.py to nDPIsrvd.h for consistency.
* Simplified Python interface as well.
 * c-captured and flow-undetected-to-pcap.py produce similiar results
 * Removed overloaded nDPIsrvd.py event structures.
 * flow-info.py prints (with a color-hash) additional information e.g. alias/source and midstream

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-16 20:43:00 +01:00
Toni Uhlig
102b61175c All colors are beautiful for py-flow-info.
* instance alias is now mandatory (will be required for future use-cases)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-01-26 19:07:16 +01:00
Toni Uhlig
ba179cc724 Increased serialization buffer size as required for recent nDPI builds.
* added .gitignore to prevent SCM check-ins of unwanted files
 * fixed invalid include path for python examples

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-11-18 18:12:38 +01:00
Toni Uhlig
8cbac2ac88 cosmetics, exception verbosity
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-10-17 12:42:26 +02:00
Toni Uhlig
ec9dc251d2 py-flow-info: Print BasicEvents.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-10-12 20:02:20 +02:00
Toni Uhlig
9ea078b05e Improved python event validation with focus on readability.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-09-30 20:35:28 +02:00
Toni Uhlig
2f7a052f43 Centralized some python code parts to make future modifications easier.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-09-29 18:37:24 +02:00
Toni Uhlig
04bbbcd055 Improved event parsing for Python scripts.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-09-26 15:15:15 +02:00
Toni Uhlig
8c61f88362 Added event validation functions and fixed Python scripts.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-09-26 11:19:21 +02:00
Toni Uhlig
84712686a7 Centralized EventName validation and moved code parts.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-09-26 00:49:34 +02:00
Toni Uhlig
6c0ac8b045 Added new flow event: FLOW_EVENT_DETECTION_UPDATE
* This event will be triggered when nDPI detection has some new information for us (hopefully).
 * Detection change is based on hashing with 32-bit murmur3 certain members of the ndpi flow struct.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-09-25 20:43:23 +02:00
Toni Uhlig
370ca7c00d Process extra packets with nDPI, still not perfect but results in a more accurate detection.
* set default user used for setuid()
 * added 2 TODOs

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-09-24 23:41:12 +02:00
Toni Uhlig
e6c2bc6373 merged code to reduce code duplicates
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-18 16:48:01 +02:00
Toni Uhlig
68fb4030ee flow-info.py, flow-undetected-to-pcap.py: throw socket error runtime exception if disconnect received
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-13 10:55:04 +02:00
Toni Uhlig
67407a8a39 flow-info.py: fixed buffering issue
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-08 19:41:44 +02:00
Toni Uhlig
9ae4aa7ce9 flow-info.py overhaul, terminal color support, ndpi protocol/category/flow-risk output
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-07 15:21:24 +02:00
Toni Uhlig
0c2b2a9750 added python flow info script, improved IPv4/IPv6 string conversion
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-06 16:48:45 +02:00