github-personal/nDPId
1
0
Fork 0
mirror of https://github.com/outbackdingo/nDPId.git synced 2026-01-27 18:19:39 +00:00
Code Issues Packages Projects Releases Wiki Activity
688 Commits 7 Branches 11 Tags
1.6rc4
Commit Graph

97 Commits

Author SHA1 Message Date
Toni Uhlig
8ebaccc27d py-flow-info: Improved analyse result printing. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-11-09 23:44:35 +01:00
Toni Uhlig
dcb595e161 bump libnDPI to b08c787fe267053afdea82701071f3878c09244b 
* fix ndpi data anylsis struct min/max issue
 * py-flow-info cosmetics in printing some information

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-11-09 19:52:36 +01:00
Toni Uhlig
b667f9e1da Forcefully reset NDPI_UNIDIRECTIONAL_TRAFFIC if classification was done after the first packet. Nonsense. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-11-08 17:07:20 +01:00
Toni Uhlig
d80ea84d2e Reset Unidirectional Traffc risk if packets from both directions processed. 
* Fixed risk hash value calculation, which was only done lower 32 bits.
 * Reduced default reader threads count to two if cross compiling.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-11-08 01:27:42 +01:00
Toni Uhlig
949fc0c35e bump libnDPI to 0db12b1390b1cc554b927230c76b05264c05b498 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-11-07 00:35:42 +01:00
Toni Uhlig
5fb706e9a6 Set timeout for nDPId-test run's. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-11-03 16:38:58 +01:00
Toni Uhlig
e124f2d660 Switched to UNIX socket use for tests. 
* use `ss` to make sure that the socket is not available anymore after every single test

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-10-20 00:25:59 +02:00
Toni Uhlig
e42e3fe406 Serialize nDPId / libnDPI versions within daemon events. 
* changed nDPI version hints / requirements

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-09-10 19:32:08 +02:00
Toni Uhlig
dfb8d3379f bump libnDPI to 1f693c3f5a5dcd9d69dffb610b9a81bd33f95382 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-08-27 21:45:14 +02:00
Toni Uhlig
cc60e819e8 Fixed invalid base64 encoding in some rare cases. 
* nDPId-test may also verify the correct encoding/decoding

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-08-24 22:51:57 +02:00
Toni Uhlig
ebb439d959 Tiny improvments. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-08-02 21:00:39 +02:00
Toni Uhlig
bc0a5782cc bump libnDPI to 2b230e28e0612e8654ad617534deb9aaaabd51b7 
* fixes loading of gambling lists which increased nDPId's memory usage *a lot*
 * nDPId: handle EINTR correctly

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-07-23 18:43:58 +02:00
Toni Uhlig
c8ec505b9c bump libnDPI to 8ea0eaa0d0c4a3be05f67ef7fa1d22c2579cf7d1 
* added build fix for Gitlab CI
 * added friendly C11 check
 * set required libnDPI versionto 4.7
   (ArchLinux ndpi-git sets version to 4.7, which is not released yet)
 * reduced sklearn-random-forest memory consumption by adjusting min. sample leaf

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-06-12 19:05:33 +02:00
lns
5a9b40779d bump libnDPI to 04f5c5196e790db8b8cc39e42c8645fb7f3dd141 
* added custom nDPI logging callback

Signed-off-by: lns <matzeton@googlemail.com>
 2023-05-30 09:30:24 +02:00
lns
8a936a5072 Fixed integer overflow for tcp timeout (>INT_MAX). 
Signed-off-by: lns <matzeton@googlemail.com>
 2023-05-26 11:17:38 +02:00
Toni Uhlig
c9514136b7 bump libnDPI to ... 
* upstream changed regression test interface, needed to adapt
 * improved libnDPI helper build script
 * updated JSON schema

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-05-24 19:30:19 +02:00
Toni Uhlig
99accd03a2 Moved datalink json key/value from error to packet events and renamed it to pkt_datalink. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-04-21 16:48:40 +02:00
Toni Uhlig
a3d20c17d1 Improved collectd risk processing to be in sync with libnDPI risks. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-01-11 06:28:10 +01:00
Toni Uhlig
464450486b bump libnDPI to a944514ddec73f79704f55aab1423e39f4ce7a03 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-01-09 11:24:40 +01:00
Toni Uhlig
655393e953 nDPid: Fixed base64encode bug which lead to invalid base64 strings. 
* py-semantic-validation: Decode base64 raw packet data as well
 * nDPIsrvd.py: Added PACKETS_PLEN_MAX
 * nDPIsrvd.py: Improved JSON parse error/exception handling

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-01-09 01:43:24 +01:00
Toni Uhlig
c5930e3510 Add collectd statistics diff test. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-12-06 19:51:53 +01:00
Toni Uhlig
d21a38cf02 Limit the size of base64 serialized raw packet data (8192 bytes per packet). 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-12-06 12:52:52 +01:00
Toni Uhlig
60741d5649 Strace support for diff tests. 
* tiny README update

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-12-03 01:21:20 +01:00
Toni Uhlig
57c5d8532b Test for diff's in flow-analyse CSV generator daemon. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-16 23:06:37 +01:00
Toni Uhlig
ce567ae5b7 Improved the point of time when to append the raw packet base64 data to the serializer. 
* nDPId-test: Increased the max-packets-per-flow-to-send from 3 to 5.
   This is quite useful for TCP as the first 3 packets are usually part of the three-way-handshake.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-15 06:25:16 +01:00
Toni Uhlig
36e428fc89 Sync unit tests. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-13 16:19:07 +01:00
Toni Uhlig
d55e397929 bump libnDPI to db9f6ec1b4018164e5bff05f115dc60711bb711b 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-31 13:53:02 +01:00
Toni Uhlig
49352698a0 nDPId: Added error event threshold to prevent event spamming which may be abused. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-17 06:36:30 +02:00
Toni Uhlig
4654faf381 Improved py-machine-learning example. 
* c-analysed: fixed quoting bug
 * nDPId: fixed invalid iat storing/serialisation
 * nDPId: free data analysis after event was sent

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
 2022-10-09 18:31:45 +02:00
lns
49ea4f8474 Small fixes. 
Signed-off-by: lns <matzeton@googlemail.com>
 2022-10-01 22:37:25 +02:00
Toni Uhlig
14f6b87551 Added nDPIsrvd-analysed to generate CSV files from analyse events. 
* nDPIsrvd.h: iterate over JSON arrays
 * nDPId: calculate l3 payload packet entropies for analysis

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-30 19:28:49 +02:00
Toni Uhlig
2103ee0811 Refactored client distributor C API. 
* Still not perfect, but the code before was not even able to deal with JSON arrays.
   Use common "speaking" function names for all functions in nDPIsrvd.h
 * Provide a more or less generic and easy extendable JSON walk function.
 * Modified C examples to align with the changed C API.
 * c-collectd: Reduced lot's of code duplication by providing mapping tables.
 * nDPId: IAT array requires one slot less (first packet has always an IAT of 0).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-25 00:54:39 +02:00
Toni Uhlig
36f1786bde nDPIsrvd.h: Fixed bug during token parsing/hashing. Do not hash array contents. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-23 00:13:19 +02:00
Toni Uhlig
9a28475bba Improved flown analyse event: 
* store packet directions
 * merged direction based IATs
 * merged direction based PKTLENs

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-22 19:07:08 +02:00
Toni Uhlig
28971cd764 flow-info.py: Command line arguments --no-color, --no-statusbar (both useful for tests/CI) and --print-analyse-results. 
* run_tests.sh: Use flow-info.py for additional DIFF tests.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-22 08:00:21 +02:00
Toni Uhlig
08f263e409 nDPId: Reduced flow-updates for TCP flows to 1/4 of the timeout value. 
* nDPId: Fixed broken validation tests.
 * nDPId: Removed TICK_RESOLUTION, not required anymore.
 * c-collectd: Improved total layer4 payload calculation/update handling.
 * c-collectd: Updated RRD Graph script according to total layer4 payload changes.
 * py-flow-info.py: Fixed several bugs and syntax errors.
 * Python scripts: Added dirname(argv[0]) as search path for nDPIsrvd.py.
 * nDPIsrvd&nDPId-test: Fixed missing EPOLLERR check.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-19 19:39:49 +02:00
Toni Uhlig
015a739efd Added layer4 payload length bins. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-19 10:14:37 +02:00
Toni Uhlig
31715295d9 bump libnDPI to 174cd739dbb1358ab012c4779e42e0221bef835c 
* ReadMe stuff
 * OpenWrt Makefile: set NEED_LINKING_AGAINST_LIBM=ON

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-18 13:34:43 +02:00
lns
b8cfe1d6d3 Fixed last pkt time. 
Signed-off-by: lns <matzeton@googlemail.com>
 2022-09-14 11:22:41 +02:00
Toni Uhlig
d4633c1192 New flow event: 'analysis'. 
* The goal was to provide a separate event for extracted feature that are not required
   and only useful for a few (e.g. someone who wants do ML).
 * Increased network buffer size to 32kB (8192 * 4).
 * Switched timestamp precision from ms to us for *ALL* timestamps.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-13 22:05:08 +02:00
Toni Uhlig
f9bd7d29ce Bump libnDPI to 37f918322c0a489b5143a987c8f1a44a6f78a6f3 and updated flow json schema file. 
* export env vars AR / CMAKE_C_COMPILER_AR and RANLIB / CMAKE_C_COMPILER_RANLIB while building libnDPI
 * nDPId check API version during startup (macro vs. function call) and print a warning if they are different

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-06 14:50:46 +02:00
Toni Uhlig
0fd59f060e Split *_l4_payload_len' into *_src_l4_payload_len' and `*_dst_l4_payload_len'. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-08-15 22:55:19 +02:00
lns
905545487d Split flow_packets_processed' into flow_src_packets_processed' and `flow_dst_packets_processed'. 
* no use for `flow_avg_l4_payload_len' -> removed
 * test/run_tests.sh does not fail if git-worktree's are used

Signed-off-by: lns <matzeton@googlemail.com>
 2022-08-15 18:36:49 +02:00
Toni Uhlig
a606586a32 bump libnDPI to 7c19de49047a5731f3107ff17854e9afe839cc61 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-07-14 03:48:06 +02:00
Toni Uhlig
b5d4da8793 bump libnDPI to 8f6a006e36eef0ae386f7e663d3ebecfad6a2dc9 
* try to use same wording wherever possible e.g.
   renamed workflow->total_l4_data_len to workflow->total_l4_payload_len

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-07-01 13:50:53 +02:00
Toni Uhlig
a80b6d7271 bump libnDPI to c287eb835b537ce64d9293a52ca13e670b6d3b0d 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-06-19 23:29:26 +02:00
lns
3caf7727fd bump libnDPI to 0b3f8ed849cdf9971224c49a3958f0904a2bbbb5 
* README/nDPId: fixed typ0

Signed-off-by: lns <matzeton@googlemail.com>
 2022-06-06 00:34:13 +02:00
lns
4fde63b5c2 Small fixes. 
Signed-off-by: lns <matzeton@googlemail.com>
 2022-04-26 11:44:31 +02:00
lns
a46fc4153d nDPId: Merged nDPId_flow_(info|finished) into nDPId_flow 
* nDPIsrvd: Fixed buffer allocation error due to missing memset() on disconnect
 * nDPIsrvd: Removed unused struct members

Signed-off-by: lns <matzeton@googlemail.com>
 2022-04-24 23:49:57 +02:00
lns
22a8d04c74 Added proper DLT_RAW dissection for IPv4 and IPv6. 
* nDPId: Improved TCP timeout handling if FIN/RST seen
	  which caused Midstream TCP flows when there shouldn't be any.
 * nDPIsrvd: Unified remote descriptor resource cleanup on disconnects/shutdown.
 * nDPIsrvd: Added additional error messages for remote descriptors.
 * py-flow-info: Better daemon status message printing.

Signed-off-by: lns <matzeton@googlemail.com>
 2022-04-24 15:42:28 +02:00