scottk/nDPId
1
0
Fork 0
mirror of https://github.com/optim-enterprises-bv/nDPId.git synced 2025-10-29 17:32:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
784 Commits 1 Branch 0 Tags
ce9752af1664f11fb6af4d58c85471978e693524
Commit Graph

784 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Toni Uhlig
8b81b170d3 Updated Github/Gitlab CI 
* instrument Clang's thread sanitizer for tests

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-12-02 23:24:06 +01:00
Toni Uhlig
2c95b31210 nDPId-test: Reworked I/O handling to prevent some endless loop scenarios. Fixed a race condition in the memory wrapper as well. 
* nDPId: Instead of sending too long JSON strings, log an error and some parts.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-12-02 22:11:57 +01:00
Toni Uhlig
532961af33 Fixed MD format issues. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-21 11:34:10 +01:00
Toni Uhlig
64f6abfdbe Unified nDPId/nDPIsrvd command line argument storage. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-21 11:26:05 +01:00
Toni Uhlig
77ee336cc9 Added Network Buffer Size CI Check. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-20 22:42:06 +01:00
Toni Uhlig
9b78939096 Updated README's. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-20 22:25:18 +01:00
Toni Uhlig
57c5d8532b Test for diff's in flow-analyse CSV generator daemon. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-16 23:06:37 +01:00
Toni Uhlig
869d4de271 Improved make daemon / daemon.sh to accept nDPId / nDPIsrvd arguments via env. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-15 07:05:29 +01:00
Toni Uhlig
ce567ae5b7 Improved the point of time when to append the raw packet base64 data to the serializer. 
* nDPId-test: Increased the max-packets-per-flow-to-send from 3 to 5.
   This is quite useful for TCP as the first 3 packets are usually part of the three-way-handshake.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-15 06:25:16 +01:00
Toni Uhlig
36e428fc89 Sync unit tests. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-13 16:19:07 +01:00
Toni Uhlig
ea1698504c nDPIsrvd: Provide workaround for change user/group. 
* nDPId/nDPIsrvd/c-examples: Parameter parsing needs to be improved
                              if `strdup()` in combination with static strings is used.
 * Other non-critical fixes.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-13 11:21:55 +01:00
Toni Uhlig
bc346a28f4 nDPId: Fixed base64 encoding issue. 
* The issue can result in an error message like:
   `Base64 encoding failed with: Buffer too small.`
   and also in too big JSON strings generated by nDPId
   which nDPIsrvd does not like as it's length is
   greater than `NETWORK_BUFFER_MAX_SIZE`.
 * nDPId will now obey `NETWORK_BUFFER_MAX_SIZE` while
   trying to base64 encode raw packet data.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-13 09:26:04 +01:00
Toni Uhlig
e629dd59cd nDPIsrvd.h: Provide two additional convenient API functions. 
* nDPIsrvd_json_buffer_string
 * nDPIsrvd_json_buffer_length

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-06 13:19:29 +01:00
Toni Uhlig
7515c8aeec Experimental systemd support. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-06 12:58:55 +01:00
Toni Uhlig
25f4ef74ac Improved examples. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-02 00:01:57 +01:00
Toni Uhlig
d55e397929 bump libnDPI to db9f6ec1b4018164e5bff05f115dc60711bb711b 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-31 13:53:02 +01:00
Toni Uhlig
d3f99f21e6 Create pidfile iff daemon mode enabled. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-31 12:45:49 +01:00
Toni Uhlig
c63cbec26d Improved nDPIsrvd-collectd statistics. 
* Improved RRD-Graph generation script and static WWW html files.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-31 12:45:15 +01:00
Toni Uhlig
805aef5de8 Increased network buffer size to 33792 bytes. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-30 22:13:07 +01:00
Toni Uhlig
2d14509f04 nDPid-test: add buffer test 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-30 20:12:17 +01:00
Toni Uhlig
916d2df6ea nDPId-test: Fixed thread sync/lock issue. 
* rarely happens in CI

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-22 01:45:14 +02:00
Toni Uhlig
46c8fc5219 Merge branch 'main' of github.com:utoni/nDPId 2022-10-20 16:13:27 +02:00
Toni Uhlig
e5f4af4890 Special Thanks to Damiano Verzulli (@verzulli). 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-20 16:12:40 +02:00
lns
cd22d56056 Add ArchLinux PKGBUILD. 
Signed-off-by: lns <matzeton@googlemail.com>
 2022-10-19 18:40:52 +02:00
Toni Uhlig
49352698a0 nDPId: Added error event threshold to prevent event spamming which may be abused. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-17 06:36:30 +02:00
Toni Uhlig
6292102f93 py-machine-learning: load and save trained models 
* added link to a pre-trained model

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-15 11:59:39 +02:00
Toni Uhlig
80f8448834 Removed discontinued examples from the ReadMe. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-13 16:47:03 +02:00
Toni Uhlig
9bf4f31418 Removed example py-ja3-checker. 
* renamed sklearn-ml.py to sklearn-random-forest.py (there is more to come!)
 * force all protocol classes to lower case

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-13 00:12:22 +02:00
Toni Uhlig
4069816d69 Improved py-machine-learning example. 
* colorize/prettify output
 * added sklearn controls/tuning options
 * disable IAT/Packet-Length features as default

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-11 20:20:01 +02:00
Toni Uhlig
bb633bde22 daemon.sh: fixed race condition 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-10 17:54:49 +02:00
Toni Uhlig
20fc74f527 Improved py-machine-learning example. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-10 16:44:12 +02:00
Toni Uhlig
2ede930eec daemon.sh: cat nDPId / nDPIsrvd log on failure 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-09 19:11:37 +02:00
Toni Uhlig
4654faf381 Improved py-machine-learning example. 
* c-analysed: fixed quoting bug
 * nDPId: fixed invalid iat storing/serialisation
 * nDPId: free data analysis after event was sent

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
 2022-10-09 18:31:45 +02:00
Toni Uhlig
b7a17d62c7 Improved OpenWrt UCI/Initscript 
* c-analysed: chuser()/chgroup()

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-06 06:54:01 +02:00
Toni Uhlig
ac46f3841f Fixed heap overflow on shutdown caused by missing remotes size/used reset. 
* introduced with 22a8d04c74

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-05 00:14:46 +02:00
Toni Uhlig
be3f466373 OpenWrt UCI/Initscript 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-03 15:57:16 +02:00
lns
b7d8564b65 Generate code coverage w/o external shell script, use CMake. 
* upload codecov/dist artifacts

Signed-off-by: lns <matzeton@googlemail.com>
 2022-10-03 15:45:17 +02:00
lns
49ea4f8474 Small fixes. 
Signed-off-by: lns <matzeton@googlemail.com>
 2022-10-01 22:37:25 +02:00
Toni Uhlig
b6060b897e c-analysed: improved feature extraction from "analyse" events 
* c-captured: update detected risks on "detection-update" events
 * c-collectd: added missing flow breed
 * c-collectd: PUTVAL macros are more flexible now

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-01 18:01:56 +02:00
Toni Uhlig
14f6b87551 Added nDPIsrvd-analysed to generate CSV files from analyse events. 
* nDPIsrvd.h: iterate over JSON arrays
 * nDPId: calculate l3 payload packet entropies for analysis

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-30 19:28:49 +02:00
Toni Uhlig
74f71643da nDPId-test: Force collector blocking mode. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-25 16:24:05 +02:00
Toni Uhlig
2103ee0811 Refactored client distributor C API. 
* Still not perfect, but the code before was not even able to deal with JSON arrays.
   Use common "speaking" function names for all functions in nDPIsrvd.h
 * Provide a more or less generic and easy extendable JSON walk function.
 * Modified C examples to align with the changed C API.
 * c-collectd: Reduced lot's of code duplication by providing mapping tables.
 * nDPId: IAT array requires one slot less (first packet has always an IAT of 0).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-25 00:54:39 +02:00
Toni Uhlig
36f1786bde nDPIsrvd.h: Fixed bug during token parsing/hashing. Do not hash array contents. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-23 00:13:19 +02:00
Toni Uhlig
9a28475bba Improved flown analyse event: 
* store packet directions
 * merged direction based IATs
 * merged direction based PKTLENs

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-22 19:07:08 +02:00
Toni Uhlig
28971cd764 flow-info.py: Command line arguments --no-color, --no-statusbar (both useful for tests/CI) and --print-analyse-results. 
* run_tests.sh: Use flow-info.py for additional DIFF tests.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-22 08:00:21 +02:00
Toni Uhlig
3c7bd6a4ba Merge branch 'main' of github.com:utoni/nDPId 2022-09-19 19:39:54 +02:00
Toni Uhlig
08f263e409 nDPId: Reduced flow-updates for TCP flows to 1/4 of the timeout value. 
* nDPId: Fixed broken validation tests.
 * nDPId: Removed TICK_RESOLUTION, not required anymore.
 * c-collectd: Improved total layer4 payload calculation/update handling.
 * c-collectd: Updated RRD Graph script according to total layer4 payload changes.
 * py-flow-info.py: Fixed several bugs and syntax errors.
 * Python scripts: Added dirname(argv[0]) as search path for nDPIsrvd.py.
 * nDPIsrvd&nDPId-test: Fixed missing EPOLLERR check.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-19 19:39:49 +02:00
Damiano Verzulli
ab7f7d05f3 Improve README 
- link to already-existing JSON-schemas have been added
- a graphical schema detailing flow-events timeline have
  been added in both PNG and source-Drawio formats.
  Link to PNG have been included in the README
 2022-09-19 17:23:11 +02:00
Toni Uhlig
015a739efd Added layer4 payload length bins. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-19 10:14:37 +02:00
Toni Uhlig
31715295d9 bump libnDPI to 174cd739dbb1358ab012c4779e42e0221bef835c 
* ReadMe stuff
 * OpenWrt Makefile: set NEED_LINKING_AGAINST_LIBM=ON

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-18 13:34:43 +02:00