scottk/nDPId
1
0
Fork 0
mirror of https://github.com/optim-enterprises-bv/nDPId.git synced 2025-11-01 18:57:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
567 Commits 1 Branch 0 Tags
c63cbec26d9ab3aa3004c9969ec67315fecd28b1
Commit Graph

139 Commits

Author SHA1 Message Date
Toni Uhlig
c63cbec26d Improved nDPIsrvd-collectd statistics. 
* Improved RRD-Graph generation script and static WWW html files.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-31 12:45:15 +01:00
Toni Uhlig
805aef5de8 Increased network buffer size to 33792 bytes. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-30 22:13:07 +01:00
Toni Uhlig
2d14509f04 nDPid-test: add buffer test 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-30 20:12:17 +01:00
Toni Uhlig
49352698a0 nDPId: Added error event threshold to prevent event spamming which may be abused. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-17 06:36:30 +02:00
Toni Uhlig
6292102f93 py-machine-learning: load and save trained models 
* added link to a pre-trained model

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-15 11:59:39 +02:00
Toni Uhlig
80f8448834 Removed discontinued examples from the ReadMe. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-13 16:47:03 +02:00
Toni Uhlig
9bf4f31418 Removed example py-ja3-checker. 
* renamed sklearn-ml.py to sklearn-random-forest.py (there is more to come!)
 * force all protocol classes to lower case

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-13 00:12:22 +02:00
Toni Uhlig
4069816d69 Improved py-machine-learning example. 
* colorize/prettify output
 * added sklearn controls/tuning options
 * disable IAT/Packet-Length features as default

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-11 20:20:01 +02:00
Toni Uhlig
20fc74f527 Improved py-machine-learning example. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-10 16:44:12 +02:00
Toni Uhlig
4654faf381 Improved py-machine-learning example. 
* c-analysed: fixed quoting bug
 * nDPId: fixed invalid iat storing/serialisation
 * nDPId: free data analysis after event was sent

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
 2022-10-09 18:31:45 +02:00
Toni Uhlig
b7a17d62c7 Improved OpenWrt UCI/Initscript 
* c-analysed: chuser()/chgroup()

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-06 06:54:01 +02:00
lns
49ea4f8474 Small fixes. 
Signed-off-by: lns <matzeton@googlemail.com>
 2022-10-01 22:37:25 +02:00
Toni Uhlig
b6060b897e c-analysed: improved feature extraction from "analyse" events 
* c-captured: update detected risks on "detection-update" events
 * c-collectd: added missing flow breed
 * c-collectd: PUTVAL macros are more flexible now

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-01 18:01:56 +02:00
Toni Uhlig
14f6b87551 Added nDPIsrvd-analysed to generate CSV files from analyse events. 
* nDPIsrvd.h: iterate over JSON arrays
 * nDPId: calculate l3 payload packet entropies for analysis

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-30 19:28:49 +02:00
Toni Uhlig
74f71643da nDPId-test: Force collector blocking mode. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-25 16:24:05 +02:00
Toni Uhlig
2103ee0811 Refactored client distributor C API. 
* Still not perfect, but the code before was not even able to deal with JSON arrays.
   Use common "speaking" function names for all functions in nDPIsrvd.h
 * Provide a more or less generic and easy extendable JSON walk function.
 * Modified C examples to align with the changed C API.
 * c-collectd: Reduced lot's of code duplication by providing mapping tables.
 * nDPId: IAT array requires one slot less (first packet has always an IAT of 0).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-25 00:54:39 +02:00
Toni Uhlig
36f1786bde nDPIsrvd.h: Fixed bug during token parsing/hashing. Do not hash array contents. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-23 00:13:19 +02:00
Toni Uhlig
9a28475bba Improved flown analyse event: 
* store packet directions
 * merged direction based IATs
 * merged direction based PKTLENs

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-22 19:07:08 +02:00
Toni Uhlig
28971cd764 flow-info.py: Command line arguments --no-color, --no-statusbar (both useful for tests/CI) and --print-analyse-results. 
* run_tests.sh: Use flow-info.py for additional DIFF tests.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-22 08:00:21 +02:00
Toni Uhlig
08f263e409 nDPId: Reduced flow-updates for TCP flows to 1/4 of the timeout value. 
* nDPId: Fixed broken validation tests.
 * nDPId: Removed TICK_RESOLUTION, not required anymore.
 * c-collectd: Improved total layer4 payload calculation/update handling.
 * c-collectd: Updated RRD Graph script according to total layer4 payload changes.
 * py-flow-info.py: Fixed several bugs and syntax errors.
 * Python scripts: Added dirname(argv[0]) as search path for nDPIsrvd.py.
 * nDPIsrvd&nDPId-test: Fixed missing EPOLLERR check.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-19 19:39:49 +02:00
Toni Uhlig
d4633c1192 New flow event: 'analysis'. 
* The goal was to provide a separate event for extracted feature that are not required
   and only useful for a few (e.g. someone who wants do ML).
 * Increased network buffer size to 32kB (8192 * 4).
 * Switched timestamp precision from ms to us for *ALL* timestamps.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-13 22:05:08 +02:00
Toni Uhlig
94aa02b298 nDPIsrvd-collectd: Stdout should be unbuffered. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-12 13:32:50 +02:00
Toni Uhlig
20ced3e636 nDPIsrvd-collectd: RRD Graph generation script and a basic static HTML5 website for viewing the generated image files. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-12 13:23:50 +02:00
Toni Uhlig
c5c7d83c97 Added https://gitlab.com/verzulli/ndpid-rt-analyzer.git to examples. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-06 11:08:01 +02:00
Toni Uhlig
dcf78ad3ed Disable timestamp generation in nDPIsrvd-collectd as default. 
* collectd's rrdtool write plugin does silently fail with those ones (dunno why)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-09-04 17:24:10 +02:00
Toni Uhlig
0fd59f060e Split *_l4_payload_len' into *_src_l4_payload_len' and `*_dst_l4_payload_len'. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-08-15 22:55:19 +02:00
Toni Uhlig
2cb2c86cb5 c-collectd: fixed incorrect PUTVAL 
* get rid of types.db

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-08-15 16:42:59 +02:00
Damiano Verzulli
96b9129918 Improve README 
- slightly improve the README to better enhance the streaming
  capability of `nDPId`, regardless of `nDPIsrvd`
- add a screencast showing the install step and the
  `nDPId` usage, alone (with ncat as unix-socket listener)
- add "build" to .gitignore
 2022-08-12 11:10:45 +02:00
Toni Uhlig
fc442180da c-collectd: fixed possible undefined behavior 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-07-14 03:48:57 +02:00
Toni Uhlig
28602ca095 README update 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-07-12 16:03:51 +02:00
lns
cdaeb1632e py-flow-dashboard: Improved graph axis scaling. 
Signed-off-by: lns <matzeton@googlemail.com>
 2022-06-16 11:37:33 +02:00
lns
2a8883a96e CMake: do not add /usr/include/ndpi to include dirs if BUILD_NDPI or STATIC_LIBNDPI_INSTALLDIR used. 
* c-collectd: fixed memory leak on failure
 * py-flow-info.py: fancy spinners and stats counting improved

Signed-off-by: lns <matzeton@googlemail.com>
 2022-06-10 14:34:30 +02:00
Toni Uhlig
664a8a077d Merge branch 'master' of github.com:lnslbrty/nDPId 2022-06-07 18:01:40 +02:00
Toni Uhlig
77a87254b6 nDPIsrvd.py: Throw SocketTimeout Exception to catch both timeout exceptions different Python versions can throw. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-06-07 17:59:47 +02:00
lns
f5b0021413 README update 
Signed-off-by: lns <matzeton@googlemail.com>
 2022-05-31 23:57:22 +02:00
lns
4fde63b5c2 Small fixes. 
Signed-off-by: lns <matzeton@googlemail.com>
 2022-04-26 11:44:31 +02:00
lns
22a8d04c74 Added proper DLT_RAW dissection for IPv4 and IPv6. 
* nDPId: Improved TCP timeout handling if FIN/RST seen
	  which caused Midstream TCP flows when there shouldn't be any.
 * nDPIsrvd: Unified remote descriptor resource cleanup on disconnects/shutdown.
 * nDPIsrvd: Added additional error messages for remote descriptors.
 * py-flow-info: Better daemon status message printing.

Signed-off-by: lns <matzeton@googlemail.com>
 2022-04-24 15:42:28 +02:00
lns
c7bf94e9f1 nDPIsrvd.(h|py): Added socket read/recv timeout. 
* nDPIsrvd.h: support for O_NONBLOCK nDPIsrvd_socket

Signed-off-by: lns <matzeton@googlemail.com>
 2022-04-17 18:56:30 +02:00
Toni Uhlig
65a9e5a18d Executing ./tests/run_tests.sh w/o zLib should not result in diff's anymore. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-03-24 01:04:49 +01:00
Toni Uhlig
c0b7bdacbc Reworked nDPIsrvd.h C-API. 
* nDPIsrvd.h: Provide nDPId thread storage.
 * nDPIsrvd.py: Fixed instance cleanup bug.
 * nDPIsrvd.h: Support for instance/thread user data and cleanup callback.
 * nDPIsrvd.h: Most recent flow time stored in thread ht instead of instance ht.
 * nDPId: Moved flow logger out the memory profilier into SIGUSR1 signal handling.
 * nDPId: Added signal fd to be usable within epoll's event handling (live-capture only!)
 * nDPId: Added information about ZLib compressions to daemon status/shutdown events.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-03-21 15:56:01 +01:00
Toni Uhlig
daaaa61519 Renamed basic event to error event for the sake of the logic. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-03-13 03:08:44 +01:00
Toni Uhlig
ed1647b944 Disconnect nDPIsrvd clients immediately instead waiting for a failed write(). 
* nDPIsrvd: Collector/Distributor logging improved
 * nDPIsrvd: Command line option for max remote descriptors
 * nDPId: Stop spamming nDPIsrvd Collector with the same events over and over again
 * nDPId: Refactored some variable names and events

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-03-13 02:28:10 +01:00
Toni Uhlig
6f1f9e65ea Fixed some pyhton issues with static class members. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-03-08 14:17:24 +01:00
Toni Uhlig
e09dd8509f Updated examples/README.md 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-03-06 17:41:38 +01:00
Toni Uhlig
29c72fb30b Removed go-dashboard example. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-03-06 17:40:35 +01:00
Toni Uhlig
46f68501d5 Added daemon event: DAEMON_EVENT_STATUS (periodically send's daemon statistics.) 
* Improved distributor timeout handling (per-thread).
 * flow-info.py / flow-dash.py: Distinguish between flow risk severities.
 * nDPId: Skip tag switch datalink packet dissection / processing.
 * nDPId: Fixed incorrect value for current active flows.
 * Improved JSON schema's.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-03-06 17:38:05 +01:00
Toni Uhlig
9db048c9d9 Serialize flow risk score / confidence. 
* bump libnDPI to 8b062295cc76a60e3905c054ce37bd17669464d1
 * removed ndpi_id_struct's

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-02-27 02:53:39 +01:00
Toni Uhlig
cb80c415d8 Improved py-flow-info to provide more optional information about received timestamps. 
* py-flow-dashboard: Added color mapping for PieCharts/Graph that make more sense
 * nDPId: Renamed `flow_type' to a more precisely `flow_state'
 * nDPId: Changed the default setting to process only as much packets as libnDPI does

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-02-05 15:27:13 +01:00
Toni Uhlig
6fd6dff14d Added additional (minimalistic) detection information to flow updates. 
This will only affect flows with the state `FT_FINISHED' (detection done).

 * nDPIsrvd.py: force use of JSON schema Draft 7 validator
 * flow-dash.py: gather/use total processed layer4 payload size
 * flow-info.py: added additional event filter
 * flow-info.py: prettified flow events printing whose detection is in progress
 * py-semantic-validation.py: added validation checks for FT_FINISHED
 * updated flow event JSON schema

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-02-04 01:12:18 +01:00
Toni Uhlig
7022d0b1c5 nDPIsrvd: Fixed memory leak caused be not clearing buffer cache after a client disconnected. 
* README.md: Fixed a typ0 and added a meh image from examples/py-flow-dashboard/flow-dash.py

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-01-26 15:38:43 +01:00