github-personal/nDPId
1
0
Fork 0
mirror of https://github.com/outbackdingo/nDPId.git synced 2026-01-27 18:19:39 +00:00
Code Issues Packages Projects Releases Wiki Activity
655 Commits 7 Branches 11 Tags
e42e3fe406c30bec09fe3be0c6bacd1efac44950
Commit Graph

176 Commits

Author SHA1 Message Date
Toni Uhlig
a7bd3570b0 Enable custom JSON filter expressions for Python scripts. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-08-27 20:08:01 +02:00
Toni Uhlig
b01498f011 Fix some GCC-12 warnings. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-08-27 12:49:39 +02:00
Toni Uhlig
5234f4621b keras-autoencoder.py: TensorBoard, SGD optimizer, KLDivergence loss function, EarlyStopping 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-08-23 22:56:59 +02:00
Toni Uhlig
86ac09a8db keras-autoencoder.py: Improved Model 
* added initial learning rate for Adam
 * plot some metrics using pyplot

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-08-20 23:05:08 +02:00
Toni Uhlig
4b3031245d keras-autoencoder.py: fixed invalid preprocessing of received base64 packet data 
* split logic into seperate jobs; nDPIsrvd and Keras
 * nDPIsrvd: break event processing and re-run `epoll_wait()` after client disconnected

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-08-15 11:21:46 +02:00
Toni Uhlig
2b881d56e7 c-captured extension 
* capture packets after error event occurred
 * add "logging" and "capture" mode

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-08-14 01:01:26 +02:00
Toni Uhlig
7b15838696 Added docker build&push to the CI. 
* update some git submodules

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-08-03 10:00:50 +02:00
Toni Uhlig
ebb439d959 Tiny improvments. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-08-02 21:00:39 +02:00
Toni Uhlig
4b923bdf44 py-flow-info: print flow src/dst packets 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-07-28 13:16:29 +02:00
Toni Uhlig
ba8236c1f7 py-flow-info: print flow src/dst bytes/packets 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-07-28 11:24:09 +02:00
Toni Uhlig
8a8de12fb3 Keras AE supports loading/saving models. 
* added training/batch size as cmdargs

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-07-22 09:25:11 +02:00
Toni Uhlig
425617abdf Added GLFW/OpenGL stats drawer written in C++. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-07-16 22:19:20 +02:00
Toni Uhlig
92b3c76446 Added Keras based Autoencode (Work-in-Progress!) 
* minor fixes

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-07-16 22:06:36 +02:00
Toni Uhlig
c8ec505b9c bump libnDPI to 8ea0eaa0d0c4a3be05f67ef7fa1d22c2579cf7d1 
* added build fix for Gitlab CI
 * added friendly C11 check
 * set required libnDPI versionto 4.7
   (ArchLinux ndpi-git sets version to 4.7, which is not released yet)
 * reduced sklearn-random-forest memory consumption by adjusting min. sample leaf

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-06-12 19:05:33 +02:00
Toni Uhlig
ca355b1fdb Updated js-rt-analyzer and js-rt-analyzer-frontend examples. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-04-21 16:51:58 +02:00
Toni Uhlig
a8d46ef343 Merge branch 'main' of github.com:utoni/nDPId 2023-02-27 02:02:12 +01:00
Toni Uhlig
0a959993bc Improved: 
* Gitlab-CI: build nDPId executable from CLI
 * C-Simple: log affected JSON line on READ/PARSE error
 * Sklearn: quality of life changes

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-02-27 01:20:23 +01:00
dependabot[bot]
595bd5c5e3 Bump werkzeug from 2.0 to 2.2.3 in /examples/py-flow-dashboard 
Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.0 to 2.2.3.
- [Release notes](https://github.com/pallets/werkzeug/releases)
- [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/werkzeug/compare/2.0.0...2.2.3)

---
updated-dependencies:
- dependency-name: werkzeug
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-02-16 17:43:31 +01:00
Toni Uhlig
4236aafa0d py-machine-learning: Print CSV line numbers for invalid lines (SKLearn Random Forest Classificator). 
* c-analysed: fix wrong length check

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-01-30 22:36:17 +01:00
Toni Uhlig
23816f1403 Revert "Revert "Minor fixes."" 
This reverts commit 42aad33ec8.
 2023-01-27 12:48:20 +01:00
Toni Uhlig
42aad33ec8 Revert "Minor fixes." 
This reverts commit 58439a6761.
 2023-01-27 02:02:16 +01:00
Toni Uhlig
c71284291e updated js-rt-analyzer* 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-01-27 01:47:49 +01:00
Toni Uhlig
58439a6761 Minor fixes. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-01-27 01:39:03 +01:00
Toni Uhlig
5e313f43f9 Small CI/CD/nDPIsrvd.py improvements. 
* Updated examples/js-rt-analyzer and examples/js-rt-analyzer-frontend

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-01-17 21:01:47 +01:00
Toni Uhlig
a3d20c17d1 Improved collectd risk processing to be in sync with libnDPI risks. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-01-11 06:28:10 +01:00
Toni Uhlig
470ed99eaf Added https://gitlab.com/verzulli/ndpid-rt-analyzer-frontend.git example. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-01-11 06:13:51 +01:00
Toni Uhlig
655393e953 nDPid: Fixed base64encode bug which lead to invalid base64 strings. 
* py-semantic-validation: Decode base64 raw packet data as well
 * nDPIsrvd.py: Added PACKETS_PLEN_MAX
 * nDPIsrvd.py: Improved JSON parse error/exception handling

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-01-09 01:43:24 +01:00
Toni Uhlig
4e19ab929c py-machine-learning / sklearn-random-forest: Quality Of Life improvments 
* fixed libnDPI submodule build on some platforms

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-12-22 22:13:08 +01:00
Toni Uhlig
c5930e3510 Add collectd statistics diff test. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-12-06 19:51:53 +01:00
Toni Uhlig
ced5f5d4b4 py-flow-info: ignore certain json lines that match various criteria 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-12-03 01:23:26 +01:00
Toni Uhlig
9b78939096 Updated README's. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-20 22:25:18 +01:00
Toni Uhlig
57c5d8532b Test for diff's in flow-analyse CSV generator daemon. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-16 23:06:37 +01:00
Toni Uhlig
36e428fc89 Sync unit tests. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-13 16:19:07 +01:00
Toni Uhlig
ea1698504c nDPIsrvd: Provide workaround for change user/group. 
* nDPId/nDPIsrvd/c-examples: Parameter parsing needs to be improved
                              if `strdup()` in combination with static strings is used.
 * Other non-critical fixes.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-13 11:21:55 +01:00
Toni Uhlig
e629dd59cd nDPIsrvd.h: Provide two additional convenient API functions. 
* nDPIsrvd_json_buffer_string
 * nDPIsrvd_json_buffer_length

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-06 13:19:29 +01:00
Toni Uhlig
25f4ef74ac Improved examples. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-11-02 00:01:57 +01:00
Toni Uhlig
d55e397929 bump libnDPI to db9f6ec1b4018164e5bff05f115dc60711bb711b 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-31 13:53:02 +01:00
Toni Uhlig
c63cbec26d Improved nDPIsrvd-collectd statistics. 
* Improved RRD-Graph generation script and static WWW html files.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-31 12:45:15 +01:00
Toni Uhlig
805aef5de8 Increased network buffer size to 33792 bytes. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-30 22:13:07 +01:00
Toni Uhlig
2d14509f04 nDPid-test: add buffer test 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-30 20:12:17 +01:00
Toni Uhlig
49352698a0 nDPId: Added error event threshold to prevent event spamming which may be abused. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-17 06:36:30 +02:00
Toni Uhlig
6292102f93 py-machine-learning: load and save trained models 
* added link to a pre-trained model

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-15 11:59:39 +02:00
Toni Uhlig
80f8448834 Removed discontinued examples from the ReadMe. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-13 16:47:03 +02:00
Toni Uhlig
9bf4f31418 Removed example py-ja3-checker. 
* renamed sklearn-ml.py to sklearn-random-forest.py (there is more to come!)
 * force all protocol classes to lower case

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-13 00:12:22 +02:00
Toni Uhlig
4069816d69 Improved py-machine-learning example. 
* colorize/prettify output
 * added sklearn controls/tuning options
 * disable IAT/Packet-Length features as default

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-11 20:20:01 +02:00
Toni Uhlig
20fc74f527 Improved py-machine-learning example. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-10 16:44:12 +02:00
Toni Uhlig
4654faf381 Improved py-machine-learning example. 
* c-analysed: fixed quoting bug
 * nDPId: fixed invalid iat storing/serialisation
 * nDPId: free data analysis after event was sent

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
 2022-10-09 18:31:45 +02:00
Toni Uhlig
b7a17d62c7 Improved OpenWrt UCI/Initscript 
* c-analysed: chuser()/chgroup()

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-06 06:54:01 +02:00
lns
49ea4f8474 Small fixes. 
Signed-off-by: lns <matzeton@googlemail.com>
 2022-10-01 22:37:25 +02:00
Toni Uhlig
b6060b897e c-analysed: improved feature extraction from "analyse" events 
* c-captured: update detected risks on "detection-update" events
 * c-collectd: added missing flow breed
 * c-collectd: PUTVAL macros are more flexible now

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2022-10-01 18:01:56 +02:00