github-personal/nDPId
1
0
Fork 0
mirror of https://github.com/outbackdingo/nDPId.git synced 2026-01-27 18:19:39 +00:00
Code Issues Packages Projects Releases Wiki Activity
410 Commits 7 Branches 11 Tags
1.4
Commit Graph

410 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Toni Uhlig
5954e46340 Build system cleanup / cosmetics. 
* libnDPI submodule update

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
1.4 		2021-06-07 16:22:49 +02:00
Toni Uhlig
54e0601fec Unified IO buffer mgmt. 
* c-collectd gives the user control over collectd-exec instance name
 * added missing collectd type `flow_l4_icmp_count`

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-06-07 15:04:46 +02:00
Toni Uhlig
382706cd20 flow-dash: Simplified and extended bar graph. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-05-28 18:41:32 +02:00
Toni Uhlig
96dc563d91 flow-dash: Added live bars visualising midstream/risky flow count. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-05-28 02:14:23 +02:00
Toni Uhlig
12e0ae98b6 Added realtime web based graph example using Plotly/Dash. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-05-27 15:05:06 +02:00
Toni Uhlig
2a59c0513c libnDPI updated to c4084ca3c7b3657659aff624158a9c4f5710f57d 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-05-26 17:26:07 +02:00
Toni Uhlig
e3d1a8a772 Added simple Python Multiprocess example. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-05-26 17:18:20 +02:00
Toni Uhlig
4b6ead68a1 nDPIsrvd-captured: skip empty flows based on flow total payload length 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-05-20 15:40:36 +02:00
Toni Uhlig
9a1c2d0ea7 Reworked layer 4 flow length naming/calculation. 
* nDPIsrvd services usually do not care about layer4 data length,
   payload length is quite more essential for further processing

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-05-20 14:55:05 +02:00
Toni Uhlig
db39772aa7 Fixed CMake global CFLAGS misuse which can cause xcompile errors. 
nDPIsrvd-captured supports skipping flows w/o any layer 4 payload.

 * libndpi update
 * run_tests does not generate any *.out files for fuzz-*.pcap anymore and
   does not fail if nDPId-test exits with value 1 (most likely caused by a libpcap failure)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-05-19 15:56:20 +02:00
Toni Uhlig
9ffaeef24d README.md update 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-05-14 00:16:47 +02:00
Toni Uhlig
3a0fbe7433 Cosmetic fixes. 
* daemon.sh script to simplify daemon testing

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-05-13 20:08:27 +02:00
Toni Uhlig
da4942b41c Use layer4 specific flow timeouts. 
* default values "stolen" from nf_conntrack

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-05-13 15:41:24 +02:00
Toni Uhlig
182867a071 Reduced superfluous Travis-CI yaml content. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
1.3 		2021-05-12 15:17:07 +02:00
Toni Uhlig
241a7fdc4f Added missing datalink types. 
* basicially C&P from nDPI reader_utils but with some more sanity checks

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-05-12 13:48:49 +02:00
Toni Uhlig
fa079d2346 Git submodule libnDPI update. 
* enable ctest to run integration tests (**only** if BUILD_NDPI=ON)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-05-12 12:46:49 +02:00
Toni Uhlig
50f9c1bba1 OpenWrt compatible build system. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
1.2 		2021-05-11 17:51:57 +02:00
Toni Uhlig
98a6dc5d3b Added GPL-3 License. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
1.1 		2021-05-11 16:33:34 +02:00
Toni Uhlig
785603c276 Clone nDPI as submodule via CMake (if BUILD_NDPI=ON). 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
1.0 		2021-05-11 14:18:31 +02:00
Toni Uhlig
5b0a751159 README.md updated 
* removed all those outdated Makefile.old references
 * added additional CMake build instructions

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-05-10 15:38:01 +02:00
Toni Uhlig
93bff603d0 py-flow-info: Support for --ipwhois, --new and --detection. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-04-30 22:39:11 +02:00
Toni Uhlig
4e3dda70a3 Unified and improved dependency build/mgmt via CMake and travis-ci. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-04-22 17:37:55 +02:00
Toni Uhlig
174bd89d8e Added JA3 blacklist downloader/checker from abuse.ch 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-04-17 16:45:03 +02:00
Toni Uhlig
a119a72d13 Added python example to check JA3 hashes against known hashes via JA3er.com 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-04-16 13:30:24 +02:00
Toni Uhlig
a0fa598cee travis-ci build CMake project 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-04-15 11:21:37 +02:00
Toni Uhlig
0c034ffba1 added some CMake install targets 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-04-15 00:07:40 +02:00
Toni Uhlig
713cd13322 added TODO 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-04-14 22:17:39 +02:00
Toni Uhlig
f713ec702b Added nDPId semantic validation test. 
* fixed inconsistent processing of remaining flows during nDPId shutdown phase
 * fixed multiple `detected' flow events
   (instead only `detection-update' flow events can occur after a `detected' flow event)
 * fixed nDPIsrvd.py invalid message buffer handling
 * improved run_tests.sh so only valid pcap capture files are getting processed
   (and some more cosmetics + logging)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-04-14 22:04:42 +02:00
Toni Uhlig
514c427917 Fixed nDPIsrvd.h flow end bug (flow end callback could never be called and caused some memory troubles). 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-04-11 12:34:38 +02:00
Toni Uhlig
131cf5385b Added IPv6 support for -I / -E. 
* added another Python search path and try-catch ModuleNotFoundError again
 * run_tests.sh checks for OpenBSD netcat (required for -q)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-04-11 00:16:52 +02:00
Toni Uhlig
ba586e1ecf nDPId-test: mimic full nDPId lifecycle 
* generate DAEMON_EVENT_INIT as well as DAEMON_EVENT_SHUTDOWN
 * process remaining flows before shutdown (and generate events)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-04-09 14:43:28 +02:00
Toni Uhlig
4e583cd4de Added JSON schema validation to run_tests.sh 
* Python3 scripts are now compatible with versions <3.6
 * improved and prettified run_tests.sh

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-04-09 14:12:48 +02:00
Toni Uhlig
0a7ad7a76a nDPId-test: added JSON distribution + JSON parsing (Multithreaded design re-using most of nDPId/nDPIsrvd core) 
* improved Makefile.old install targets
 * splitted nDPIsrvd_parse into nDPIsrvd_parse_line and nDPIsrvd_parse_all for the sake of readability
 * minor Python script improvments (check for nDPIsrvd.py on multiple locations, may be superseeded by setuptools in the future)
 * some paths needs to be absolute (chdir() during daemonize) and therefor additional checks introduced
 * test run script checks and fails if certain files are are missing (PCAP file <=> result output file)
 * removed not very useful "internal format error" JSON serialization if a BUG for same exists
 * fixed invalid l4 type statistics counters for nDPIsrvd-collectd

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-04-09 00:18:35 +02:00
Toni Uhlig
e576162a43 Reverted internal ndpi structs to dynamic memory allocation hoping that it will help to reduce the average memory consumption. 
How? After the detection finished, internal ndpi structs can be free'd as they are not needed anymore.

 * Set the amount of max. packets to process via subopt.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-03-26 14:46:35 +01:00
Toni Uhlig
38c6904bff run_tests.sh checks/print-stderr-on-fail 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-03-24 15:04:59 +01:00
Toni Uhlig
ef4a22ffac Updated README.md 
* added examples/README.md and schema/README.md

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-03-24 14:44:27 +01:00
Toni Uhlig
1ab6b9e042 Updated test outputs and pinned travis-ci's nDPI git repo sync to a specific commit hash. 
* fixed Makefile.old COpts hell

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-03-24 13:16:55 +01:00
Toni Uhlig
77b4b88b14 Added diff'able nDPId-test JSON dumps and travis-ci integration. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-03-24 11:33:04 +01:00
Toni Uhlig
e835d36f63 Added nDPId-test as all-in-one JSON dumper. 
* fixed invalid flow event schema type
 * added run_tests.sh to generate/diff JSON dumps
 * renamed lot's of vars/fns in nDPId.c/nDPIsrvd.c, so nDPId-test.c can include "*.c"
 * improved CMake dependency checks

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-03-24 11:28:56 +01:00
Toni Uhlig
bdc8c5df2a Reduced code duplication. Preps for nDPId-test. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-03-23 14:25:56 +01:00
Toni Uhlig
c68c1750ba Switched to CMake build system. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-03-17 17:41:19 +01:00
Toni Uhlig
1c3ef69faa nDPIsrvd collectd-exec overhaul. 
* Install targets updated.
 * Removed nDPIsrvd.h token validation function (done automatically by token_get).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-03-15 14:39:43 +01:00
Toni Uhlig
9a06b97473 Fixed collectd-exec issues. 
* Added collectd example config and types.db

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-03-12 17:30:04 +01:00
Toni Uhlig
772b67b767 Added basic collectd-exec example. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-03-10 18:32:09 +01:00
Toni Uhlig
412d8feba0 Added missing enum strings. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-03-09 17:46:16 +01:00
Toni Uhlig
1073c9626b nDPIsrvd refactoring 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-03-05 12:24:24 +01:00
Toni Uhlig
4699263d65 Fixed possible NULL pointer deref in dependencies/nDPIsrvd.h and examples/c-captured/c-captured.c 
* `make install` C examples
 * examples/py-flow-info/flow-info.py supports filtering
 * added breed to examples/py-flow-info/flow-info.py

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-02-28 11:27:22 +01:00
Toni Uhlig
06ff353099 Added JA3 / SSL SHA1 fingerprint blacklists. 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-02-27 17:47:49 +01:00
Toni Uhlig
e0310d7e1d Finalized examples/c-captured to dump packet bytes to PCAP for further analysis. 
* Fixed memory holes in nDPId structs.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-02-24 18:43:26 +01:00
Toni Uhlig
0b5b177c14 Extended nDPIsrvd.h with address parsing. 
* nDPId supports looading of custom nDPI protocol/category files
 * extended JSON schemas according to nDPI / nDPId JSON serializing
 * removed memory holes in nDPId
 * extended examples/c-captured

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2021-02-23 14:46:47 +01:00