Commit Graph

460 Commits

Author SHA1 Message Date
Toni Uhlig
80e1eedbef nDPId: Added some error messages when workflow init fails.
* Fixed invalid array subscript typ0 (caused some trouble..)
 * bump libnDPI to 2cd0479204301c50c6149706fcd4df3058b2a8cc

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-26 15:12:28 +01:00
Toni Uhlig
4bae9d0344 py-flow-dashboard: added tab layout and event pie chart
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-26 02:34:10 +01:00
Toni Uhlig
29a1b13e7a Improved Plotly/Dash example. It is now somehow informative.
* TCP timeout after FIN/RST: switched back to the value from a35fc1d5ea
 * py-flow-info: reset 'guessed' flag after detection/detection-update received

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-25 11:16:41 +01:00
Toni Uhlig
9e07a57566 Major nDPId extension. Sorry for the huge commit.
- nDPId: fixed invalid IP4/IP6 tuple compare
 - nDPIsrvd: fixed caching issue (finally)
 - added tiny c example (can be used to check flow manager sanity)
 - c-captured: use flow_last_seen timestamp from `struct nDPIsrvd_flow`
 - README.md update: added example JSON sequence
 - nDPId: added new flow event `update` necessary for correct
   timeout handling (and other future use-cases)
 - nDPIsrvd.h and nDPIsrvd.py: switched to an instance
   (consists of an alias/source tuple) based flow manager
 - every flow related event **must** now serialize `alias`, `source`,
   `flow_id`, `flow_last_seen` and `flow_idle_time` to make the timeout
   handling and verification process work correctly
 - nDPIsrvd.h: ability to profile any dynamic memory (de-)allocation
 - nDPIsrvd.py: removed PcapPacket class (unused)
 - py-flow-dashboard and py-flow-multiprocess: fixed race condition
 - py-flow-info: print statusbar with probably useful information
 - nDPId/nDPIsrvd.h: switched from packet-flow only timestamps (`pkt_*sec`)
   to a generic flow event timestamp `ts_msec`
 - nDPId-test: added additional checks
 - nDPId: increased ICMP flow timeout
 - nDPId: using event based i/o if capturing packets from a device
 - nDPIsrvd: fixed memory leak on shutdown if remote descriptors
   were still connected

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-20 00:50:38 +01:00
Toni Uhlig
a35fc1d5ea Removed py-flow-undetected-to-pcap and py-risky-flow-to-pcap. Done by c-captured anyway.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-21 12:01:45 +01:00
Toni Uhlig
cfecf3e110 go-dashboard renaming, ignore go-mod and it's file structure
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-21 11:45:45 +01:00
Toni Uhlig
25b974af67 Use blocking I/O to prevent data loss if nDPIsrvd too slow.
* Fixed MemoryProfiler stack overflow.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-16 15:59:38 +01:00
Toni Uhlig
d389f04135 MemoryProfiling: Advanced flow usage logging.
* nDPId-test: disable #include <syslog.h> if NO_MAIN macro defined
 * nDPId-test: mock syslog flags and functions
 * gitlab-ci: force -Werror

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-13 17:30:21 +01:00
Toni Uhlig
9075706714 nDPId-test: Set max buffer size for remote descriptors useful to test caching/buffering.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-04 14:08:25 +01:00
Toni Uhlig
1f6d1fbd67 Added timestamp validation test.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-02 12:15:41 +01:00
Toni Uhlig
d93c33aa74 Additional semantic validation tests.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-02 09:26:23 +01:00
Toni Uhlig
8ecd1b48ef c-captured: Improved format string in nDPIsrvd_write_flow_info_cb.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-08 19:01:39 +02:00
Toni Uhlig
3af8de5a58 Fixed compile error due to missing stdint.h include before ndpi_typedefs.h
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-08 11:42:27 +02:00
Toni Uhlig
315f90f982 Fixed invalid "flow_last_seen" timestamp for the first packet.
* After the first packet was processed, "flow_last_seen" was still 0.
   This behaviour is invalid as the first packet may contain l4 payload data e.g. for UDP
   and it also breaks nDPId json consistency "flow_first_seen" > 0, but "flow_last_seen" == 0.
 * JSON schema: set minimum timestamp value for Epoch timestamps to 24710 for flow_*_seen and
   1 for pcap packet ts. Those values are dependant on some manipulated pcap's in libnDPI/tests/pcap.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-08 11:31:58 +02:00
Toni Uhlig
fe77c44e3f Added support/debug function to write flow(-user) related info.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-08 10:56:23 +02:00
Toni Uhlig
3726311276 bump libnDPI to 181a03c5ad41bda533fbfa307627939c2ff30b75
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-05 23:39:11 +02:00
Toni Uhlig
a523c348f3 More CMake warnings/errors/fixes added.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-29 18:40:12 +02:00
Toni Uhlig
5a6b2aa261 CMake and CI extensions
* CPack support for debian packages
 * Use CPack version string for nDPId

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-29 15:46:47 +02:00
Toni Uhlig
992d3a207d dumb fuzzer: randpkt vs nDPId-test
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-20 00:28:44 +02:00
Toni Uhlig
7829bfe4e6 CI extended and fixups
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-19 11:30:55 +02:00
Toni Uhlig
4fa1694b05 Github Actions integration
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-17 18:59:49 +02:00
Toni Uhlig
c5be804725 Removed Travis-CI support as they do not support OpenSource anymore.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-16 16:48:55 +02:00
Toni Uhlig
655f38b68f Fixed some typ0's and reduced ICMP timeout to 10s.
* nDPId: Renamed some of the misleading terms, still TODO for nDPIsrvd
 * CMake improvments

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-16 16:43:30 +02:00
Toni Uhlig
4edf3bf7e6 Merge commit '1fa53c5bf8d0717f784c79abaa5111f88ab00221' 2021-09-15 17:04:21 +02:00
Toni Uhlig
1fa53c5bf8 Squashed 'dependencies/uthash/' changes from 8e67ced..bf15263
bf15263 Fix a "bug" in the example where option 3 interfered with option 1's counter.
b6e24ef Use `malloc(sizeof *s)` in example code.
a109c6b Stop using `gets` in example.c.
c85c9e1 fix: fix utstack example's compiling error
86e6776 Replace *.github.com urls with *.github.io (#227)
e493aa9 Bump version to 2.3.0.
ae2ac52 Fix README.md to display the *actual* TravisCI status.
134e241 Silence -Wswitch-default warnings, and add it to the TravisCI config.
62fefa6 Fix some typos in userguide.txt, and re-remove spaces in macro definitions.
37d2021 tests: add whitespaces to example code
524ca1a doc: add whitespaces to documentation
0f6c619 Fix a typo in the documentation for HASH_COUNT. NFC.
388134a Rename uthash_memcmp to HASH_KEYCMP, step 3.
053bed1 Eliminate HASH_FCN; change the handling of HASH_FUNCTION to match HASH_KEYCMP.
f0e1bd9 Refactor test93.c to avoid scan-build warnings.
45af88c Remove two dead writes in tests, to silence scan-build warnings.
66e2668 Bump version to 2.2.0.
973bd67 uthash.h: Swap multiplicands to put the widest ones first.
15ad042 Always include <stdint.h>, unless HASH_NO_STDINT is defined by the user.
6b4768b Rename uthash_memcmp to HASH_KEYCMP, step 2.
e64c7f0 Update tests/README to describe the most recently added tests. NFC.
c62796c HASH_CLEAR after some tests, to eliminate "memory leak" warnings.
7f0aadb Support spaces in $exe path
0831d9a uthash.h: fix compiler warning -Wcast-qual
ba2fbfd utarray.h: preserve constness in utarray_str_cpy

git-subtree-dir: dependencies/uthash
git-subtree-split: bf15263081be6229be31addd48566df93921cb46
2021-09-15 17:04:21 +02:00
Toni Uhlig
2a5e5a020b Merge commit '8e096b19c1e0b45ccd43cc89d9d80b59bd783529' 2021-09-15 17:03:59 +02:00
Toni Uhlig
8e096b19c1 Squashed 'dependencies/jsmn/' changes from 053d3cd..1aa2e8f
1aa2e8f Update README.md (#203)
b85f161 Update README.md (#213)
23f13d2 Merge pull request #108 from olmokramer/patch-1

git-subtree-dir: dependencies/jsmn
git-subtree-split: 1aa2e8f80849c983466b165d53542da9b1bd1b32
2021-09-15 17:03:59 +02:00
Toni Uhlig
e54c2df63b nDPIsrvd: Fixed anther bug, introduced during refactoring -_-
nDPId-test: Collect information about JSON string length's.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-15 14:33:13 +02:00
Toni Uhlig
c152e41cfb README.md ascii update
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-14 18:54:33 +02:00
Toni Uhlig
aa89800ff9 fixed Warnings / build error / cosmetics
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-14 18:38:37 +02:00
Toni Uhlig
ea0b04d648 bump libnDPI to 0eb7a0388c4549ebbf8cd7a10d398088005cc2de
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-14 18:19:47 +02:00
Toni Uhlig
6faded3cc7 Improved and Fixed another buffering issue caused by removing an outgoing fd too early from epoll queue (EPOLLOUT).
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-08-05 02:02:51 +02:00
Toni Uhlig
d48508b4af Improved nDPIsrvd buffer bloat handling using caching.
* still allow blocking mode (with send timeout)
 * improved daemon start/stop test script

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-08-04 17:19:15 +02:00
Toni Uhlig
f4c8d96dd9 Gitlab-CI
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-08-03 13:06:12 +02:00
Toni Uhlig
3a76035570 bump libnDPI to 6b7e5fa8d251f11c1bae16ea892a43a92b098480
* fixed linking issue by using CMake to check if explicit link against libm required
 * make nDPIsrvd collectd exit if parent pid changed, meaning that collectd died somehow
 * nDPId-test restores SIGPIPE to the default handler (termination), so abnormal connection drop's do now have consequences

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-08-03 12:37:59 +02:00
Toni Uhlig
c32461b032 bump libnDPI to b95bd0358fd43d9fdfdc5266e3c8923b91e1d4db
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-07-14 11:55:17 +02:00
Toni Uhlig
6f04807236 Build JSMN with support for parent links.
* nDPIsrvd.h: iterate over subtokens
 * nDPIsrvd-captured: select/ unselect risky flows to capture

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-07-13 03:35:35 +02:00
Toni Uhlig
19e4038ce5 bump libnDPI to ced6fca184a4549333c2d582e53419f66cd99ec1
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-29 17:32:37 +02:00
Toni Uhlig
7d6366ebfc Updated CMake nDPId-test target;
* w/o zLib
 * gcrypt requires to be enabled

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-18 14:34:09 +02:00
Toni Uhlig
114365a480 Enable memory profiling for nDPId-test.
* print a summary

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-18 13:37:44 +02:00
Toni Uhlig
db87d45edb Added zLib compression parameters to control compression conditions.
* more structs are now "compressable"
 * fixed missing DAEMON_RECONNECT event
 * improved memory profiler

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-16 19:28:02 +02:00
Toni Uhlig
fac7648326 Support for zLib flow memory compression. Experimental.
Please use this feature only for testing purposes.
It will change or be removed in the future.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-14 15:33:29 +02:00
Toni Uhlig
98b11f814f Removed setting CC, CFLAGS and LDFLAGS explicitly for libnDPI build (BUILD_NDPI=ON).
* for xcompile targets e.g. for OpenWrt, this env vars are already set

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-09 14:14:25 +02:00
Toni Uhlig
e20280cb43 libndpi update
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-09 11:38:31 +02:00
Toni Uhlig
4d6ea33aa4 Trying to fix BUILD_NDPI for xcompilation.
* added a CMake warning as well

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-09 00:28:54 +02:00
Toni Uhlig
55ecf068b3 Generate a valid version tuple if build was triggered from an unpacked make dist archive.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-08 21:20:44 +02:00
Toni Uhlig
d3ebb84ce4 Fixed broken libnDPI build (BUILD_NDPI=ON) if Ninja used as Generator.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-08 21:00:18 +02:00
Toni Uhlig
7daeee141d make dist
* fixed run_tests.sh file check bug, CI compat
 * updated results due to libnDPI submodule update

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-08 16:18:54 +02:00
Toni Uhlig
a41ddafa88 Git tag/commit version printing for nDPId/nDPIsrvd. Reduces confusion.
* disabled subshell spawn for run_tests.sh, common pitfall while using counters

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-08 15:23:33 +02:00
Toni Uhlig
30502ff0a0 Fixed make daemon target.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-07 19:35:45 +02:00