485 Commits

Author SHA1 Message Date
lns
9aeff586bd bump libnDPI to 8b2c9860be8b0663bfe9fc3b6defc041bb90e5b2
Signed-off-by: lns <matzeton@googlemail.com>
1.5
2022-04-18 19:26:27 +02:00
lns
c7bf94e9f1 nDPIsrvd.(h|py): Added socket read/recv timeout.
* nDPIsrvd.h: support for O_NONBLOCK nDPIsrvd_socket

Signed-off-by: lns <matzeton@googlemail.com>
2022-04-17 18:56:30 +02:00
lns
a2547321bb Added more CCs to Github Actions workflow.
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-17 11:28:59 +02:00
lns
c283b89afd Refactored buffer subsystem.
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-16 23:21:24 +02:00
lns
db83f82d29 Fixed build if BUILD_NDPI=ON. May happen during XCompilation.
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-16 22:18:19 +02:00
Toni Uhlig
645aeaf5b4 Avoid CMake searching for gcrypt as default.
* Not necessary anymore coz libnDPI has now a builtin gcrypt-light

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-04-02 01:21:15 +02:00
Toni Uhlig
9f9e881b3f bump libnDPI to bb12837ca75efc2691ecb18fd5f56e2d097ef26b
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-24 02:16:33 +01:00
Toni Uhlig
65a9e5a18d Executing ./tests/run_tests.sh w/o zLib should not result in diff's anymore.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-24 01:04:49 +01:00
Toni Uhlig
c0b7bdacbc Reworked nDPIsrvd.h C-API.
* nDPIsrvd.h: Provide nDPId thread storage.
 * nDPIsrvd.py: Fixed instance cleanup bug.
 * nDPIsrvd.h: Support for instance/thread user data and cleanup callback.
 * nDPIsrvd.h: Most recent flow time stored in thread ht instead of instance ht.
 * nDPId: Moved flow logger out the memory profilier into SIGUSR1 signal handling.
 * nDPId: Added signal fd to be usable within epoll's event handling (live-capture only!)
 * nDPId: Added information about ZLib compressions to daemon status/shutdown events.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-21 15:56:01 +01:00
Toni Uhlig
daaaa61519 Renamed basic event to error event for the sake of the logic.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-13 03:08:44 +01:00
Toni Uhlig
ed1647b944 Disconnect nDPIsrvd clients immediately instead waiting for a failed write().
* nDPIsrvd: Collector/Distributor logging improved
 * nDPIsrvd: Command line option for max remote descriptors
 * nDPId: Stop spamming nDPIsrvd Collector with the same events over and over again
 * nDPId: Refactored some variable names and events

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-13 02:28:10 +01:00
Toni Uhlig
dd35d9da3f CI: Fixed missing lcov prereq.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-12 11:17:03 +01:00
Toni Uhlig
f884a538ce Code coverage generation using LCOV.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-11 18:29:38 +01:00
Toni Uhlig
41757ecf1c Added nDPIsrvd TCP/IP support for distributors.
* nDPIsrvd: Improved distributor client disconnect detection
 * nDPIsrvd: Fixed invalid usage of epoll_add instead of epoll_mod
 * nPDIsrvd: Improved logging for distributor clients

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-10 14:26:07 +01:00
Toni Uhlig
6f1f9e65ea Fixed some pyhton issues with static class members.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-08 14:17:24 +01:00
Toni Uhlig
d0985a5732 Fixed build error regarding missing LINKTYPE_* define's.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-06 17:58:25 +01:00
Toni Uhlig
e09dd8509f Updated examples/README.md
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-06 17:41:38 +01:00
Toni Uhlig
29c72fb30b Removed go-dashboard example.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-06 17:40:35 +01:00
Toni Uhlig
46f68501d5 Added daemon event: DAEMON_EVENT_STATUS (periodically send's daemon statistics.)
* Improved distributor timeout handling (per-thread).
 * flow-info.py / flow-dash.py: Distinguish between flow risk severities.
 * nDPId: Skip tag switch datalink packet dissection / processing.
 * nDPId: Fixed incorrect value for current active flows.
 * Improved JSON schema's.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-06 17:38:05 +01:00
Toni Uhlig
9db048c9d9 Serialize flow risk score / confidence.
* bump libnDPI to 8b062295cc76a60e3905c054ce37bd17669464d1
 * removed ndpi_id_struct's

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-27 02:53:39 +01:00
Toni Uhlig
cb80c415d8 Improved py-flow-info to provide more optional information about received timestamps.
* py-flow-dashboard: Added color mapping for PieCharts/Graph that make more sense
 * nDPId: Renamed `flow_type' to a more precisely `flow_state'
 * nDPId: Changed the default setting to process only as much packets as libnDPI does

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-05 15:27:13 +01:00
Toni Uhlig
6fd6dff14d Added additional (minimalistic) detection information to flow updates.
This will only affect flows with the state `FT_FINISHED' (detection done).

 * nDPIsrvd.py: force use of JSON schema Draft 7 validator
 * flow-dash.py: gather/use total processed layer4 payload size
 * flow-info.py: added additional event filter
 * flow-info.py: prettified flow events printing whose detection is in progress
 * py-semantic-validation.py: added validation checks for FT_FINISHED
 * updated flow event JSON schema

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-04 01:12:18 +01:00
Toni Uhlig
f9e4c58854 Added logging interface used by nDPId, nDPIsrvd and nDPId-test.
* fixed GitLab pipeline
 * nDPId: added static assert (just for a test)
 * nDPId: memory profiling for total bytes compressed
 * nDPId-test: enable zLib compression if configured with ENABLE_ZLIB

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-03 03:48:37 +01:00
Toni Uhlig
1a0d7ddbfa Process additional layer 3 protocols.
* bump libnDPI to c53c82d4823b5a8f856d1375155ac5112b68e8af
 * run_tests.sh: improved execution from non-git directories e.g. via `make dist`
 * updated JSON schema to be more restrictive
 * nDPId: splitted generic get_ip_from_sockaddr into IPv4/IPv6 to prevent compiler warnings on some platforms

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-31 20:54:02 +01:00
Toni Uhlig
7022d0b1c5 nDPIsrvd: Fixed memory leak caused be not clearing buffer cache after a client disconnected.
* README.md: Fixed a typ0 and added a meh image from examples/py-flow-dashboard/flow-dash.py

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-26 15:38:43 +01:00
Toni Uhlig
80e1eedbef nDPId: Added some error messages when workflow init fails.
* Fixed invalid array subscript typ0 (caused some trouble..)
 * bump libnDPI to 2cd0479204301c50c6149706fcd4df3058b2a8cc

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-26 15:12:28 +01:00
Toni Uhlig
4bae9d0344 py-flow-dashboard: added tab layout and event pie chart
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-26 02:34:10 +01:00
Toni Uhlig
29a1b13e7a Improved Plotly/Dash example. It is now somehow informative.
* TCP timeout after FIN/RST: switched back to the value from a35fc1d5ea
 * py-flow-info: reset 'guessed' flag after detection/detection-update received

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-25 11:16:41 +01:00
Toni Uhlig
9e07a57566 Major nDPId extension. Sorry for the huge commit.
- nDPId: fixed invalid IP4/IP6 tuple compare
 - nDPIsrvd: fixed caching issue (finally)
 - added tiny c example (can be used to check flow manager sanity)
 - c-captured: use flow_last_seen timestamp from `struct nDPIsrvd_flow`
 - README.md update: added example JSON sequence
 - nDPId: added new flow event `update` necessary for correct
   timeout handling (and other future use-cases)
 - nDPIsrvd.h and nDPIsrvd.py: switched to an instance
   (consists of an alias/source tuple) based flow manager
 - every flow related event **must** now serialize `alias`, `source`,
   `flow_id`, `flow_last_seen` and `flow_idle_time` to make the timeout
   handling and verification process work correctly
 - nDPIsrvd.h: ability to profile any dynamic memory (de-)allocation
 - nDPIsrvd.py: removed PcapPacket class (unused)
 - py-flow-dashboard and py-flow-multiprocess: fixed race condition
 - py-flow-info: print statusbar with probably useful information
 - nDPId/nDPIsrvd.h: switched from packet-flow only timestamps (`pkt_*sec`)
   to a generic flow event timestamp `ts_msec`
 - nDPId-test: added additional checks
 - nDPId: increased ICMP flow timeout
 - nDPId: using event based i/o if capturing packets from a device
 - nDPIsrvd: fixed memory leak on shutdown if remote descriptors
   were still connected

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-20 00:50:38 +01:00
Toni Uhlig
a35fc1d5ea Removed py-flow-undetected-to-pcap and py-risky-flow-to-pcap. Done by c-captured anyway.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-21 12:01:45 +01:00
Toni Uhlig
cfecf3e110 go-dashboard renaming, ignore go-mod and it's file structure
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-21 11:45:45 +01:00
Toni Uhlig
25b974af67 Use blocking I/O to prevent data loss if nDPIsrvd too slow.
* Fixed MemoryProfiler stack overflow.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-16 15:59:38 +01:00
Toni Uhlig
d389f04135 MemoryProfiling: Advanced flow usage logging.
* nDPId-test: disable #include <syslog.h> if NO_MAIN macro defined
 * nDPId-test: mock syslog flags and functions
 * gitlab-ci: force -Werror

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-13 17:30:21 +01:00
Toni Uhlig
9075706714 nDPId-test: Set max buffer size for remote descriptors useful to test caching/buffering.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-04 14:08:25 +01:00
Toni Uhlig
1f6d1fbd67 Added timestamp validation test.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-02 12:15:41 +01:00
Toni Uhlig
d93c33aa74 Additional semantic validation tests.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-02 09:26:23 +01:00
Toni Uhlig
8ecd1b48ef c-captured: Improved format string in nDPIsrvd_write_flow_info_cb.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-08 19:01:39 +02:00
Toni Uhlig
3af8de5a58 Fixed compile error due to missing stdint.h include before ndpi_typedefs.h
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-08 11:42:27 +02:00
Toni Uhlig
315f90f982 Fixed invalid "flow_last_seen" timestamp for the first packet.
* After the first packet was processed, "flow_last_seen" was still 0.
   This behaviour is invalid as the first packet may contain l4 payload data e.g. for UDP
   and it also breaks nDPId json consistency "flow_first_seen" > 0, but "flow_last_seen" == 0.
 * JSON schema: set minimum timestamp value for Epoch timestamps to 24710 for flow_*_seen and
   1 for pcap packet ts. Those values are dependant on some manipulated pcap's in libnDPI/tests/pcap.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-08 11:31:58 +02:00
Toni Uhlig
fe77c44e3f Added support/debug function to write flow(-user) related info.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-08 10:56:23 +02:00
Toni Uhlig
3726311276 bump libnDPI to 181a03c5ad41bda533fbfa307627939c2ff30b75
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-05 23:39:11 +02:00
Toni Uhlig
a523c348f3 More CMake warnings/errors/fixes added.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-29 18:40:12 +02:00
Toni Uhlig
5a6b2aa261 CMake and CI extensions
* CPack support for debian packages
 * Use CPack version string for nDPId

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-29 15:46:47 +02:00
Toni Uhlig
992d3a207d dumb fuzzer: randpkt vs nDPId-test
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-20 00:28:44 +02:00
Toni Uhlig
7829bfe4e6 CI extended and fixups
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-19 11:30:55 +02:00
Toni Uhlig
4fa1694b05 Github Actions integration
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-17 18:59:49 +02:00
Toni Uhlig
c5be804725 Removed Travis-CI support as they do not support OpenSource anymore.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-16 16:48:55 +02:00
Toni Uhlig
655f38b68f Fixed some typ0's and reduced ICMP timeout to 10s.
* nDPId: Renamed some of the misleading terms, still TODO for nDPIsrvd
 * CMake improvments

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-16 16:43:30 +02:00
Toni Uhlig
4edf3bf7e6 Merge commit '1fa53c5bf8d0717f784c79abaa5111f88ab00221' 2021-09-15 17:04:21 +02:00
Toni Uhlig
1fa53c5bf8 Squashed 'dependencies/uthash/' changes from 8e67ced..bf15263
bf15263 Fix a "bug" in the example where option 3 interfered with option 1's counter.
b6e24ef Use `malloc(sizeof *s)` in example code.
a109c6b Stop using `gets` in example.c.
c85c9e1 fix: fix utstack example's compiling error
86e6776 Replace *.github.com urls with *.github.io (#227)
e493aa9 Bump version to 2.3.0.
ae2ac52 Fix README.md to display the *actual* TravisCI status.
134e241 Silence -Wswitch-default warnings, and add it to the TravisCI config.
62fefa6 Fix some typos in userguide.txt, and re-remove spaces in macro definitions.
37d2021 tests: add whitespaces to example code
524ca1a doc: add whitespaces to documentation
0f6c619 Fix a typo in the documentation for HASH_COUNT. NFC.
388134a Rename uthash_memcmp to HASH_KEYCMP, step 3.
053bed1 Eliminate HASH_FCN; change the handling of HASH_FUNCTION to match HASH_KEYCMP.
f0e1bd9 Refactor test93.c to avoid scan-build warnings.
45af88c Remove two dead writes in tests, to silence scan-build warnings.
66e2668 Bump version to 2.2.0.
973bd67 uthash.h: Swap multiplicands to put the widest ones first.
15ad042 Always include <stdint.h>, unless HASH_NO_STDINT is defined by the user.
6b4768b Rename uthash_memcmp to HASH_KEYCMP, step 2.
e64c7f0 Update tests/README to describe the most recently added tests. NFC.
c62796c HASH_CLEAR after some tests, to eliminate "memory leak" warnings.
7f0aadb Support spaces in $exe path
0831d9a uthash.h: fix compiler warning -Wcast-qual
ba2fbfd utarray.h: preserve constness in utarray_str_cpy

git-subtree-dir: dependencies/uthash
git-subtree-split: bf15263081be6229be31addd48566df93921cb46
2021-09-15 17:04:21 +02:00