194 Commits

Author SHA1 Message Date
Toni Uhlig
99accd03a2 Moved datalink json key/value from error to packet events and renamed it to pkt_datalink.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-04-21 16:48:40 +02:00
Toni Uhlig
655393e953 nDPid: Fixed base64encode bug which lead to invalid base64 strings.
* py-semantic-validation: Decode base64 raw packet data as well
 * nDPIsrvd.py: Added PACKETS_PLEN_MAX
 * nDPIsrvd.py: Improved JSON parse error/exception handling

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-01-09 01:43:24 +01:00
Toni Uhlig
d21a38cf02 Limit the size of base64 serialized raw packet data (8192 bytes per packet).
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-12-06 12:52:52 +01:00
Toni Uhlig
2c95b31210 nDPId-test: Reworked I/O handling to prevent some endless loop scenarios. Fixed a race condition in the memory wrapper as well.
* nDPId: Instead of sending too long JSON strings, log an error and some parts.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-12-02 22:11:57 +01:00
Toni Uhlig
64f6abfdbe Unified nDPId/nDPIsrvd command line argument storage.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-11-21 11:26:05 +01:00
Toni Uhlig
ce567ae5b7 Improved the point of time when to append the raw packet base64 data to the serializer.
* nDPId-test: Increased the max-packets-per-flow-to-send from 3 to 5.
   This is quite useful for TCP as the first 3 packets are usually part of the three-way-handshake.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-11-15 06:25:16 +01:00
Toni Uhlig
ea1698504c nDPIsrvd: Provide workaround for change user/group.
* nDPId/nDPIsrvd/c-examples: Parameter parsing needs to be improved
                              if `strdup()` in combination with static strings is used.
 * Other non-critical fixes.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-11-13 11:21:55 +01:00
Toni Uhlig
bc346a28f4 nDPId: Fixed base64 encoding issue.
* The issue can result in an error message like:
   `Base64 encoding failed with: Buffer too small.`
   and also in too big JSON strings generated by nDPId
   which nDPIsrvd does not like as it's length is
   greater than `NETWORK_BUFFER_MAX_SIZE`.
 * nDPId will now obey `NETWORK_BUFFER_MAX_SIZE` while
   trying to base64 encode raw packet data.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-11-13 09:26:04 +01:00
Toni Uhlig
7515c8aeec Experimental systemd support.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-11-06 12:58:55 +01:00
Toni Uhlig
49352698a0 nDPId: Added error event threshold to prevent event spamming which may be abused.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-10-17 06:36:30 +02:00
Toni Uhlig
4654faf381 Improved py-machine-learning example.
* c-analysed: fixed quoting bug
 * nDPId: fixed invalid iat storing/serialisation
 * nDPId: free data analysis after event was sent

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
2022-10-09 18:31:45 +02:00
Toni Uhlig
14f6b87551 Added nDPIsrvd-analysed to generate CSV files from analyse events.
* nDPIsrvd.h: iterate over JSON arrays
 * nDPId: calculate l3 payload packet entropies for analysis

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-30 19:28:49 +02:00
Toni Uhlig
2103ee0811 Refactored client distributor C API.
* Still not perfect, but the code before was not even able to deal with JSON arrays.
   Use common "speaking" function names for all functions in nDPIsrvd.h
 * Provide a more or less generic and easy extendable JSON walk function.
 * Modified C examples to align with the changed C API.
 * c-collectd: Reduced lot's of code duplication by providing mapping tables.
 * nDPId: IAT array requires one slot less (first packet has always an IAT of 0).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-25 00:54:39 +02:00
Toni Uhlig
9a28475bba Improved flown analyse event:
* store packet directions
 * merged direction based IATs
 * merged direction based PKTLENs

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-22 19:07:08 +02:00
Toni Uhlig
08f263e409 nDPId: Reduced flow-updates for TCP flows to 1/4 of the timeout value.
* nDPId: Fixed broken validation tests.
 * nDPId: Removed TICK_RESOLUTION, not required anymore.
 * c-collectd: Improved total layer4 payload calculation/update handling.
 * c-collectd: Updated RRD Graph script according to total layer4 payload changes.
 * py-flow-info.py: Fixed several bugs and syntax errors.
 * Python scripts: Added dirname(argv[0]) as search path for nDPIsrvd.py.
 * nDPIsrvd&nDPId-test: Fixed missing EPOLLERR check.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-19 19:39:49 +02:00
Toni Uhlig
015a739efd Added layer4 payload length bins.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-19 10:14:37 +02:00
Toni Uhlig
efaa76e978 Provide thread sync via locking on architectures that do not support Compare&Swap.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-17 18:27:17 +02:00
lns
b8cfe1d6d3 Fixed last pkt time.
Signed-off-by: lns <matzeton@googlemail.com>
2022-09-14 11:22:41 +02:00
Toni Uhlig
d4633c1192 New flow event: 'analysis'.
* The goal was to provide a separate event for extracted feature that are not required
   and only useful for a few (e.g. someone who wants do ML).
 * Increased network buffer size to 32kB (8192 * 4).
 * Switched timestamp precision from ms to us for *ALL* timestamps.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-13 22:05:08 +02:00
Toni Uhlig
3bc6627dcc nDPId: Removed thread_id nonsense as it does not provide any useful information and is not portable at all, not even on Linux systems ..
* nDPId: Removed blocking I/O warning, which causes logspams..

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-10 23:11:03 +02:00
Toni Uhlig
7594180301 include fix
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-10 22:55:06 +02:00
Toni Uhlig
a992c79ab6 Fixed compilation warnings on linux32 platforms.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-10 22:45:12 +02:00
Toni Uhlig
6fe5d1da69 Do not use pthread_t as numeric value. Some systems define pthread_t as struct *
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-10 22:34:00 +02:00
Toni Uhlig
f9bd7d29ce Bump libnDPI to 37f918322c0a489b5143a987c8f1a44a6f78a6f3 and updated flow json schema file.
* export env vars AR / CMAKE_C_COMPILER_AR and RANLIB / CMAKE_C_COMPILER_RANLIB while building libnDPI
 * nDPId check API version during startup (macro vs. function call) and print a warning if they are different

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-06 14:50:46 +02:00
lns
d646ec5ab4 nDPId: Fixed fcntl() issue; invalid fcntl() set after a blocking-write.
* nDPId: imrpvoed collector socket error messages on connect/write/etc failures
 * reverted `netcat` parts of the README

Signed-off-by: lns <matzeton@googlemail.com>
2022-08-29 15:29:07 +02:00
lns
d9fadae718 nDPId: improved error messages if UNIX/UDP endpoint refuses connections/datagrams
Signed-off-by: lns <matzeton@googlemail.com>
2022-08-27 14:18:59 +02:00
Toni Uhlig
5e09a00062 nDPId: support for custom UDP endpoints
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-08-27 14:18:59 +02:00
lns
d0b0a50609 nDPId: improved error messages if UNIX/UDP endpoint refuses connections/datagrams
Signed-off-by: lns <matzeton@googlemail.com>
2022-08-27 13:04:17 +02:00
Toni Uhlig
e2e7c82d7f nDPId: support for custom UDP endpoints
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-08-27 13:04:17 +02:00
Toni Uhlig
0fd59f060e Split *_l4_payload_len' into *_src_l4_payload_len' and `*_dst_l4_payload_len'.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-08-15 22:55:19 +02:00
lns
905545487d Split flow_packets_processed' into flow_src_packets_processed' and `flow_dst_packets_processed'.
* no use for `flow_avg_l4_payload_len' -> removed
 * test/run_tests.sh does not fail if git-worktree's are used

Signed-off-by: lns <matzeton@googlemail.com>
2022-08-15 18:36:49 +02:00
lns
ae37631e23 Do not SIGSEGV if a subopt has no value.
Signed-off-by: lns <matzeton@googlemail.com>
2022-08-08 09:33:26 +02:00
Toni Uhlig
ef94b83a62 Replaced outdated nDPI version info with the correct one.
* add CI job to verify the lowest known-to-work-libnDPI-version

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-14 04:09:15 +02:00
Toni Uhlig
b5d4da8793 bump libnDPI to 8f6a006e36eef0ae386f7e663d3ebecfad6a2dc9
* try to use same wording wherever possible e.g.
   renamed workflow->total_l4_data_len to workflow->total_l4_payload_len

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-01 13:50:53 +02:00
lns
3caf7727fd bump libnDPI to 0b3f8ed849cdf9971224c49a3958f0904a2bbbb5
* README/nDPId: fixed typ0

Signed-off-by: lns <matzeton@googlemail.com>
2022-06-06 00:34:13 +02:00
lns
4fde63b5c2 Small fixes.
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-26 11:44:31 +02:00
lns
a46fc4153d nDPId: Merged nDPId_flow_(info|finished) into nDPId_flow
* nDPIsrvd: Fixed buffer allocation error due to missing memset() on disconnect
 * nDPIsrvd: Removed unused struct members

Signed-off-by: lns <matzeton@googlemail.com>
2022-04-24 23:49:57 +02:00
lns
22a8d04c74 Added proper DLT_RAW dissection for IPv4 and IPv6.
* nDPId: Improved TCP timeout handling if FIN/RST seen
	  which caused Midstream TCP flows when there shouldn't be any.
 * nDPIsrvd: Unified remote descriptor resource cleanup on disconnects/shutdown.
 * nDPIsrvd: Added additional error messages for remote descriptors.
 * py-flow-info: Better daemon status message printing.

Signed-off-by: lns <matzeton@googlemail.com>
2022-04-24 15:42:28 +02:00
Toni Uhlig
9f9e881b3f bump libnDPI to bb12837ca75efc2691ecb18fd5f56e2d097ef26b
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-24 02:16:33 +01:00
Toni Uhlig
65a9e5a18d Executing ./tests/run_tests.sh w/o zLib should not result in diff's anymore.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-24 01:04:49 +01:00
Toni Uhlig
c0b7bdacbc Reworked nDPIsrvd.h C-API.
* nDPIsrvd.h: Provide nDPId thread storage.
 * nDPIsrvd.py: Fixed instance cleanup bug.
 * nDPIsrvd.h: Support for instance/thread user data and cleanup callback.
 * nDPIsrvd.h: Most recent flow time stored in thread ht instead of instance ht.
 * nDPId: Moved flow logger out the memory profilier into SIGUSR1 signal handling.
 * nDPId: Added signal fd to be usable within epoll's event handling (live-capture only!)
 * nDPId: Added information about ZLib compressions to daemon status/shutdown events.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-21 15:56:01 +01:00
Toni Uhlig
daaaa61519 Renamed basic event to error event for the sake of the logic.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-13 03:08:44 +01:00
Toni Uhlig
ed1647b944 Disconnect nDPIsrvd clients immediately instead waiting for a failed write().
* nDPIsrvd: Collector/Distributor logging improved
 * nDPIsrvd: Command line option for max remote descriptors
 * nDPId: Stop spamming nDPIsrvd Collector with the same events over and over again
 * nDPId: Refactored some variable names and events

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-13 02:28:10 +01:00
Toni Uhlig
d0985a5732 Fixed build error regarding missing LINKTYPE_* define's.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-06 17:58:25 +01:00
Toni Uhlig
46f68501d5 Added daemon event: DAEMON_EVENT_STATUS (periodically send's daemon statistics.)
* Improved distributor timeout handling (per-thread).
 * flow-info.py / flow-dash.py: Distinguish between flow risk severities.
 * nDPId: Skip tag switch datalink packet dissection / processing.
 * nDPId: Fixed incorrect value for current active flows.
 * Improved JSON schema's.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-06 17:38:05 +01:00
Toni Uhlig
9db048c9d9 Serialize flow risk score / confidence.
* bump libnDPI to 8b062295cc76a60e3905c054ce37bd17669464d1
 * removed ndpi_id_struct's

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-27 02:53:39 +01:00
Toni Uhlig
cb80c415d8 Improved py-flow-info to provide more optional information about received timestamps.
* py-flow-dashboard: Added color mapping for PieCharts/Graph that make more sense
 * nDPId: Renamed `flow_type' to a more precisely `flow_state'
 * nDPId: Changed the default setting to process only as much packets as libnDPI does

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-05 15:27:13 +01:00
Toni Uhlig
6fd6dff14d Added additional (minimalistic) detection information to flow updates.
This will only affect flows with the state `FT_FINISHED' (detection done).

 * nDPIsrvd.py: force use of JSON schema Draft 7 validator
 * flow-dash.py: gather/use total processed layer4 payload size
 * flow-info.py: added additional event filter
 * flow-info.py: prettified flow events printing whose detection is in progress
 * py-semantic-validation.py: added validation checks for FT_FINISHED
 * updated flow event JSON schema

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-04 01:12:18 +01:00
Toni Uhlig
f9e4c58854 Added logging interface used by nDPId, nDPIsrvd and nDPId-test.
* fixed GitLab pipeline
 * nDPId: added static assert (just for a test)
 * nDPId: memory profiling for total bytes compressed
 * nDPId-test: enable zLib compression if configured with ENABLE_ZLIB

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-03 03:48:37 +01:00
Toni Uhlig
1a0d7ddbfa Process additional layer 3 protocols.
* bump libnDPI to c53c82d4823b5a8f856d1375155ac5112b68e8af
 * run_tests.sh: improved execution from non-git directories e.g. via `make dist`
 * updated JSON schema to be more restrictive
 * nDPId: splitted generic get_ip_from_sockaddr into IPv4/IPv6 to prevent compiler warnings on some platforms

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-31 20:54:02 +01:00