Commit Graph

505 Commits

Author SHA1 Message Date
Damiano Verzulli
96b9129918 Improve README
- slightly improve the README to better enhance the streaming
  capability of `nDPId`, regardless of `nDPIsrvd`
- add a screencast showing the install step and the
  `nDPId` usage, alone (with ncat as unix-socket listener)
- add "build" to .gitignore
2022-08-12 11:10:45 +02:00
lns
ae37631e23 Do not SIGSEGV if a subopt has no value.
Signed-off-by: lns <matzeton@googlemail.com>
2022-08-08 09:33:26 +02:00
Toni Uhlig
ef94b83a62 Replaced outdated nDPI version info with the correct one.
* add CI job to verify the lowest known-to-work-libnDPI-version

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-14 04:09:15 +02:00
Toni Uhlig
fc442180da c-collectd: fixed possible undefined behavior
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-14 03:48:57 +02:00
Toni Uhlig
a606586a32 bump libnDPI to 7c19de49047a5731f3107ff17854e9afe839cc61
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-14 03:48:06 +02:00
Toni Uhlig
4a397ac646 Github Actions: Renamed branch 'master' to 'main'.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-12 16:06:05 +02:00
Toni Uhlig
28602ca095 README update
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-12 16:03:51 +02:00
Toni Uhlig
b5d4da8793 bump libnDPI to 8f6a006e36eef0ae386f7e663d3ebecfad6a2dc9
* try to use same wording wherever possible e.g.
   renamed workflow->total_l4_data_len to workflow->total_l4_payload_len

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-01 13:50:53 +02:00
Toni Uhlig
a80b6d7271 bump libnDPI to c287eb835b537ce64d9293a52ca13e670b6d3b0d
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-06-19 23:29:26 +02:00
lns
cdaeb1632e py-flow-dashboard: Improved graph axis scaling.
Signed-off-by: lns <matzeton@googlemail.com>
2022-06-16 11:37:33 +02:00
lns
2a8883a96e CMake: do not add /usr/include/ndpi to include dirs if BUILD_NDPI or STATIC_LIBNDPI_INSTALLDIR used.
* c-collectd: fixed memory leak on failure
 * py-flow-info.py: fancy spinners and stats counting improved

Signed-off-by: lns <matzeton@googlemail.com>
2022-06-10 14:34:30 +02:00
Toni Uhlig
664a8a077d Merge branch 'master' of github.com:lnslbrty/nDPId 2022-06-07 18:01:40 +02:00
Toni Uhlig
77a87254b6 nDPIsrvd.py: Throw SocketTimeout Exception to catch both timeout exceptions different Python versions can throw.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-06-07 17:59:47 +02:00
lns
3caf7727fd bump libnDPI to 0b3f8ed849cdf9971224c49a3958f0904a2bbbb5
* README/nDPId: fixed typ0

Signed-off-by: lns <matzeton@googlemail.com>
2022-06-06 00:34:13 +02:00
lns
f5b0021413 README update
Signed-off-by: lns <matzeton@googlemail.com>
2022-05-31 23:57:22 +02:00
Toni
73ca7fff3c Updated CI badges. 2022-05-08 21:41:01 +02:00
lns
4fde63b5c2 Small fixes.
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-26 11:44:31 +02:00
lns
0385653023 Github Actions: Build nDPId against lowest supported libnDPI release (4.2)
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-25 00:45:54 +02:00
lns
a46fc4153d nDPId: Merged nDPId_flow_(info|finished) into nDPId_flow
* nDPIsrvd: Fixed buffer allocation error due to missing memset() on disconnect
 * nDPIsrvd: Removed unused struct members

Signed-off-by: lns <matzeton@googlemail.com>
2022-04-24 23:49:57 +02:00
lns
22a8d04c74 Added proper DLT_RAW dissection for IPv4 and IPv6.
* nDPId: Improved TCP timeout handling if FIN/RST seen
	  which caused Midstream TCP flows when there shouldn't be any.
 * nDPIsrvd: Unified remote descriptor resource cleanup on disconnects/shutdown.
 * nDPIsrvd: Added additional error messages for remote descriptors.
 * py-flow-info: Better daemon status message printing.

Signed-off-by: lns <matzeton@googlemail.com>
2022-04-24 15:42:28 +02:00
lns
9aeff586bd bump libnDPI to 8b2c9860be8b0663bfe9fc3b6defc041bb90e5b2
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-18 19:26:27 +02:00
lns
c7bf94e9f1 nDPIsrvd.(h|py): Added socket read/recv timeout.
* nDPIsrvd.h: support for O_NONBLOCK nDPIsrvd_socket

Signed-off-by: lns <matzeton@googlemail.com>
2022-04-17 18:56:30 +02:00
lns
a2547321bb Added more CCs to Github Actions workflow.
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-17 11:28:59 +02:00
lns
c283b89afd Refactored buffer subsystem.
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-16 23:21:24 +02:00
lns
db83f82d29 Fixed build if BUILD_NDPI=ON. May happen during XCompilation.
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-16 22:18:19 +02:00
Toni Uhlig
645aeaf5b4 Avoid CMake searching for gcrypt as default.
* Not necessary anymore coz libnDPI has now a builtin gcrypt-light

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-04-02 01:21:15 +02:00
Toni Uhlig
9f9e881b3f bump libnDPI to bb12837ca75efc2691ecb18fd5f56e2d097ef26b
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-24 02:16:33 +01:00
Toni Uhlig
65a9e5a18d Executing ./tests/run_tests.sh w/o zLib should not result in diff's anymore.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-24 01:04:49 +01:00
Toni Uhlig
c0b7bdacbc Reworked nDPIsrvd.h C-API.
* nDPIsrvd.h: Provide nDPId thread storage.
 * nDPIsrvd.py: Fixed instance cleanup bug.
 * nDPIsrvd.h: Support for instance/thread user data and cleanup callback.
 * nDPIsrvd.h: Most recent flow time stored in thread ht instead of instance ht.
 * nDPId: Moved flow logger out the memory profilier into SIGUSR1 signal handling.
 * nDPId: Added signal fd to be usable within epoll's event handling (live-capture only!)
 * nDPId: Added information about ZLib compressions to daemon status/shutdown events.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-21 15:56:01 +01:00
Toni Uhlig
daaaa61519 Renamed basic event to error event for the sake of the logic.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-13 03:08:44 +01:00
Toni Uhlig
ed1647b944 Disconnect nDPIsrvd clients immediately instead waiting for a failed write().
* nDPIsrvd: Collector/Distributor logging improved
 * nDPIsrvd: Command line option for max remote descriptors
 * nDPId: Stop spamming nDPIsrvd Collector with the same events over and over again
 * nDPId: Refactored some variable names and events

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-13 02:28:10 +01:00
Toni Uhlig
dd35d9da3f CI: Fixed missing lcov prereq.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-12 11:17:03 +01:00
Toni Uhlig
f884a538ce Code coverage generation using LCOV.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-11 18:29:38 +01:00
Toni Uhlig
41757ecf1c Added nDPIsrvd TCP/IP support for distributors.
* nDPIsrvd: Improved distributor client disconnect detection
 * nDPIsrvd: Fixed invalid usage of epoll_add instead of epoll_mod
 * nPDIsrvd: Improved logging for distributor clients

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-10 14:26:07 +01:00
Toni Uhlig
6f1f9e65ea Fixed some pyhton issues with static class members.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-08 14:17:24 +01:00
Toni Uhlig
d0985a5732 Fixed build error regarding missing LINKTYPE_* define's.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-06 17:58:25 +01:00
Toni Uhlig
e09dd8509f Updated examples/README.md
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-06 17:41:38 +01:00
Toni Uhlig
29c72fb30b Removed go-dashboard example.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-06 17:40:35 +01:00
Toni Uhlig
46f68501d5 Added daemon event: DAEMON_EVENT_STATUS (periodically send's daemon statistics.)
* Improved distributor timeout handling (per-thread).
 * flow-info.py / flow-dash.py: Distinguish between flow risk severities.
 * nDPId: Skip tag switch datalink packet dissection / processing.
 * nDPId: Fixed incorrect value for current active flows.
 * Improved JSON schema's.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-06 17:38:05 +01:00
Toni Uhlig
9db048c9d9 Serialize flow risk score / confidence.
* bump libnDPI to 8b062295cc76a60e3905c054ce37bd17669464d1
 * removed ndpi_id_struct's

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-27 02:53:39 +01:00
Toni Uhlig
cb80c415d8 Improved py-flow-info to provide more optional information about received timestamps.
* py-flow-dashboard: Added color mapping for PieCharts/Graph that make more sense
 * nDPId: Renamed `flow_type' to a more precisely `flow_state'
 * nDPId: Changed the default setting to process only as much packets as libnDPI does

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-05 15:27:13 +01:00
Toni Uhlig
6fd6dff14d Added additional (minimalistic) detection information to flow updates.
This will only affect flows with the state `FT_FINISHED' (detection done).

 * nDPIsrvd.py: force use of JSON schema Draft 7 validator
 * flow-dash.py: gather/use total processed layer4 payload size
 * flow-info.py: added additional event filter
 * flow-info.py: prettified flow events printing whose detection is in progress
 * py-semantic-validation.py: added validation checks for FT_FINISHED
 * updated flow event JSON schema

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-04 01:12:18 +01:00
Toni Uhlig
f9e4c58854 Added logging interface used by nDPId, nDPIsrvd and nDPId-test.
* fixed GitLab pipeline
 * nDPId: added static assert (just for a test)
 * nDPId: memory profiling for total bytes compressed
 * nDPId-test: enable zLib compression if configured with ENABLE_ZLIB

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-03 03:48:37 +01:00
Toni Uhlig
1a0d7ddbfa Process additional layer 3 protocols.
* bump libnDPI to c53c82d4823b5a8f856d1375155ac5112b68e8af
 * run_tests.sh: improved execution from non-git directories e.g. via `make dist`
 * updated JSON schema to be more restrictive
 * nDPId: splitted generic get_ip_from_sockaddr into IPv4/IPv6 to prevent compiler warnings on some platforms

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-31 20:54:02 +01:00
Toni Uhlig
7022d0b1c5 nDPIsrvd: Fixed memory leak caused be not clearing buffer cache after a client disconnected.
* README.md: Fixed a typ0 and added a meh image from examples/py-flow-dashboard/flow-dash.py

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-26 15:38:43 +01:00
Toni Uhlig
80e1eedbef nDPId: Added some error messages when workflow init fails.
* Fixed invalid array subscript typ0 (caused some trouble..)
 * bump libnDPI to 2cd0479204301c50c6149706fcd4df3058b2a8cc

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-26 15:12:28 +01:00
Toni Uhlig
4bae9d0344 py-flow-dashboard: added tab layout and event pie chart
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-26 02:34:10 +01:00
Toni Uhlig
29a1b13e7a Improved Plotly/Dash example. It is now somehow informative.
* TCP timeout after FIN/RST: switched back to the value from a35fc1d5ea
 * py-flow-info: reset 'guessed' flag after detection/detection-update received

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-25 11:16:41 +01:00
Toni Uhlig
9e07a57566 Major nDPId extension. Sorry for the huge commit.
- nDPId: fixed invalid IP4/IP6 tuple compare
 - nDPIsrvd: fixed caching issue (finally)
 - added tiny c example (can be used to check flow manager sanity)
 - c-captured: use flow_last_seen timestamp from `struct nDPIsrvd_flow`
 - README.md update: added example JSON sequence
 - nDPId: added new flow event `update` necessary for correct
   timeout handling (and other future use-cases)
 - nDPIsrvd.h and nDPIsrvd.py: switched to an instance
   (consists of an alias/source tuple) based flow manager
 - every flow related event **must** now serialize `alias`, `source`,
   `flow_id`, `flow_last_seen` and `flow_idle_time` to make the timeout
   handling and verification process work correctly
 - nDPIsrvd.h: ability to profile any dynamic memory (de-)allocation
 - nDPIsrvd.py: removed PcapPacket class (unused)
 - py-flow-dashboard and py-flow-multiprocess: fixed race condition
 - py-flow-info: print statusbar with probably useful information
 - nDPId/nDPIsrvd.h: switched from packet-flow only timestamps (`pkt_*sec`)
   to a generic flow event timestamp `ts_msec`
 - nDPId-test: added additional checks
 - nDPId: increased ICMP flow timeout
 - nDPId: using event based i/o if capturing packets from a device
 - nDPIsrvd: fixed memory leak on shutdown if remote descriptors
   were still connected

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-20 00:50:38 +01:00
Toni Uhlig
a35fc1d5ea Removed py-flow-undetected-to-pcap and py-risky-flow-to-pcap. Done by c-captured anyway.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-21 12:01:45 +01:00