75 Commits

Author SHA1 Message Date
Toni Uhlig
c8e1d1b7a5 Extend CI pipeline build and test.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-01-19 11:53:43 +01:00
Toni Uhlig
876aef98e1 Improved collectd example.
* similiar behavior to influxd example
 * gauges and counters are now handled properly

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-01-05 11:26:53 +01:00
Toni Uhlig
142a435bf6 Add InfluxDB push daemon.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-12-07 10:00:25 +01:00
Toni Uhlig
3416db11dc Updated ReadMe's, ToDo's and ChangeLog.
* 1.6-release

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-20 23:39:47 +01:00
Toni Uhlig
b0c343a795 Workaround for libpap (<1.9.0) on Ubuntu-18.04
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-20 12:41:24 +01:00
Toni Uhlig
d5266b7f44 Support simple config file reading via systemd environment file.
* cfg file path defaults to PREFIX/etc/default/ndpid

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-20 12:27:40 +01:00
Toni Uhlig
55c8a848d3 Fixed missing deflate during flow event json serializing.
* caused by recently added serializing some nDPI data even packet processing is still ongoing

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-08 13:16:57 +01:00
Toni
93498fff02 Apple/BSD port (#30)
* Add MacOS to Github CI builds.
* Fixed libnDPI-4.8 CI build.
* Fixed missing include for `struct sockaddr*`.
* Reworked IPv4 address and netmask retrieval.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-06 13:25:49 +01:00
Toni
1b67927169 Event I/O abstraction layer. (#28)
* Finalize Event I/O abstraction layer.
* Fix possible fd leakage, Gitlab-CI build and error logging.
* Fixed possible uninitialized signalfd variable.
* Fixed possible memory leak.
* Fixed some SonarCloud complaints.
* Fixed nDPId-test nDPIsrvd-arpa-mockup stuck indefinitely.
* Add nDPId / nDPIsrvd command line option to use poll() on Linux instead of the default epoll().

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-06 12:38:15 +01:00
Toni Uhlig
5335d84fe5 Add DBUS suspicious flow event notification daemon.
* nDPIsrvd.h: support for closing/resetting a nDPIsrvd_socket (required for a reconnect)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-03 16:07:28 +01:00
Toni Uhlig
73b8c378f2 nDPId event I/O fixes.
* forcibly disable epoll even if available
 * nDPId-test event I/O selftest
 * CI event I/O tests

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-05 12:32:45 +02:00
Toni Uhlig
e42e3fe406 Serialize nDPId / libnDPI versions within daemon events.
* changed nDPI version hints / requirements

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-09-10 19:32:08 +02:00
Toni Uhlig
96b0a8a474 Add event I/O abstraction.
* required to support non-Linux OS e.g. Mac OS X / BSD
 * see Github issue #19

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-09-10 19:11:58 +02:00
Toni Uhlig
091fd4d116 Added CMake option BUILD_NDPI_FORCE_GIT_UPDATE to fix broken submodule caches in GitLab CIs..
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-27 22:37:55 +02:00
Toni Uhlig
dd4357c238 CMake: install header files for experimental usage
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-05 01:15:48 +02:00
Toni Uhlig
7bd8081cd2 bump libpcap dependency to 1.9.0
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-07-27 18:20:36 +02:00
Toni Uhlig
967381a599 get-and-build-libndpi.sh uses GMake MAKEFLAGS for sub-make (required for e.g. jobserver)
* fixed invalid CMake `test -r ...`

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-07-16 22:06:31 +02:00
Toni Uhlig
c8ec505b9c bump libnDPI to 8ea0eaa0d0c4a3be05f67ef7fa1d22c2579cf7d1
* added build fix for Gitlab CI
 * added friendly C11 check
 * set required libnDPI versionto 4.7
   (ArchLinux ndpi-git sets version to 4.7, which is not released yet)
 * reduced sklearn-random-forest memory consumption by adjusting min. sample leaf

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-06-12 19:05:33 +02:00
lns
2b1db0a556 Required libnDPI version increases (>=4.8.0) due to an API change.
* fix CI issues

Signed-off-by: lns <matzeton@googlemail.com>
2023-05-31 12:53:49 +02:00
lns
d0c070a800 Added CentOs and ArchLinux to the CI.
* added some additional checks in get-and-build-libndpi.sh
 * CMake fallback library checks

Signed-off-by: lns <matzeton@googlemail.com>
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-05-28 00:46:58 +02:00
Toni Uhlig
c9514136b7 bump libnDPI to ...
* upstream changed regression test interface, needed to adapt
 * improved libnDPI helper build script
 * updated JSON schema

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-05-24 19:30:19 +02:00
Toni Uhlig
07efb1efd4 Added distclean-libnDPI target to CMake.
* Gitlab-CI: Additional job for debian packages
 * Install Python examples iff BUILD_EXAMPLES=ON

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-01-10 01:03:34 +01:00
Toni Uhlig
e9443d7618 Fix libnDPI build script.
* added ntop Webinar 2022 reference

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-12-26 19:35:12 +01:00
Toni Uhlig
869d4de271 Improved make daemon / daemon.sh to accept nDPId / nDPIsrvd arguments via env.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-11-15 07:05:29 +01:00
Toni Uhlig
7515c8aeec Experimental systemd support.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-11-06 12:58:55 +01:00
Toni Uhlig
9bf4f31418 Removed example py-ja3-checker.
* renamed sklearn-ml.py to sklearn-random-forest.py (there is more to come!)
 * force all protocol classes to lower case

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-10-13 00:12:22 +02:00
lns
b7d8564b65 Generate code coverage w/o external shell script, use CMake.
* upload codecov/dist artifacts

Signed-off-by: lns <matzeton@googlemail.com>
2022-10-03 15:45:17 +02:00
Toni Uhlig
b6060b897e c-analysed: improved feature extraction from "analyse" events
* c-captured: update detected risks on "detection-update" events
 * c-collectd: added missing flow breed
 * c-collectd: PUTVAL macros are more flexible now

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-10-01 18:01:56 +02:00
Toni Uhlig
14f6b87551 Added nDPIsrvd-analysed to generate CSV files from analyse events.
* nDPIsrvd.h: iterate over JSON arrays
 * nDPId: calculate l3 payload packet entropies for analysis

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-30 19:28:49 +02:00
Toni Uhlig
20ced3e636 nDPIsrvd-collectd: RRD Graph generation script and a basic static HTML5 website for viewing the generated image files.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-12 13:23:50 +02:00
Toni Uhlig
83409e5b79 Use CMake XCompile and collect host-triplet from ${CC}.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-11 16:50:56 +02:00
Toni Uhlig
f9bd7d29ce Bump libnDPI to 37f918322c0a489b5143a987c8f1a44a6f78a6f3 and updated flow json schema file.
* export env vars AR / CMAKE_C_COMPILER_AR and RANLIB / CMAKE_C_COMPILER_RANLIB while building libnDPI
 * nDPId check API version during startup (macro vs. function call) and print a warning if they are different

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-06 14:50:46 +02:00
Toni Uhlig
e2e7c82d7f nDPId: support for custom UDP endpoints
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-08-27 13:04:17 +02:00
lns
2a8883a96e CMake: do not add /usr/include/ndpi to include dirs if BUILD_NDPI or STATIC_LIBNDPI_INSTALLDIR used.
* c-collectd: fixed memory leak on failure
 * py-flow-info.py: fancy spinners and stats counting improved

Signed-off-by: lns <matzeton@googlemail.com>
2022-06-10 14:34:30 +02:00
lns
db83f82d29 Fixed build if BUILD_NDPI=ON. May happen during XCompilation.
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-16 22:18:19 +02:00
Toni Uhlig
645aeaf5b4 Avoid CMake searching for gcrypt as default.
* Not necessary anymore coz libnDPI has now a builtin gcrypt-light

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-04-02 01:21:15 +02:00
Toni Uhlig
9f9e881b3f bump libnDPI to bb12837ca75efc2691ecb18fd5f56e2d097ef26b
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-24 02:16:33 +01:00
Toni Uhlig
daaaa61519 Renamed basic event to error event for the sake of the logic.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-13 03:08:44 +01:00
Toni Uhlig
ed1647b944 Disconnect nDPIsrvd clients immediately instead waiting for a failed write().
* nDPIsrvd: Collector/Distributor logging improved
 * nDPIsrvd: Command line option for max remote descriptors
 * nDPId: Stop spamming nDPIsrvd Collector with the same events over and over again
 * nDPId: Refactored some variable names and events

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-13 02:28:10 +01:00
Toni Uhlig
dd35d9da3f CI: Fixed missing lcov prereq.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-12 11:17:03 +01:00
Toni Uhlig
f884a538ce Code coverage generation using LCOV.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-11 18:29:38 +01:00
Toni Uhlig
9db048c9d9 Serialize flow risk score / confidence.
* bump libnDPI to 8b062295cc76a60e3905c054ce37bd17669464d1
 * removed ndpi_id_struct's

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-27 02:53:39 +01:00
Toni Uhlig
f9e4c58854 Added logging interface used by nDPId, nDPIsrvd and nDPId-test.
* fixed GitLab pipeline
 * nDPId: added static assert (just for a test)
 * nDPId: memory profiling for total bytes compressed
 * nDPId-test: enable zLib compression if configured with ENABLE_ZLIB

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-03 03:48:37 +01:00
Toni Uhlig
9e07a57566 Major nDPId extension. Sorry for the huge commit.
- nDPId: fixed invalid IP4/IP6 tuple compare
 - nDPIsrvd: fixed caching issue (finally)
 - added tiny c example (can be used to check flow manager sanity)
 - c-captured: use flow_last_seen timestamp from `struct nDPIsrvd_flow`
 - README.md update: added example JSON sequence
 - nDPId: added new flow event `update` necessary for correct
   timeout handling (and other future use-cases)
 - nDPIsrvd.h and nDPIsrvd.py: switched to an instance
   (consists of an alias/source tuple) based flow manager
 - every flow related event **must** now serialize `alias`, `source`,
   `flow_id`, `flow_last_seen` and `flow_idle_time` to make the timeout
   handling and verification process work correctly
 - nDPIsrvd.h: ability to profile any dynamic memory (de-)allocation
 - nDPIsrvd.py: removed PcapPacket class (unused)
 - py-flow-dashboard and py-flow-multiprocess: fixed race condition
 - py-flow-info: print statusbar with probably useful information
 - nDPId/nDPIsrvd.h: switched from packet-flow only timestamps (`pkt_*sec`)
   to a generic flow event timestamp `ts_msec`
 - nDPId-test: added additional checks
 - nDPId: increased ICMP flow timeout
 - nDPId: using event based i/o if capturing packets from a device
 - nDPIsrvd: fixed memory leak on shutdown if remote descriptors
   were still connected

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-20 00:50:38 +01:00
Toni Uhlig
25b974af67 Use blocking I/O to prevent data loss if nDPIsrvd too slow.
* Fixed MemoryProfiler stack overflow.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-16 15:59:38 +01:00
Toni Uhlig
d389f04135 MemoryProfiling: Advanced flow usage logging.
* nDPId-test: disable #include <syslog.h> if NO_MAIN macro defined
 * nDPId-test: mock syslog flags and functions
 * gitlab-ci: force -Werror

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-13 17:30:21 +01:00
Toni Uhlig
a523c348f3 More CMake warnings/errors/fixes added.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-29 18:40:12 +02:00
Toni Uhlig
5a6b2aa261 CMake and CI extensions
* CPack support for debian packages
 * Use CPack version string for nDPId

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-29 15:46:47 +02:00
Toni Uhlig
7829bfe4e6 CI extended and fixups
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-19 11:30:55 +02:00
Toni Uhlig
4fa1694b05 Github Actions integration
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-17 18:59:49 +02:00