mirror of
https://github.com/optim-enterprises-bv/siembol.git
synced 2025-10-29 17:42:43 +00:00
Documentation update (#798)
* Adding compatibility info. * Fixing a typo. * Adding list of content. * Fixing formatting. * Fixing formatting. * Adding info about Quickstart. * Fix formatting. * Create a list of content. * Fix format. * Fix the table of content. * Fix typo. * Fix typo again. * Fix format. * Trying out a different format. * Fix formatting. * Final formatting update. * Fixing issue with a "-" * Fixing an issue with a "-" * Update introduction.md * Update formatting. * Update formatting. * Add a list of contents. * Fix typo. * Added list of contents. * Fix spaces. * Delete placeholder section. * Fix names of sections in the list. * Remove 'how to". * Remove compatibility info. * Simplifying headers.
This commit is contained in:
Celina Cywinska
@@ -5,7 +5,7 @@
|
||||
Siembol provides a scalable, advanced security analytics framework based on open-source big data technologies. Siembol normalizes, enriches, and alerts on data from various sources, which allows security teams to respond to attacks before they become incidents.
|
||||
|
||||
- [Introduction](/docs/introduction/introduction.md)
|
||||
- [How to try Siembol](/docs/introduction/how-tos/quickstart.md)
|
||||
- [How to try Siembol - Quickstart](/docs/introduction/how-tos/quickstart.md)
|
||||
- [How to contribute](/docs/introduction/how-tos/how_to_contribute.md)
|
||||
- [Siembol UI](/docs/siembol_ui/siembol_ui.md)
|
||||
- [Adding a new configuration](/docs/siembol_ui/how-tos/how_to_add_new_config_in_siembol_ui.md)
|
||||
|
||||
@@ -1,4 +1,22 @@
|
||||
# Deployment
|
||||
|
||||
- [1. Build artifacts](#build-artifacts)
|
||||
* [1.1 Java artifacts](#java-artifacts)
|
||||
* [1.2 Docker images](#docker-images)
|
||||
* [1.3 Application configuration files](#application-configuration-files)
|
||||
- [2. Infrastructure dependencies](#infrastructure-dependencies)
|
||||
- [3. Helm charts](#helm-charts)
|
||||
* [3.1 Enrichment Store](#enrichment-store)
|
||||
* [3.2 Oauth2 Proxy](#oauth2-proxy)
|
||||
* [3.3 Ingress for dependencies](#ingress-for-dependencies)
|
||||
* [3.4 Enrichment Store Parameters](#enrichment-store)
|
||||
* [3.5 Storm Parameters](#storm)
|
||||
* [3.6 Enable & Disable components](#enable-and-disable-components)
|
||||
* [3.7 Customize Helm Chart](#customize-helm-chart)
|
||||
|
||||
|
||||
|
||||
|
||||
## Build artifacts
|
||||
Building and publishing artifacts are executed by [GitHub Actions](/.github/workflows/ci.yml) triggered by events in the siembol repository.
|
||||
|
||||
@@ -174,7 +192,7 @@ For the heartbeat to work config has to be added to each Siembol service type us
|
||||
See [siembol monitoring](how-tos/how_to_setup_siembol_monitoring.md) for configuration details.
|
||||
|
||||
|
||||
### Enable & Disable components
|
||||
### Enable and Disable components
|
||||
|
||||
All Siembol components can be enabled or disabled by modifying the `enabled_apps` list.
|
||||
By default the enabled_apps list consists of these components:
|
||||
|
||||
@@ -1,7 +1,22 @@
|
||||
How to Contribute
|
||||
=================
|
||||
|
||||
How to contribute to the siembol Java project
|
||||
- [1. Contribute to the siembol Java project](#contribute-to-the-siembol-java-project)
|
||||
* [1.1 Environment](#environment)
|
||||
* [1.2 Compile and install](#compile-and-install)
|
||||
* [1.3 Solving dependencies conflicts](#solving-dependencies-conflicts)
|
||||
* [1.4 Increasing the version of submodules](#increasing-the-version-of-submodules)
|
||||
- [2. Contribute to config editor UI project](#contribute-to-config-editor-ui-project)
|
||||
* [2.1 NodeJS version](#nodejs-version)
|
||||
* [2.2 Angular version](#angular-version)
|
||||
* [2.3 Build](#build)
|
||||
* [2.4 Development server](#development-server)
|
||||
* [2.5 Running unit tests](#running-unit-tests)
|
||||
* [2.6 Linting](#linting)
|
||||
* [2.7 Package lock file](#package-lock-file)
|
||||
* [2.8 Increasing the config editor's UI version](#increasing-the-config-editors-ui-version)
|
||||
|
||||
Contribute to the siembol Java project
|
||||
---------------------------------------------
|
||||
|
||||
### Environment
|
||||
@@ -9,7 +24,7 @@ How to contribute to the siembol Java project
|
||||
- [Maven](https://maven.apache.org/guides/) - version `3.5+`
|
||||
- [Java Development Kit 17+](https://adoptopenjdk.net/)
|
||||
|
||||
### How to compile and install
|
||||
### Compile and install
|
||||
|
||||
To install the project locally:
|
||||
|
||||
@@ -31,7 +46,7 @@ To build the project but skip testing:
|
||||
mvn clean package -DskipTests=true
|
||||
```
|
||||
|
||||
### How to solve dependencies conflicts
|
||||
### Solving dependencies conflicts
|
||||
|
||||
To obtain the dependency trees that can help to resolve issues with conflict in dependencies
|
||||
|
||||
@@ -39,7 +54,7 @@ To obtain the dependency trees that can help to resolve issues with conflict in
|
||||
mvn dependency:tree
|
||||
```
|
||||
|
||||
### How to increase the version of submodules
|
||||
### Increasing the version of submodules
|
||||
|
||||
To increment versions in all submodules:
|
||||
|
||||
@@ -50,8 +65,8 @@ mvn versions:set -DnewVersion=your_new_version
|
||||
> **_note:_** Incrementing the version is recommended for each PR with siembol java code change. Use version name with `SNAPSHOT` if you are not preparing a release version.
|
||||
|
||||
|
||||
How to contribute to config editor UI project
|
||||
---------------------------------------------
|
||||
Contribute to config editor UI project
|
||||
---------------------------------------
|
||||
|
||||
### NodeJS version
|
||||
|
||||
@@ -121,6 +136,6 @@ Always update and commit dependencies in the [package lock file](/config-editor/
|
||||
Get-Content package-lock.json | ConvertFrom-Json | ConvertTo-Json -compress -Depth 10 | Out-File package-lock2.json -Encoding ascii ; rm package-lock.json ; mv package-lock2.json package-lock.json
|
||||
```
|
||||
|
||||
### Increase the config editor's UI version
|
||||
### Increasing the config editor's UI version
|
||||
|
||||
The config editor's UI version should be increased after each code change in [package.json](/config-editor/config-editor-ui/package.json).
|
||||
|
||||
@@ -1,5 +1,26 @@
|
||||
Introduction
|
||||
=============
|
||||
|
||||
- [1. Siembol](#siembol)
|
||||
- [2. History](#history)
|
||||
- [3. Siembol vs Metron](#siembol-vs-metron)
|
||||
* [3.1 Components for Alert Escalation](#components-for-alert-escalation)
|
||||
* [3.2 Integration with Other Systems](#integration-with-other-systems)
|
||||
* [3.3 Advanced Parsing Framework for Building Fault-Tolerant Parsers](#advanced-parsing-framework-for-building-fault-tolerant-parsers)
|
||||
* [3.4 Advanced Enrichment Component](#advanced-enrichment-component)
|
||||
* [3.5 Configurations and Rules](#configurations-and-rules)
|
||||
* [3.6 Easy Installation](#easy-installation)
|
||||
- [4. Use Cases](#use-cases)
|
||||
* [4.1 SIEM Log Collection](#siem-log-collection-using-open-source-technologies)
|
||||
* [4.2 Detection of Leaks and Attacks on Infrastructure](#detection-of-leaks-and-attacks-on-infrastructure)
|
||||
- [5. High Level Architecture](#high-level-architecture)
|
||||
* [5.1 Data Pipelines](#data-pipelines)
|
||||
* [5.2 Services](#services)
|
||||
* [5.3 Infrastructure dependencies](#infrastructure-dependencies)
|
||||
* [5.4 Architecture](#architecture)
|
||||
|
||||
Siembol
|
||||
=======
|
||||
--------
|
||||
|
||||
Siembol provides a scalable, advanced security analytics framework based on open-source big data technologies. Siembol normalizes, enriches, and alerts on data from various sources, allowing security teams to respond to attacks before they become incidents.
|
||||
|
||||
@@ -10,7 +31,7 @@ Siembol was developed in-house at G-Research as a security data processing appli
|
||||
|
||||
As early adopters of Metron, we believed in the product and tried hard to adapt it to our needs. Ultimately, we recognized its limitations and we began to add the missing features and shore up its instabilities. Sadly, by the time we were able to give back to the Metron community, Metron's time had passed. However, as we still believe in the core mission of Metron, we are releasing our work under the project name, 'Siembol'. We hope this will provide the security community with an effective alternative, filling the void left by Metron's move to the [Apache Attic](https://attic.apache.org/).
|
||||
|
||||
How Siembol Improves upon Metron
|
||||
Siembol vs Metron
|
||||
--------------------------------
|
||||
|
||||
_Components for alert escalation._ CSIRT security teams can easily create a rule-based alert from a single data source, or they can create advanced correlation rules that combine various data sources. We are planning to release a tool for translating Sigma rule specification into the alerting rule engine soon after open-sourcing.
|
||||
@@ -33,7 +54,9 @@ _Easy installation for use with prepared Docker images and Helm charts._ Metron
|
||||
- Security teams can easily create a rule-based alert from a single data source, or they can create advanced correlation rules that combine various data sources.
|
||||
- Siembol UI supports translating Sigma rule specification (generic and open signature format for SIEM alerting [https://github.com/SigmaHQ/sigma](https://github.com/SigmaHQ/sigma)) into the Siembol alerting rule.
|
||||
|
||||
### Integration with Other Systems – Siembol Response
|
||||
### Integration with Other Systems
|
||||
|
||||
Siembol's approach:
|
||||
|
||||
- Easy way to integrate Siembol with other systems such as [Jira](https://www.atlassian.com/software/jira), [Cortex](https://github.com/TheHive-Project/Cortex), [ELK](https://www.elastic.co/what-is/elk-stack), and [LDAP](https://ldap.com/).
|
||||
- Functionality to provide additional enrichments about an alert, such as [ELK](https://www.elastic.co/what-is/elk-stack) searches or [LDAP](https://ldap.com/) searches, with the option to filter the alert as part of an automatic incident response.
|
||||
@@ -53,7 +76,9 @@ _Easy installation for use with prepared Docker images and Helm charts._ Metron
|
||||
|
||||
- Defining rules for selecting enrichment logic, joining enrichment tables, and defining how to enrich the processed log.
|
||||
|
||||
### Configurations and Rules are Defined by a Web Application Siembol UI
|
||||
### Configurations and Rules
|
||||
|
||||
Configurations and rules are defined by a web application Siembol UI.
|
||||
|
||||
- All configurations are stored in JSON format and edited by web forms in order to avoid mistakes and speed up creation and learning time.
|
||||
- Configurations are stored in Git repositories.
|
||||
@@ -65,8 +90,9 @@ _Easy installation for use with prepared Docker images and Helm charts._ Metron
|
||||
- All Siembol services can have multiple instances with authorization based on OIDC group membership. This allows multi-tenancy usage without the need to deploy multiple instances of Siembol.
|
||||
- We are planning to test and tune OAUTH/OIDC integration with popular identity providers.
|
||||
|
||||
### Easy Installation to Try Out with Prepared Docker Images and Helm Charts
|
||||
### Easy Installation
|
||||
|
||||
- Easy installation is possible with prepared Docker Images and Helm Charts.
|
||||
- Siembol supports deployment on external [Hadoop](https://hadoop.apache.org/) clusters to ensure high performance. However, we are providing k8s Helm charts for all deployment dependencies in order to test Siembol in development environments.
|
||||
|
||||
|
||||
@@ -82,7 +108,7 @@ Use-Cases
|
||||
- At G-Research we use Siembol to parse, normalize, enrich and detect approximately 150k events a second. Per day, this adds up to volumes of approximately 15TB of raw data, or 13 billion events.
|
||||
|
||||
|
||||
### Detection Tool for Detection of Leaks and Attacks on Infrastructure
|
||||
### Detection of Leaks and Attacks on Infrastructure
|
||||
|
||||
- Siembol can be used as a tool for detecting attacks or leaks by teams responsible for the system platform. For example, the Big Data team at G-Research is using Siembol to detect leaks and attacks on the [Hadoop](https://hadoop.apache.org/) platform. These detections are then used as another data source within the Siembol SIEM log collection for the CSIRT team handling these incidents.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user