mirror of
https://github.com/optim-enterprises-bv/siembol.git
synced 2025-10-30 01:52:52 +00:00
Documentation: improve README. (#808)
* Add list of contents. * Add list of contents. * Add list of contents. * Add list of contents. * Fix header. * First draft of a new readme. * Made requested changes. Added badges, added link to changelog, and changed wording. * update readme * add services.md * fix path * fix grammar * add list of guides * Fixing formatting * remove colon Co-authored-by: Celinka Cywińska <celinkacywinska@Celinkas-MacBook-Air.local>
This commit is contained in:
Celina Cywinska
118
README.md
118
README.md
@@ -1,37 +1,91 @@
|
||||
|
||||
|
||||
[](https://www.blackhat.com/us-21/arsenal/schedule/index.html#siembol-an-open-source-real-time-siem-tool-based-on-big-data-technologies-24038)
|
||||
[](https://www.blackhat.com/us-21/arsenal/schedule/index.html#siembol-an-open-source-real-time-siem-tool-based-on-big-data-technologies-24038)
|
||||
[](https://www.blackhat.com/eu-21/arsenal/schedule/index.html#siembol-an-open-source-real-time-siem-tool-based-on-big-data-technologies-25165)
|
||||
[](https://www.blackhat.com/us-22/arsenal/schedule/#siembol-an-open-source-real-time-siem-tool-based-on-big-data-technologies-27927)
|
||||
|
||||
[](https://www.apache.org/licenses/LICENSE-2.0)
|
||||
|
||||
# Siembol
|
||||
|
||||
Siembol provides a scalable, advanced security analytics framework based on open-source big data technologies. Siembol normalizes, enriches, and alerts on data from various sources, which allows security teams to respond to attacks before they become incidents.
|
||||
|
||||
- [Introduction](/docs/introduction/introduction.md)
|
||||
- [How to try Siembol - Quickstart](/docs/introduction/how-tos/quickstart.md)
|
||||
- [How to contribute](/docs/introduction/how-tos/how_to_contribute.md)
|
||||
- [Siembol UI](/docs/siembol_ui/siembol_ui.md)
|
||||
- [Adding a new configuration](/docs/siembol_ui/how-tos/how_to_add_new_config_in_siembol_ui.md)
|
||||
- [Submitting configurations](/docs/siembol_ui/how-tos/how_to_submit_config_in_siembol_ui.md)
|
||||
- [Importing a sigma rule](/docs/siembol_ui/how-tos/how_to_import_sigma_rules.md)
|
||||
- [Releasing configurations](/docs/siembol_ui/how-tos/how_to_release_configurations_in_siembol_ui.md)
|
||||
- [Testing configurations](/docs/siembol_ui/how-tos/how_to_test_config_in_siembol_ui.md)
|
||||
- [Testing release](/docs/siembol_ui/how-tos/how_to_test_release_in_siembol_ui.md)
|
||||
- [Adding links to the homepage](/docs/siembol_ui/how-tos/how_to_add_links_to_siembol_ui_home_page.md)
|
||||
- [Setting up OAUTH2 OIDC](/docs/siembol_ui/how-tos/how_to_setup_oauth2_oidc_in_siembol_ui.md)
|
||||
- [Modifying the layout](/docs/siembol_ui/how-tos/how_to_modify_ui_layout.md)
|
||||
- [Managing applications](/docs/siembol_ui/how-tos/how_to_manage_applications.md)
|
||||
- [Use ui-bootstrap file](/docs/siembol_ui/how-tos/how_to_use_ui_bootstrap_file.md)
|
||||
- [Filter configs and save searches](/docs/siembol_ui/how-tos/how_to_filter_configs_and_save_searches.md)
|
||||
- Siembol services
|
||||
- [Setting up a service in the config editor rest](/docs/services/how-tos/how_to_set_up_service_in_config_editor_rest.md)
|
||||
- [Alerting service](/docs/services/siembol_alerting_services.md)
|
||||
- [Parsing service](/docs/services/siembol_parsing_services.md)
|
||||
- [How to setup NetFlow v9 parsing](/docs/services/how-tos/how_to_setup_netflow_v9_parsing.md)
|
||||
- [Enrichment service](/docs/services/siembol_enrichment_service.md)
|
||||
- [Setting up an enrichment table](/docs/services/how-tos/how_to_set_up_enrichment_table.md)
|
||||
- [Response service](/docs/services/siembol_response_service.md)
|
||||
- [Writing a response plugin](/docs/services/how-tos/how_to_write_response_plugin.md)
|
||||
- [Siembol deployment](/docs/deployment/deployment.md)
|
||||
- [Setting up ZooKeeper nodes](/docs/deployment/how-tos/how_to_set_up_zookeeper_nodes.md)
|
||||
- [Setting up a GitHub webhook](/docs/deployment/how-tos/how_to_setup_github_webhook.md)
|
||||
- [Tuning the performance of Storm topologies](/docs/deployment/how-tos/how_to_tune_performance_of_storm_topologies.md)
|
||||
- [Setting up Kerberos for external dependencies](/docs/deployment/how-tos/how_to_set_up_kerberos_for_external_dependencies.md)
|
||||
- [Customize Helm chart](/docs/deployment/how-tos/how_to_customize_helm_charts.md)
|
||||
Introduction
|
||||
------------
|
||||
|
||||
Siembol is an open-source, real-time security information and event management tool developed in-house at G-Research.
|
||||
|
||||
Siembol's use cases:
|
||||
|
||||
* **SIEM Log Collection Using Open Source Technologies**
|
||||
|
||||
Siembol can be used to centralize both security data collecting and the monitoring of logs from different sources.
|
||||
|
||||
* **Detection of Leaks and Attacks on Infrastructure**
|
||||
|
||||
Siembol can be used as a tool for detecting attacks or leaks by teams responsible for the system platform.
|
||||
|
||||
For more extensive introduction, visit: [Introduction](/docs/introduction/introduction.md).
|
||||
|
||||
Installation
|
||||
------------
|
||||
|
||||
To install locally, visit: [Quickstart Guide](/docs/introduction/how-tos/quickstart.md).
|
||||
|
||||
How to contribute
|
||||
-----------------
|
||||
|
||||
If you wish to contribute to Siembol, first read: [Contribution Guide](/docs/introduction/how-tos/how_to_contribute.md).
|
||||
|
||||
#### Code of Conduct
|
||||
G-Research has adopted a Code of Conduct that is to be honored by everyone who participates in the Siembol community formally or informally.
|
||||
Please read the full text: [Code of Conduct](/CODE_OF_CONDUCT.md)
|
||||
|
||||
####
|
||||
All notable changes to this project are documented in this file: [CHANGELOG](/CHANGELOG.md)
|
||||
|
||||
Siembol UI
|
||||
-------------
|
||||
|
||||
To learn more about Siembol's UI, visit: [Siembol UI](/docs/siembol_ui/siembol_ui.md).
|
||||
|
||||
There you will find guides on:
|
||||
- [Adding a new configuration](/docs/siembol_ui/how-tos/how_to_add_new_config_in_siembol_ui.md)
|
||||
- [Submitting configurations](/docs/siembol_ui/how-tos/how_to_submit_config_in_siembol_ui.md)
|
||||
- [Importing a sigma rule](/docs/siembol_ui/how-tos/how_to_import_sigma_rules.md)
|
||||
- [Releasing configurations](/docs/siembol_ui/how-tos/how_to_release_configurations_in_siembol_ui.md)
|
||||
- [Testing configurations](/docs/siembol_ui/how-tos/how_to_test_config_in_siembol_ui.md)
|
||||
- [Testing release](/docs/siembol_ui/how-tos/how_to_test_release_in_siembol_ui.md)
|
||||
- [Adding links to the homepage](/docs/siembol_ui/how-tos/how_to_add_links_to_siembol_ui_home_page.md)
|
||||
- [Setting up OAUTH2 OIDC](/docs/siembol_ui/how-tos/how_to_setup_oauth2_oidc_in_siembol_ui.md)
|
||||
- [Modifying the layout](/docs/siembol_ui/how-tos/how_to_modify_ui_layout.md)
|
||||
- [Managing applications](/docs/siembol_ui/how-tos/how_to_manage_applications.md)
|
||||
- [Use ui-bootstrap file](/docs/siembol_ui/how-tos/how_to_use_ui_bootstrap_file.md)
|
||||
- [Filter configs and save searches](/docs/siembol_ui/how-tos/how_to_filter_configs_and_save_searches.md)
|
||||
|
||||
Services
|
||||
---------
|
||||
|
||||
To explore Siembol's services, visit: [Siembol services](/docs/services/services.md).
|
||||
|
||||
There you will find guides on:
|
||||
- [Setting up a service in the config editor rest](/docs/services/how-tos/how_to_set_up_service_in_config_editor_rest.md)
|
||||
- [Alerting service](/docs/services/siembol_alerting_services.md)
|
||||
- [Parsing service](/docs/services/siembol_parsing_services.md)
|
||||
- [Setting up NetFlow v9 parsing](/docs/services/how-tos/how_to_setup_netflow_v9_parsing.md)
|
||||
- [Enrichment service](/docs/services/siembol_enrichment_service.md)
|
||||
- [Setting up an enrichment table](/docs/services/how-tos/how_to_set_up_enrichment_table.md)
|
||||
- [Response service](/docs/services/siembol_response_service.md)
|
||||
- [Writing a response plugin](/docs/services/how-tos/how_to_write_response_plugin.md)
|
||||
|
||||
Deployment
|
||||
----------
|
||||
|
||||
To deploy Siembol, refer to: [Siembol deployment](/docs/deployment/deployment.md).
|
||||
|
||||
There you will find guides on:
|
||||
- [Setting up ZooKeeper nodes](/docs/deployment/how-tos/how_to_set_up_zookeeper_nodes.md)
|
||||
- [Setting up a GitHub webhook](/docs/deployment/how-tos/how_to_setup_github_webhook.md)
|
||||
- [Tuning the performance of Storm topologies](/docs/deployment/how-tos/how_to_tune_performance_of_storm_topologies.md)
|
||||
- [Setting up Kerberos for external dependencies](/docs/deployment/how-tos/how_to_set_up_kerberos_for_external_dependencies.md)
|
||||
- [Customizing Helm chart](/docs/deployment/how-tos/how_to_customize_helm_charts.md)
|
||||
|
||||
10
docs/services/services.md
Normal file
10
docs/services/services.md
Normal file
@@ -0,0 +1,10 @@
|
||||
# Siembol services
|
||||
|
||||
- [Setting up a service in the config editor rest](/docs/services/how-tos/how_to_set_up_service_in_config_editor_rest.md)
|
||||
- [Alerting service](/docs/services/siembol_alerting_services.md)
|
||||
- [Parsing service](/docs/services/siembol_parsing_services.md)
|
||||
- [Setting up NetFlow v9 parsing](/docs/services/how-tos/how_to_setup_netflow_v9_parsing.md)
|
||||
- [Enrichment service](/docs/services/siembol_enrichment_service.md)
|
||||
- [Setting up an enrichment table](/docs/services/how-tos/how_to_set_up_enrichment_table.md)
|
||||
- [Response service](/docs/services/siembol_response_service.md)
|
||||
- [Writing a response plugin](/docs/services/how-tos/how_to_write_response_plugin.md)
|
||||
@@ -1,4 +1,19 @@
|
||||
# Siembol Alerting Services
|
||||
|
||||
- [1. Overview](#overview)
|
||||
- [2. Alert service](#alert-service)
|
||||
* [2.1 Common rule fields](#common-rule-fields)
|
||||
* [2.2 Alert rule](#alert-rule)
|
||||
* [2.2.1 Matchers](#matchers)
|
||||
* [2.3 Global Tags and Rule Protection](#global-tags-and-rule-protection)
|
||||
- [3. Correlation Rule](#correlation-rule)
|
||||
* [3.1 Overview](#overview-1)
|
||||
* [3.2 Correlation alert rule](#correlation-alert-rule)
|
||||
- [4. Admin config](#admin-config)
|
||||
* [4.1 Common admin config fields](#common-admin-config-fields)
|
||||
* [4.2 Alert admin config](#alert-admin-config)
|
||||
* [4.3 Correlation alert admin config](#correlation-alert-admin-config)
|
||||
|
||||
## Overview
|
||||
Siembol alert is a detection engine used to filter matching events from an incoming data stream based on a configurable rule set. The correlation alert allows you to group several detections together before raising an alert.
|
||||
## Alert service
|
||||
|
||||
@@ -1,4 +1,11 @@
|
||||
# Siembol Enrichment Service
|
||||
|
||||
- [1. Overview](#overview)
|
||||
* [1.1 Enrichment rule](#enrichment-rule)
|
||||
* [1.1.1 Matchers](#matchers)
|
||||
* [1.1.2 Table Mapping](#table-mapping)
|
||||
- [2. Admin config](#admin-config)
|
||||
|
||||
## Overview
|
||||
Siembol Enrichment is an enrichment engine used to add useful data to events to assist in detection and investigations.
|
||||
|
||||
@@ -81,4 +88,4 @@ Note: you can only enrich from one table per rule. If you want to enrich the sam
|
||||
- `enriching.input.topics`- The list of kafka input topics for reading messages
|
||||
- `enriching.output.topic` - Output kafka topic name for correctly processed messages
|
||||
- `enriching.error.topic` - Output kafka topic name for error messages
|
||||
- `enriching.tables.hdfs.uri` - The url for hdfs cluster where enriching tables are stored
|
||||
- `enriching.tables.hdfs.uri` - The url for hdfs cluster where enriching tables are stored
|
||||
|
||||
@@ -1,4 +1,31 @@
|
||||
# Siembol Parsing Services
|
||||
|
||||
- [1. Overview](#overview)
|
||||
* [1.1 Key concepts](#key-concepts)
|
||||
* [1.2 Common fields](#common-fields)
|
||||
- [2. Parser config](#parser-config)
|
||||
* [2.1 Parser Attributes](#parser-attributes)
|
||||
* [2.2 Parser Extractors](#parser-extractors)
|
||||
* [2.2.1 Overview](#overview-1)
|
||||
* [2.2.2 Common extractor attributes](#common-extractor-attributes)
|
||||
* [2.2.3 Pattern extractor](#pattern-extractor)
|
||||
* [2.2.4 Key value Extractor](#key-value-extractor)
|
||||
* [2.2.5 CSV Extractor](#csv-extractor)
|
||||
* [2.2.6 Json Extractor](#json-extractor)
|
||||
* [2.2.7 Json Path Extractor](#json-path-extractor)
|
||||
* [2.3 Parser Transformations](#parser-transformations)
|
||||
* [2.3.1 Overview](#overview-2)
|
||||
* [2.3.2 Common fields](#common-fields-1)
|
||||
- [3. Parsing application](#parsing-application)
|
||||
* [3.1 Overview](#overview-3)
|
||||
* [3.2 Single Parser](#single-parser)
|
||||
* [3.3 Router parsing](#router-parsing)
|
||||
* [3.4 Topic routing parsing](#topic-routing-parsing)
|
||||
* [3.5 Header routing parsing](#header-routing-parsing)
|
||||
- [4. Admin config](#admin-config)
|
||||
|
||||
|
||||
|
||||
## Overview
|
||||
Siembol provides parsing services for normalising logs into messages with one layer of key/value pairs. Clean normalised data is very important for further processing such as alerting.
|
||||
### Key concepts
|
||||
|
||||
@@ -1,4 +1,27 @@
|
||||
# Siembol Response Service
|
||||
|
||||
- [1. Overview](#overview)
|
||||
- [2. Siembol Response Rule](#siembol-response-rule)
|
||||
* [2.1 Evaluation](#evaluation)
|
||||
* [2.2 Response Rule](#response-rule)
|
||||
* [2.3 Provided evaluators](#provided-evaluators)
|
||||
* [2.3.1 Fixed result](#fixed-result)
|
||||
* [2.3.2 Matching](#matching)
|
||||
* [2.3.3 Json path assignment](#json-path-assignment)
|
||||
* [2.3.4 Markdown table formatter](#markdown-table-formatter)
|
||||
* [2.3.5 Array markdown table formatter](#array-markdown-table-formatter)
|
||||
* [2.3.6 Array reducer](#array-reducer)
|
||||
* [2.3.7 Alert throttling](#alert-throttling)
|
||||
* [2.3.8 Sleep](#sleep)
|
||||
* [2.3.9 Kafka writer](#kafka-writer)
|
||||
* [2.3.10 Time exclusion](#time-exclusion)
|
||||
- [3. Plugins](#plugins)
|
||||
* [3.1 Plugin architecture](#plugin-architecture)
|
||||
* [3.2 Evaluators - GR open source plans](#evaluators-\-\-gr-open-source-plans)
|
||||
- [4. Application Properties](#application-properties)
|
||||
* [4.1 Authentication](#authentication)
|
||||
* [4.1.1 Oauth2 Authentication](#oauth2-authentication)
|
||||
|
||||
## Overview
|
||||
Siembol response is a service for defining a response to an alert. It brings a functionality:
|
||||
- To integrate siembol with other systems such as jira, ldap, elk, the hive, cortex etc.
|
||||
@@ -126,7 +149,10 @@ Siembol response plugins allows to extend the functionality of siembol response
|
||||
### Plugin architecture
|
||||
A Siembol response plugin is a shaded jar file that includes all its dependencies see [how to write response plugin](how-tos/how_to_write_response_plugin.md). The plugins can be copied in a directory where they will be loaded by the [springboot properties launcher](https://docs.spring.io/spring-boot/docs/current/reference/html/appendix-executable-jar-format.html) The plugin is also integrated into siembol UI and its evaluators can be used in the similar way as the ones provided directly by siembol response.
|
||||
|
||||
### Evaluators implemented internally at GR that we are planning to open source
|
||||
### Evaluators - GR open source plans
|
||||
|
||||
Evaluators implemented internally at GR that are planned to become open source:
|
||||
|
||||
- Elk search
|
||||
- calling an Elastic Search query using LUCENE or json query syntax
|
||||
- Elk store
|
||||
|
||||
Reference in New Issue
Block a user