Documentation: improve README. (#808)

* Add list of contents.

* Add list of contents.

* Add list of contents.

* Add list of contents.

* Fix header.

* First draft of a new readme.

* Made requested changes.

Added badges, added link to changelog, and changed wording.

* update readme

* add services.md

* fix path

* fix grammar

* add list of guides

* Fixing formatting

* remove colon

Co-authored-by: Celinka Cywińska <celinkacywinska@Celinkas-MacBook-Air.local>

README.md

@@ -1,37 +1,91 @@
 ![Siembol](logo.svg)
 
-[![Black Hat Arsenal](https://raw.githubusercontent.com/toolswatch/badges/54ad78bc63b24ce445e8241f179fe1ddeecf8eef/arsenal/usa/2021.svg)](https://www.blackhat.com/us-21/arsenal/schedule/index.html#siembol-an-open-source-real-time-siem-tool-based-on-big-data-technologies-24038) 
+[![Black Hat Arsenal](https://raw.githubusercontent.com/toolswatch/badges/54ad78bc63b24ce445e8241f179fe1ddeecf8eef/arsenal/usa/2021.svg)](https://www.blackhat.com/us-21/arsenal/schedule/index.html#siembol-an-open-source-real-time-siem-tool-based-on-big-data-technologies-24038)
+[![Black Hat Arsenal](https://raw.githubusercontent.com/toolswatch/badges/master/arsenal/europe/2021.svg?sanitize=true)](https://www.blackhat.com/eu-21/arsenal/schedule/index.html#siembol-an-open-source-real-time-siem-tool-based-on-big-data-technologies-25165)
+[![Black Hat Arsenal](https://raw.githubusercontent.com/toolswatch/badges/master/arsenal/usa/2022.svg?sanitize=true)](https://www.blackhat.com/us-22/arsenal/schedule/#siembol-an-open-source-real-time-siem-tool-based-on-big-data-technologies-27927)
+
+[![Apache License](https://img.shields.io/badge/License-Apache%202.0-blue)](https://www.apache.org/licenses/LICENSE-2.0)
+
+# Siembol 
 
 Siembol provides a scalable, advanced security analytics framework based on open-source big data technologies. Siembol normalizes, enriches, and alerts on data from various sources, which allows security teams to respond to attacks before they become incidents.
 
-- [Introduction](/docs/introduction/introduction.md)
+Introduction
-    - [How to try Siembol - Quickstart](/docs/introduction/how-tos/quickstart.md)
+------------
-    - [How to contribute](/docs/introduction/how-tos/how_to_contribute.md)
+
-- [Siembol UI](/docs/siembol_ui/siembol_ui.md)
+Siembol is an open-source, real-time security information and event management tool developed in-house at G-Research.
-    - [Adding a new configuration](/docs/siembol_ui/how-tos/how_to_add_new_config_in_siembol_ui.md)
+ 
-    - [Submitting configurations](/docs/siembol_ui/how-tos/how_to_submit_config_in_siembol_ui.md)
+Siembol's use cases:
-    - [Importing a sigma rule](/docs/siembol_ui/how-tos/how_to_import_sigma_rules.md)
+
-    - [Releasing configurations](/docs/siembol_ui/how-tos/how_to_release_configurations_in_siembol_ui.md)
+* **SIEM Log Collection Using Open Source Technologies**
-    - [Testing configurations](/docs/siembol_ui/how-tos/how_to_test_config_in_siembol_ui.md)
+
-    - [Testing release](/docs/siembol_ui/how-tos/how_to_test_release_in_siembol_ui.md)  
+Siembol can be used to centralize both security data collecting and the monitoring of logs from different sources.
-    - [Adding links to the homepage](/docs/siembol_ui/how-tos/how_to_add_links_to_siembol_ui_home_page.md)
+
-    - [Setting up OAUTH2 OIDC](/docs/siembol_ui/how-tos/how_to_setup_oauth2_oidc_in_siembol_ui.md)
+* **Detection of Leaks and Attacks on Infrastructure**
-    - [Modifying the layout](/docs/siembol_ui/how-tos/how_to_modify_ui_layout.md)
+
-    - [Managing applications](/docs/siembol_ui/how-tos/how_to_manage_applications.md)
+Siembol can be used as a tool for detecting attacks or leaks by teams responsible for the system platform. 
-    - [Use ui-bootstrap file](/docs/siembol_ui/how-tos/how_to_use_ui_bootstrap_file.md)
+
-    - [Filter configs and save searches](/docs/siembol_ui/how-tos/how_to_filter_configs_and_save_searches.md)
+For more extensive introduction, visit: [Introduction](/docs/introduction/introduction.md).
-- Siembol services    
+
-    - [Setting up a service in the config editor rest](/docs/services/how-tos/how_to_set_up_service_in_config_editor_rest.md)
+Installation
-    - [Alerting service](/docs/services/siembol_alerting_services.md)
+------------
-    - [Parsing service](/docs/services/siembol_parsing_services.md)
+
-        - [How to setup NetFlow v9 parsing](/docs/services/how-tos/how_to_setup_netflow_v9_parsing.md)
+To install locally, visit: [Quickstart Guide](/docs/introduction/how-tos/quickstart.md).
-    - [Enrichment service](/docs/services/siembol_enrichment_service.md)
+
-        - [Setting up an enrichment table](/docs/services/how-tos/how_to_set_up_enrichment_table.md)
+How to contribute
-    - [Response service](/docs/services/siembol_response_service.md)
+-----------------
-        - [Writing a response plugin](/docs/services/how-tos/how_to_write_response_plugin.md)
+
-- [Siembol deployment](/docs/deployment/deployment.md)
+If you wish to contribute to Siembol, first read: [Contribution Guide](/docs/introduction/how-tos/how_to_contribute.md).
-    - [Setting up ZooKeeper nodes](/docs/deployment/how-tos/how_to_set_up_zookeeper_nodes.md)
+
-    - [Setting up a GitHub webhook](/docs/deployment/how-tos/how_to_setup_github_webhook.md)
+#### Code of Conduct
-    - [Tuning the performance of Storm topologies](/docs/deployment/how-tos/how_to_tune_performance_of_storm_topologies.md)
+G-Research has adopted a Code of Conduct that is to be honored by everyone who participates in the Siembol community formally or informally.
-    - [Setting up Kerberos for external dependencies](/docs/deployment/how-tos/how_to_set_up_kerberos_for_external_dependencies.md)
+Please read the full text: [Code of Conduct](/CODE_OF_CONDUCT.md)
-    - [Customize Helm chart](/docs/deployment/how-tos/how_to_customize_helm_charts.md)
+
+####
+All notable changes to this project are documented in this file: [CHANGELOG](/CHANGELOG.md)
+
+Siembol UI
+-------------
+
+To learn more about Siembol's UI, visit: [Siembol UI](/docs/siembol_ui/siembol_ui.md).
+
+There you will find guides on:
+- [Adding a new configuration](/docs/siembol_ui/how-tos/how_to_add_new_config_in_siembol_ui.md)
+- [Submitting configurations](/docs/siembol_ui/how-tos/how_to_submit_config_in_siembol_ui.md)
+- [Importing a sigma rule](/docs/siembol_ui/how-tos/how_to_import_sigma_rules.md)
+- [Releasing configurations](/docs/siembol_ui/how-tos/how_to_release_configurations_in_siembol_ui.md)
+- [Testing configurations](/docs/siembol_ui/how-tos/how_to_test_config_in_siembol_ui.md)
+- [Testing release](/docs/siembol_ui/how-tos/how_to_test_release_in_siembol_ui.md)  
+- [Adding links to the homepage](/docs/siembol_ui/how-tos/how_to_add_links_to_siembol_ui_home_page.md)
+- [Setting up OAUTH2 OIDC](/docs/siembol_ui/how-tos/how_to_setup_oauth2_oidc_in_siembol_ui.md)
+- [Modifying the layout](/docs/siembol_ui/how-tos/how_to_modify_ui_layout.md)
+- [Managing applications](/docs/siembol_ui/how-tos/how_to_manage_applications.md)
+- [Use ui-bootstrap file](/docs/siembol_ui/how-tos/how_to_use_ui_bootstrap_file.md)
+- [Filter configs and save searches](/docs/siembol_ui/how-tos/how_to_filter_configs_and_save_searches.md)
+
+Services
+---------
+
+To explore Siembol's services, visit: [Siembol services](/docs/services/services.md).
+
+There you will find guides on:
+- [Setting up a service in the config editor rest](/docs/services/how-tos/how_to_set_up_service_in_config_editor_rest.md)
+- [Alerting service](/docs/services/siembol_alerting_services.md)
+- [Parsing service](/docs/services/siembol_parsing_services.md)
+  - [Setting up NetFlow v9 parsing](/docs/services/how-tos/how_to_setup_netflow_v9_parsing.md)
+- [Enrichment service](/docs/services/siembol_enrichment_service.md)
+  - [Setting up an enrichment table](/docs/services/how-tos/how_to_set_up_enrichment_table.md)
+- [Response service](/docs/services/siembol_response_service.md)
+  - [Writing a response plugin](/docs/services/how-tos/how_to_write_response_plugin.md)
+        
+Deployment
+----------
+
+To deploy Siembol, refer to: [Siembol deployment](/docs/deployment/deployment.md).
+
+There you will find guides on:
+- [Setting up ZooKeeper nodes](/docs/deployment/how-tos/how_to_set_up_zookeeper_nodes.md)
+- [Setting up a GitHub webhook](/docs/deployment/how-tos/how_to_setup_github_webhook.md)
+- [Tuning the performance of Storm topologies](/docs/deployment/how-tos/how_to_tune_performance_of_storm_topologies.md)
+- [Setting up Kerberos for external dependencies](/docs/deployment/how-tos/how_to_set_up_kerberos_for_external_dependencies.md)
+- [Customizing Helm chart](/docs/deployment/how-tos/how_to_customize_helm_charts.md)

docs/services/services.md

@@ -0,0 +1,10 @@
+# Siembol services
+
+- [Setting up a service in the config editor rest](/docs/services/how-tos/how_to_set_up_service_in_config_editor_rest.md)
+- [Alerting service](/docs/services/siembol_alerting_services.md)
+- [Parsing service](/docs/services/siembol_parsing_services.md)
+  - [Setting up NetFlow v9 parsing](/docs/services/how-tos/how_to_setup_netflow_v9_parsing.md)
+- [Enrichment service](/docs/services/siembol_enrichment_service.md)
+  - [Setting up an enrichment table](/docs/services/how-tos/how_to_set_up_enrichment_table.md)
+- [Response service](/docs/services/siembol_response_service.md)
+  - [Writing a response plugin](/docs/services/how-tos/how_to_write_response_plugin.md)

docs/services/siembol_alerting_services.md

@@ -1,4 +1,19 @@
 # Siembol Alerting Services
+
+- [1. Overview](#overview)
+- [2. Alert service](#alert-service)
+  * [2.1 Common rule fields](#common-rule-fields)
+  * [2.2 Alert rule](#alert-rule)
+      * [2.2.1 Matchers](#matchers)
+  * [2.3 Global Tags and Rule Protection](#global-tags-and-rule-protection)
+- [3. Correlation Rule](#correlation-rule)
+  * [3.1 Overview](#overview-1)
+  * [3.2 Correlation alert rule](#correlation-alert-rule)
+- [4. Admin config](#admin-config)
+  * [4.1 Common admin config fields](#common-admin-config-fields)
+  * [4.2 Alert admin config](#alert-admin-config)
+  * [4.3 Correlation alert admin config](#correlation-alert-admin-config)
+
 ## Overview
 Siembol alert is a detection engine used to filter matching events from an incoming data stream based on a configurable rule set. The correlation alert allows you to group several detections together before raising an alert.
 ## Alert service

docs/services/siembol_enrichment_service.md

@@ -1,4 +1,11 @@
 # Siembol Enrichment Service
+
+- [1. Overview](#overview)
+  * [1.1 Enrichment rule](#enrichment-rule)
+      * [1.1.1 Matchers](#matchers)
+      * [1.1.2 Table Mapping](#table-mapping)
+- [2. Admin config](#admin-config)
+
 ## Overview
 Siembol Enrichment is an enrichment engine used to add useful data to events to assist in detection and investigations.  
 
@@ -81,4 +88,4 @@ Note: you can only enrich from one table per rule. If you want to enrich the sam
 - `enriching.input.topics`- The list of kafka input topics for reading messages
 - `enriching.output.topic` - Output kafka topic name for correctly processed messages
 - `enriching.error.topic` - Output kafka topic name for error messages
-- `enriching.tables.hdfs.uri` - The url for hdfs cluster where enriching tables are stored
+- `enriching.tables.hdfs.uri` - The url for hdfs cluster where enriching tables are stored

docs/services/siembol_parsing_services.md

@@ -1,4 +1,31 @@
 # Siembol Parsing Services
+
+- [1. Overview](#overview)
+  * [1.1 Key concepts](#key-concepts)
+  * [1.2 Common fields](#common-fields)
+- [2. Parser config](#parser-config)
+  * [2.1 Parser Attributes](#parser-attributes)
+  * [2.2 Parser Extractors](#parser-extractors)
+      * [2.2.1 Overview](#overview-1)
+      * [2.2.2 Common extractor attributes](#common-extractor-attributes)
+      * [2.2.3 Pattern extractor](#pattern-extractor)
+      * [2.2.4 Key value Extractor](#key-value-extractor)
+      * [2.2.5 CSV Extractor](#csv-extractor)
+      * [2.2.6 Json Extractor](#json-extractor)
+      * [2.2.7 Json Path Extractor](#json-path-extractor)
+  * [2.3 Parser Transformations](#parser-transformations)
+      * [2.3.1 Overview](#overview-2)
+      * [2.3.2 Common fields](#common-fields-1)
+- [3. Parsing application](#parsing-application)
+  * [3.1 Overview](#overview-3)
+  * [3.2 Single Parser](#single-parser)
+  * [3.3 Router parsing](#router-parsing)
+  * [3.4 Topic routing parsing](#topic-routing-parsing)
+  * [3.5 Header routing parsing](#header-routing-parsing)
+- [4. Admin config](#admin-config)
+
+
+
 ## Overview
 Siembol provides parsing services for normalising logs into messages with one layer of key/value pairs. Clean normalised data is very important for further processing such as alerting.
 ### Key concepts

docs/services/siembol_response_service.md

@@ -1,4 +1,27 @@
 # Siembol Response Service
+
+- [1. Overview](#overview)
+- [2. Siembol Response Rule](#siembol-response-rule)
+  * [2.1 Evaluation](#evaluation)
+  * [2.2 Response Rule](#response-rule)
+  * [2.3 Provided evaluators](#provided-evaluators)
+      * [2.3.1 Fixed result](#fixed-result)
+      * [2.3.2 Matching](#matching)
+      * [2.3.3 Json path assignment](#json-path-assignment)
+      * [2.3.4 Markdown table formatter](#markdown-table-formatter)
+      * [2.3.5 Array markdown table formatter](#array-markdown-table-formatter)
+      * [2.3.6 Array reducer](#array-reducer)
+      * [2.3.7 Alert throttling](#alert-throttling)
+      * [2.3.8 Sleep](#sleep)
+      * [2.3.9 Kafka writer](#kafka-writer)
+      * [2.3.10 Time exclusion](#time-exclusion)
+- [3. Plugins](#plugins)
+  * [3.1 Plugin architecture](#plugin-architecture)
+  * [3.2 Evaluators - GR open source plans](#evaluators-\-\-gr-open-source-plans)
+- [4. Application Properties](#application-properties)
+  * [4.1 Authentication](#authentication)
+    * [4.1.1 Oauth2 Authentication](#oauth2-authentication)
+
 ## Overview
 Siembol response is a service for defining a response to an alert. It brings a functionality: 
 - To integrate siembol with other systems such as jira, ldap, elk, the hive, cortex etc.
@@ -126,7 +149,10 @@ Siembol response plugins allows to extend the functionality of siembol response
 ### Plugin architecture
 A Siembol response plugin is a shaded jar file that includes all its dependencies see [how to write response plugin](how-tos/how_to_write_response_plugin.md). The plugins can be copied in a directory where they will be loaded by the [springboot properties launcher](https://docs.spring.io/spring-boot/docs/current/reference/html/appendix-executable-jar-format.html) The plugin is also integrated into siembol UI and its evaluators can be used in the similar way as the ones provided directly by siembol response.
 
-### Evaluators implemented internally at GR that we are planning to open source
+###  Evaluators - GR open source plans
+
+Evaluators implemented internally at GR that are planned to become open source:
+
 - Elk search 
     - calling an Elastic Search query using LUCENE or json query syntax
 - Elk store