scottk/siembol
1
0
Fork 0
mirror of https://github.com/optim-enterprises-bv/siembol.git synced 2025-10-30 10:02:19 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
dependabot/maven/org.codehaus.mojo-versions-maven-plugin-2.16.0
siembol/docs/introduction/how-tos/quickstart.md
Yassin Raman d50e822b71 deployment: prometheus, grafana and new version of storm with siembol metrics (#692) 
* grafana and prometheus

* updated storm to latest

* grafana and prometheus

* grafana and prometheus

* grafana and prometheus

* grafana and prometheus

* version

* kafka lag exporter

* include kafka lag

* log level and change interval for scraping

* update to use values.yaml file

* fix

* added latency graphs

* added latency graphs, fix

* added monitoring to clean up

* using official docker image

* remove outdated comment

* fix, indentation

* clean up

* host names added

* rename and clean up

* rename and clean up

* tag

* jmx exporter cm added

* jmx exporter cm added

* jmx exporter volume mount

* jmx exporter install

* storm version

* add jmx to ps

* fix for changing port

* add wget to install

* not needed, we're using ingress-dns add on and configuring actual dns resolver

* fix

* fix post request

* add protection matches to dashboard

* separate traffic generator from main siembol install

* update

* update

* move file to correct loc

* typo

* fix

Co-authored-by: Yassin Raman <yassin@gmail.co>
Co-authored-by: yasram1 <yasram1@github.com>
Co-authored-by: Celie Valentiny <cvalentiny@hotmail.fr>
2022-07-14 11:59:27 +01:00

3.6 KiB
Raw Permalink Blame History

Quickstart Guide

Local Install

1. Run minikube.sh

deployment/quickstart_install/sh-scripts/minikube.sh

or

deployment/quickstart_install/ps-scripts/minikube.ps1

2. Install dependencies

1. Run dependencies.sh

deployment/quickstart_install/sh-scripts/dependencies.sh

or

deployment/quickstart_install/ps-scripts/dependencies.ps1

3. Prepare GitHub

1. Prepare Siembol Config Repository

  1. Go to https://github.com/G-Research/siembol-config
  2. Fork into your own organization or personal account

2. Create GitHub token

  1. Go to https://github.com/settings/tokens
  2. Click Generate new token
  3. Select "repo - Full control of private repositories" scope
  4. Hit "Generate token"
  5. Keep this token value as you will need it for the next step.

3. Run demoInstall.sh

  1. This will ask for your github details related to the Siembol Config repository and the token created in previous step.
deployment/quickstart_install/sh-scripts/demoInstall.sh

or

deployment/quickstart_install/ps-scripts/demoInstall.ps1

4. Siembol install

To install Siembol in the cluster

helm install siembol deployment/helm-k8s/ -f deployment/helm-k8s/values.yaml -n=siembol

This step might take a few minutes depending on the specs of your development machine.

Check it out!

In a browser, go to:

You should now see the Siembol UI homepage. You can also try Storm UI to see running topologies:

Enrichment tables

To add enrichment tables.

deployment/quickstart_install/sh-scripts/enrichmentStore.sh

or

deployment/quickstart_install/ps-scripts/enrichmentStore.ps1

Monitoring

We are scraping Siembol metrics using Prometheus and displaying the metrics in a Grafana dashboard.

To install these components:

  1. Download charts:
helm dependency update deployment/helm-k8s/monitoring-tools/
  1. Install the charts:
helm install monitoring deployment/helm-k8s/monitoring-tools/ -n=siembol

To see the dashboard:

You can also see the raw Prometheus metrics from:

Kafka UI

We are using Kafdrop UI to view topics and messages. You can send messages to parsing topics and to test siembol configs.

  1. To install this and create a kafka client pod:
deployment/quickstart_install/sh-scripts/kafkaExtra.sh

or

deployment/quickstart_install/ps-scripts/kafkaExtra.ps1
  1. Exec into the kafka client pod:
kubectl exec --tty -i kafka-client --namespace siembol -- bash
  1. Connect to the broker:
kafka-console-producer.sh \
--broker-list kafka-0.kafka-headless.siembol.svc.cluster.local:9092 \
--topic <your-topic>
  1. Produce your message in terminal window

Traffic Generator

We have a simple deployment which generates messages which Siembol processes. The Kafka topics, frequency and messages can be modified here

  1. To run this:
helm install traffic-generator deployment/helm-k8s/kafka-traffic-generator/ -n=siembol

Cleaning up

If you're done poking about on a local instance, you can clean up with:

  1. For cleaning up siembol resources and dependencies:
deployment/quickstart_install/sh-scripts/cleanUp.sh

or

deployment/quickstart_install/ps-scripts/cleanUp.ps1
  1. For deleting everything; delete the siembol minikube profile:
minikube delete -p siembol
sudo rm /etc/resolver/minikube-*