mirror of
https://github.com/optim-enterprises-bv/kubernetes.git
synced 2025-11-03 19:58:17 +00:00
305 lines
11 KiB
Markdown
305 lines
11 KiB
Markdown
## Getting started with Kubernetes on Mesos
|
|
|
|
<!-- TODO: Update, clean up. -->
|
|
|
|
Mesos allows dynamic sharing of cluster resources between Kubernetes and other first-class Mesos frameworks such as [Hadoop][1], [Spark][2], and [Chronos][3].
|
|
Mesos ensures applications from different frameworks running on your cluster are isolated and that resources are allocated fairly.
|
|
|
|
Running Kubernetes on Mesos allows you to easily move Kubernetes workloads from one cloud provider to another to your own physical datacenter.
|
|
|
|
This tutorial will walk you through setting up Kubernetes on a Mesos cluster on [Google Cloud Plaform][4].
|
|
It provides a step by step walk through of adding Kubernetes to a Mesos cluster and running the classic GuestBook demo application.
|
|
The walkthrough presented here is based on the v0.4.x series of the Kubernetes-Mesos project, which itself is based on Kubernetes v0.11.0.
|
|
|
|
### Prerequisites
|
|
|
|
* Understanding of [Apache Mesos][11]
|
|
* Mesos cluster on [Google Compute Engine][5]
|
|
* Identify the Mesos master node external IP from Mesosphere [cluster launch pad][12]
|
|
* A [VPN connection to the cluster][6].
|
|
|
|
### Deploy Kubernetes-Mesos
|
|
|
|
Log into the master node over SSH, replacing the placeholder below with the correct IP address.
|
|
|
|
```bash
|
|
ssh jclouds@${ip_address_of_master_node}
|
|
```
|
|
|
|
Build Kubernetes-Mesos.
|
|
|
|
```bash
|
|
$ git clone https://github.com/mesosphere/kubernetes-mesos k8sm
|
|
$ mkdir -p bin && sudo docker run --rm -v $(pwd)/bin:/target \
|
|
-v $(pwd)/k8sm:/snapshot -e GIT_BRANCH=release-0.4 \
|
|
mesosphere/kubernetes-mesos:build
|
|
```
|
|
|
|
Set some environment variables.
|
|
The internal IP address of the master is visible via the cluster details page on the Mesosphere launchpad, or may be obtained via `hostname -i`.
|
|
|
|
```bash
|
|
$ export servicehost=$(hostname -i)
|
|
$ export mesos_master=${servicehost}:5050
|
|
$ export KUBERNETES_MASTER=http://${servicehost}:8888
|
|
```
|
|
### Deploy etcd
|
|
Start etcd and verify that it is running:
|
|
|
|
```bash
|
|
$ sudo docker run -d --hostname $(uname -n) --name etcd -p 4001:4001 -p 7001:7001 coreos/etcd
|
|
```
|
|
|
|
```bash
|
|
$ sudo docker ps
|
|
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
|
|
fd7bac9e2301 coreos/etcd:latest "/etcd" 5s ago Up 3s 2379/tcp, 2380/... etcd
|
|
```
|
|
It's also a good idea to ensure your etcd instance is reachable by testing it
|
|
```bash
|
|
curl -L http://$servicehost:4001/v2/keys/
|
|
```
|
|
If connectivity is OK, you will see an output of the available keys in etcd (if any).
|
|
|
|
### Start Kubernetes-Mesos Services
|
|
Start the kubernetes-mesos API server, controller manager, and scheduler on a Mesos master node:
|
|
|
|
```bash
|
|
$ ./bin/km apiserver \
|
|
--address=${servicehost} \
|
|
--mesos_master=${mesos_master} \
|
|
--etcd_servers=http://${servicehost}:4001 \
|
|
--portal_net=10.10.10.0/24 \
|
|
--port=8888 \
|
|
--cloud_provider=mesos \
|
|
--v=1 >apiserver.log 2>&1 &
|
|
|
|
$ ./bin/km controller-manager \
|
|
--master=$servicehost:8888 \
|
|
--mesos_master=${mesos_master} \
|
|
--v=1 >controller.log 2>&1 &
|
|
|
|
$ ./bin/km scheduler \
|
|
--address=${servicehost} \
|
|
--mesos_master=${mesos_master} \
|
|
--etcd_servers=http://${servicehost}:4001 \
|
|
--mesos_user=root \
|
|
--api_servers=$servicehost:8888 \
|
|
--v=2 >scheduler.log 2>&1 &
|
|
```
|
|
|
|
Also on the master node, we'll start up a proxy instance to act as a
|
|
public-facing service router, for testing the web interface a little
|
|
later on.
|
|
|
|
```bash
|
|
$ sudo ./bin/km proxy \
|
|
--bind_address=${servicehost} \
|
|
--etcd_servers=http://${servicehost}:4001 \
|
|
--logtostderr=true >proxy.log 2>&1 &
|
|
```
|
|
|
|
Disown your background jobs so that they'll stay running if you log out.
|
|
|
|
```bash
|
|
$ disown -a
|
|
```
|
|
#### Validate KM Services
|
|
Interact with the kubernetes-mesos framework via `kubectl`:
|
|
|
|
```bash
|
|
$ bin/kubectl get pods
|
|
POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS
|
|
```
|
|
|
|
```bash
|
|
$ bin/kubectl get services # your service IPs will likely differ
|
|
NAME LABELS SELECTOR IP PORT
|
|
kubernetes component=apiserver,provider=kubernetes <none> 10.10.10.2 443
|
|
kubernetes-ro component=apiserver,provider=kubernetes <none> 10.10.10.1 80
|
|
```
|
|
Lastly, use the Mesos CLI tool to validate the Kubernetes scheduler framework has been registered and running:
|
|
```bash
|
|
$ mesos state | grep "Kubernetes"
|
|
"name": "Kubernetes",
|
|
```
|
|
Or, look for Kubernetes in the Mesos web GUI. You can get there by clicking the
|
|
Mesos logo on the Mesosphere launchpad page, or by pointing your browser to
|
|
`http://${mesos_master}`. Make sure you have an active [VPN connection][6].
|
|
Go to the Frameworks tab, and look for an active framework named "Kubernetes".
|
|
|
|
## Spin up a pod
|
|
|
|
Write a JSON pod description to a local file:
|
|
|
|
```bash
|
|
$ cat <<EOPOD >nginx.json
|
|
{ "kind": "Pod",
|
|
"apiVersion": "v1beta1",
|
|
"id": "nginx-id-01",
|
|
"desiredState": {
|
|
"manifest": {
|
|
"version": "v1beta1",
|
|
"containers": [{
|
|
"name": "nginx-01",
|
|
"image": "nginx",
|
|
"ports": [{
|
|
"containerPort": 80,
|
|
"hostPort": 31000
|
|
}],
|
|
"livenessProbe": {
|
|
"enabled": true,
|
|
"type": "http",
|
|
"initialDelaySeconds": 30,
|
|
"httpGet": {
|
|
"path": "/index.html",
|
|
"port": "8081"
|
|
}
|
|
}
|
|
}]
|
|
}
|
|
},
|
|
"labels": {
|
|
"name": "foo"
|
|
} }
|
|
EOPOD
|
|
```
|
|
|
|
Send the pod description to Kubernetes using the `kubectl` CLI:
|
|
|
|
```bash
|
|
$ bin/kubectl create -f nginx.json
|
|
nginx-id-01
|
|
```
|
|
|
|
Wait a minute or two while `dockerd` downloads the image layers from the internet.
|
|
We can use the `kubectl` interface to monitor the status of our pod:
|
|
|
|
```bash
|
|
$ bin/kubectl get pods
|
|
POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS
|
|
nginx-id-01 172.17.5.27 nginx-01 nginx 10.72.72.178/10.72.72.178 cluster=gce,name=foo Running
|
|
```
|
|
|
|
Verify that the pod task is running in the Mesos web GUI. Click on the
|
|
Kubernetes framework. The next screen should show the running Mesos task that
|
|
started the Kubernetes pod.
|
|
|
|
## Run the Example Guestbook App
|
|
|
|
Following the instructions from the kubernetes-mesos [examples/guestbook][7]:
|
|
|
|
```bash
|
|
$ export ex=k8sm/examples/guestbook
|
|
$ bin/kubectl create -f $ex/redis-master.json
|
|
$ bin/kubectl create -f $ex/redis-master-service.json
|
|
$ bin/kubectl create -f $ex/redis-slave-controller.json
|
|
$ bin/kubectl create -f $ex/redis-slave-service.json
|
|
$ bin/kubectl create -f $ex/frontend-controller.json
|
|
|
|
$ cat <<EOS >/tmp/frontend-service
|
|
{
|
|
"id": "frontend",
|
|
"kind": "Service",
|
|
"apiVersion": "v1beta1",
|
|
"port": 9998,
|
|
"selector": {
|
|
"name": "frontend"
|
|
},
|
|
"publicIPs": [
|
|
"${servicehost}"
|
|
]
|
|
}
|
|
EOS
|
|
$ bin/kubectl create -f /tmp/frontend-service
|
|
```
|
|
|
|
Watch your pods transition from `Pending` to `Running`:
|
|
|
|
```bash
|
|
$ watch 'bin/kubectl get pods'
|
|
```
|
|
|
|
Review your Mesos cluster's tasks:
|
|
|
|
```bash
|
|
$ mesos ps
|
|
TIME STATE RSS CPU %MEM COMMAND USER ID
|
|
0:00:05 R 41.25 MB 0.5 64.45 none root 0597e78b-d826-11e4-9162-42010acb46e2
|
|
0:00:08 R 41.58 MB 0.5 64.97 none root 0595b321-d826-11e4-9162-42010acb46e2
|
|
0:00:10 R 41.93 MB 0.75 65.51 none root ff8fff87-d825-11e4-9162-42010acb46e2
|
|
0:00:10 R 41.93 MB 0.75 65.51 none root 0597fa32-d826-11e4-9162-42010acb46e2
|
|
0:00:05 R 41.25 MB 0.5 64.45 none root ff8e01f9-d825-11e4-9162-42010acb46e2
|
|
0:00:10 R 41.93 MB 0.75 65.51 none root fa1da063-d825-11e4-9162-42010acb46e2
|
|
0:00:08 R 41.58 MB 0.5 64.97 none root b9b2e0b2-d825-11e4-9162-42010acb46e2
|
|
```
|
|
The number of Kubernetes pods listed earlier (from `bin/kubectl get pods`) should equal to the number active Mesos tasks listed the previous listing (`mesos ps`).
|
|
|
|
Next, determine the internal IP address of the front end [service portal][8]:
|
|
|
|
```bash
|
|
$ bin/kubectl get services
|
|
NAME LABELS SELECTOR IP PORT
|
|
kubernetes component=apiserver,provider=kubernetes <none> 10.10.10.2 443
|
|
kubernetes-ro component=apiserver,provider=kubernetes <none> 10.10.10.1 80
|
|
redismaster <none> name=redis-master 10.10.10.49 10000
|
|
redisslave name=redisslave name=redisslave 10.10.10.109 10001
|
|
frontend <none> name=frontend 10.10.10.149 9998
|
|
```
|
|
|
|
Interact with the frontend application via curl using the front-end service IP address from above:
|
|
|
|
```bash
|
|
$ curl http://${frontend_service_ip_address}:9998/index.php?cmd=get\&key=messages
|
|
{"data": ""}
|
|
```
|
|
|
|
Or via the Redis CLI:
|
|
|
|
```bash
|
|
$ sudo apt-get install redis-tools
|
|
$ redis-cli -h ${redis_master_service_ip_address} -p 10000
|
|
10.233.254.108:10000> dump messages
|
|
"\x00\x06,world\x06\x00\xc9\x82\x8eHj\xe5\xd1\x12"
|
|
```
|
|
#### Test Guestbook App
|
|
Or interact with the frontend application via your browser, in 2 steps:
|
|
|
|
First, open the firewall on the master machine.
|
|
|
|
```bash
|
|
# determine the internal port for the frontend service portal
|
|
$ sudo iptables-save|grep -e frontend # -- port 36336 in this case
|
|
-A KUBE-PORTALS-CONTAINER -d 10.10.10.149/32 -p tcp -m comment --comment frontend -m tcp --dport 9998 -j DNAT --to-destination 10.22.183.23:36336
|
|
-A KUBE-PORTALS-CONTAINER -d 10.22.183.23/32 -p tcp -m comment --comment frontend -m tcp --dport 9998 -j DNAT --to-destination 10.22.183.23:36336
|
|
-A KUBE-PORTALS-HOST -d 10.10.10.149/32 -p tcp -m comment --comment frontend -m tcp --dport 9998 -j DNAT --to-destination 10.22.183.23:36336
|
|
-A KUBE-PORTALS-HOST -d 10.22.183.23/32 -p tcp -m comment --comment frontend -m tcp --dport 9998 -j DNAT --to-destination 10.22.183.23:36336
|
|
|
|
# open up access to the internal port for the frontend service portal
|
|
$ sudo iptables -A INPUT -i eth0 -p tcp -m state --state NEW,ESTABLISHED -m tcp \
|
|
--dport ${internal_frontend_service_port} -j ACCEPT
|
|
```
|
|
|
|
Next, add a firewall rule in the Google Cloud Platform Console. Choose Compute >
|
|
Compute Engine > Networks, click on the name of your mesosphere-* network, then
|
|
click "New firewall rule" and allow access to TCP port 9998.
|
|
|
|
![Google Cloud Platform firewall configuration][9]
|
|
|
|
Now, you can visit the guestbook in your browser!
|
|
|
|
![Kubernetes Guestbook app running on Mesos][10]
|
|
|
|
[1]: http://mesosphere.com/docs/tutorials/run-hadoop-on-mesos-using-installer
|
|
[2]: http://mesosphere.com/docs/tutorials/run-spark-on-mesos
|
|
[3]: http://mesosphere.com/docs/tutorials/run-chronos-on-mesos
|
|
[4]: http://cloud.google.com
|
|
[5]: https://google.mesosphere.com
|
|
[6]: http://mesosphere.com/docs/getting-started/cloud/google/mesosphere/#vpn-setup
|
|
[7]: https://github.com/mesosphere/kubernetes-mesos/tree/v0.4.0/examples/guestbook
|
|
[8]: https://github.com/GoogleCloudPlatform/kubernetes/blob/v0.11.0/docs/services.md#ips-and-portals
|
|
[9]: mesos/k8s-firewall.png
|
|
[10]: mesos/k8s-guestbook.png
|
|
[11]: http://mesos.apache.org/
|
|
[12]: https://google.mesosphere.com/clusters
|