b492b1a23a
* Change CIDR from 172.15.0.0/16, which isn't a reserved private range * Use a smaller CIDR, /24 is sufficient
5.3 KiB
Torus Storage
The Torus example provisions a 3 node CoreOS cluster, with etcd3 and Torus, to demonstrate a stand-alone storage cluster. Each of the 3 nodes runs a Torus instance which makes 1GiB of space available (configured per node by "torus_storage_size" in machine group metadata).
Requirements
Ensure that you've gone through the bootcfg with rkt guide and understand the basics. In particular, you should be able to:
- Use rkt or Docker to start
bootcfg - Create a network boot environment with
coreos/dnsmasq - Create the example libvirt client VMs
/etc/hostsentries fornode[1-3].example.com(or pass custom names tok8s-certgen)- Install the Torus binaries
Examples
The examples statically assign IP addresses to libvirt client VMs created by scripts/libvirt. The examples can be used for physical machines if you update the MAC addresses. See network setup and deployment.
- torus - iPXE boot a Torus cluster
Assets
Download the CoreOS image assets referenced in the target profile.
./scripts/get-coreos stable 1185.3.0 ./examples/assets
Containers
Use rkt or docker to start bootcfg and mount torus example. Create a network boot environment and power-on your machines. Revisit bootcfg with rkt or bootcfg with Docker for help.
Client machines should network boot and provision themselves.
Verify
Install the Torus binaries on your laptop. Torus uses etcd3 for coordination and metadata storage, so any etcd node in the cluster can be queried with torusctl.
./torusctl --etcd node1.example.com:2379 list-peers
Run list-peers to report the status of data nodes in the Torus cluster.
+--------------------------------+--------------------------------------+---------+------+--------+---------------+--------------+
| ADDRESS | UUID | SIZE | USED | MEMBER | UPDATED | REB/REP DATA |
+--------------------------------+--------------------------------------+---------+------+--------+---------------+--------------+
| http://node1.example.com:40000 | 67145622-52cb-11e6-a886-525400a19cae | 1.0 GiB | 0 B | OK | 2 seconds ago | 0 B/sec |
| http://node2.example.com:40000 | 6978182a-52cb-11e6-b41d-525400b22f86 | 1.0 GiB | 0 B | OK | 3 seconds ago | 0 B/sec |
| http://node3.example.com:40000 | 6e0e4d7d-52cb-11e6-af25-525400c36177 | 1.0 GiB | 0 B | OK | now | 0 B/sec |
+--------------------------------+--------------------------------------+---------+------+--------+---------------+--------------+
Torus has already initialized its metadata within etcd3 to format the cluster and added all peers to the pool. Each node provides 1 GiB of storage and has MEMBER status OK.
Volume Creation
Create a new replicated, virtual block device or volume on Torus.
./torusctl --etcd=node1.example.com:2379 block create hello 500MiB
List the current volumes,
./torusctl --etcd=node1.example.com:2379 volume list
and verify that hello was created.
+-------------+---------+
| VOLUME NAME | SIZE |
+-------------+---------+
| hello | 500 MiB |
+-------------+---------+
Filesystems and Mounting
Let's attach the Torus volume, create a filesystem, and add some files. Add the nbd kernel module.
sudo modprobe nbd
sudo ./torusblk --etcd=node1.example.com:2379 nbd hello
In a new shell, create a new filesystem on the volume and mount it on your system.
sudo mkfs.ext4 /dev/nbd0
sudo mkdir -p /mnt/hello
sudo mount /dev/nbd0 -o discard,noatime /mnt/hello
Check that the mounted filesystem is present.
$ mount | grep nbd
/dev/nbd0 on /mnt/hello type ext4 (rw,noatime,seclabel,discard,data=ordered)
Simulate Failure
By default, Torus uses a replication factor of 2. You may write some data and poweroff one of the three nodes if you wish.
sudo sh -c "echo 'hello world' > /mnt/hello/world"
sudo virsh destroy node3 # actually equivalent to poweroff
Check the Torus data nodes.
$ ./torusctl --etcd node1.example.com:2379 list-peers
+--------------------------+--------------------------------------+---------+--------+--------+---------------+--------------+
| ADDRESS | UUID | SIZE | USED | MEMBER | UPDATED | REB/REP DATA |
+--------------------------+--------------------------------------+---------+--------+--------+---------------+--------------+
| http://node1.example.com:40000 | 016fad6a-2e23-11e6-8ced-525400a19cae | 1.0 GiB | 22 MiB | OK | 3 seconds ago | 0 B/sec |
| http://node2.example.com:40000 | 0c67d31c-2e23-11e6-91f5-525400b22f86 | 1.0 GiB | 22 MiB | OK | 3 seconds ago | 0 B/sec |
| | 0408cbba-2e23-11e6-9871-525400c36177 | ??? | ??? | DOWN | Missing | |
+--------------------------+--------------------------------------+---------+--------+--------+---------------+--------------+
Balanced: true Usage: 2.15%
Going Further
See the Torus project to learn more about Torus and contribute.