CoreOS cluster on KVM

Ansible playbook to create a cluster of CoreOS servers running on KVM virtual machines.

Motivation

CoreOS can be installed on various platforms hosted by cloud providers or on your computer. In the latter case Vagrant is a popular method to Start a cluster using VirtualBox.

This playbook was created for two reasons:

• Provision cluster with another popular hypervisor -- KVM
• Enable custom options in the provisioning process, particularly to use LVM partitions for the VM images
• Post-installation configuration to setup etcd instances on all VMs

Quick start

Creating the cluster is as easy as choosing the desired number of machines in the cluster and running the playbook. It will download stable CoreOS image from the official site, create N (default 3) KVM virtual machines and boot up the cluster.

$ git clone https://github.com/misho-kr/coreos-cluster-kvm
Cloning into 'coreos-cluster-kvm'...
$ cd coreos-cluster-kvm
$ ansible-playbook create.yml -i hosts --ask-become-pass

Obviously you need to have Ansible on your machine in order to run the playbook. Follow the instuctions in the Ansible documentation to install it from distro repositories, from source or with pip.

While the playbook does its thing you can read the next sections to understand how to manage the cluster after it is created, and the available configurations options for your next cluster.

Cluster management

There are several Ansible scripts to manage the cluster after it was created.

Start up, Shutdown, Restart

$ ansible-playbook -i hosts admin.yml -t start
...

$ ansible-playbook -i hosts admin.yml -t shutdown
...

$ ansible-playbook -i hosts admin.yml -t restart
...

Destroy the cluster

$ ansible-playbook -i hosts destroy.yml
...

Create

These playbook variables that can be used to customize the provisioned cluster:

• vm_count (default=3) -- number of virtual machines in the cluster
• vg_name -- use this volume group to create logical volumes for vm images; if not defined then use plain filesystem
• cloud_init_profile
• basic (default) -- minimum setup, set hostname and upload ssh keys
• etcd_fleet -- etcd and fleet services are started on each VM
• discovery_token -- pass etcd discovery token for use by the etcd cluster, if not provided one will be generated from https://discovery.etcd.io/new
• use_discovery_token (default=yes) -- discovery_token will be used to find all neighbors in the cluster
• force_download -- fresh vm image will be downloaded even if one exists in the download folder

The variables can be passed to the playbook at the command line, or be set permanently in the main configuration file group_vars/all.yml. For example thi command will create cluster with 9 virtual machines on LV group named "lv_coreos":

$ ansible-playbook create.yml -i hosts -e vm_count=9 -e vg_name=lv_coreos

Note: if you choose non-default number of VMs, open group_vars/all.yml and update the vm_count variable. This will avoid the need to pass vm_count argument to the playbook every time you run create or other management command.

Repeated execution

This option is not that useful anymore, but it is handy when troubleshooting some problem.

The playbook consists of several roles which are like seperate steps of one big procedure to provision the cluster. Each role has a tag which allows to skip it or select it for exclusive execution. For example to skip the creation of cloud-init file for each virtual machine, request that roles and tasks with init tag be skipped:

$ ansible-playbook -i hosts create.yml -e "vm_count=5" --skip-tags=init
...

Features

The goal of the playbook was to automate many of the cluster provisioning tasks:

• Virtual machines can be created on plain filesystem or on Logical Volume Manager logical volume
• etcd cluster can be initialized via either
• discovery token
• explicitly setting the peer hostnames
• If using etcd discovery token, new one will be acquired every time new cluster is provisioned (tokens can be reused)
• CoreOS image is downloaded only the first time the playbook runs. The cached image will be reused until it is deleted, or if explicitly requested with the force_download option.

TODO list

• At the end of provisioning the cluster, create an inventory file with all CoreOS virtual machines listed in it
• Implement a method to specify the virtual machines by name in the inventory file
• Currently the virtual machines are given uninspiring names like coreos-1, coreos-2, etc.
• The prefix coreos should be configurable parameter
• Even better if user can prepare an inventory file with virtial machines names, and use that to provision the cluster
• Fedora was used to develop the playbook
• Test the playbook on other platforms -- CentOS, Ubuntu, etc.

Issues and workarounds

• Can not provision cluster of size 1, i.e. the variable vm_count should be at least 2
• After a cluster is provisioned it has to be restarted in order for each VM to properly acquire the domain name from the DHCP server (this is handled by the playbook, no need for manual action)