In an ideal world, we’d all love our infrastructure to be adaptive and self-healing. What better than to sit back and watch the infrastructure expand to handle additional load and contract, without any human interaction? The lesser the operator dependence, the fewer chances of operator error on a production system. This and the next couple of posts will try and cover what’s needed to build that kind of infrastructure with off-the-shelf tools, Docker Swarm and a little bit of Python code.
Swarm or Kubernetes?
Docker Swarm is the orchestration tool from Docker that comes with Docker Engine and is available with both the Community Edition (CE) and Enterprise Edition (EE) versions of Docker Engine. For a while, Swarm was the only orchestrator that came installed. Then Kubernetes was added to the mix. While both Swarm and Kubernetes aim to simplify container orchestration, their design philosophies are different. Swarm is part of the Docker daemon and requires no external containers to run. Kubernetes, on the other hand, makes no assumptions about the runtime environment and ships its engine as a set of containers that work together.
Both Docker Swarm and Kubernetes allow for aggregating many machines into a cluster and scheduling and managing services on the cluster as a whole. While this abstraction takes away much complexity from the container management workflow, it introduces other issues. Unlike Kubernetes, Swarm doesn’t come with a dashboard. Neither does it monitor node health and resource consumption. But all is not lost. In the process of learning how to build a scalable swarm cluster that can scale up and down depending on resource utilisation, we’ll also look at how to monitor the cluster resources as well as visualise how services and containers deploy across the cluster. We shall not be looking at Kubernetes in this series of posts, but it shouldn’t be hard to adapt this to Kubernetes as well.
Auto Scaling and Microscaling
Scaling infrastructure horizontally scales machines as well as containers that comprise various services in the cluster. Here’s what we plan to achieve in the end:
We see a swarm cluster composed of a number of nodes. We also see a couple of services deployed on the cluster. The idea is that the number of service instances will scale according to incoming load from clients. Of course, this scale can’t be infinite. Beyond a certain point, the cluster will saturate, and we’ll need to add more nodes to handle the load. So we are scaling at two places—one at the service level, based on incoming requests from clients, and one at the node level where we add and remove nodes based on nodes’ resource consumption.
To achieve this model of deployment, let’s look at what we’ll need to do. To scale nodes, we only need to be able to look at resource consumption (CPU load, memory consumption, etc.) of all the nodes. If it exceeds a set threshold, we add a machine. Conversely, if we find a node’s resource consumption below a certain threshold for long enough, we can terminate the machine. This will cause Docker Swarm to re-distribute the containers that were running on that node to the remaining nodes in the cluster.
To make the decision about expanding or contracting the cluster, we need to be able to collect metrics for all the machines in the cluster. If we were implementing this on AWS, we might simply set up the cluster as an Auto Scaling group and let AWS’ Cloudwatch monitor the cluster metrics, and let AWS expand or contract the group. This is relatively straightforward. But it might be necessary to get more fine-grained insight into per-service resource consumption, and that is what we shall cover in this post. In the next couple of posts, we shall also look at a way to scale services by request load.
One of the prerequisites for this model to work is that the services we expect to scale need to be stateless, or any state stored should be outside of the service container. Otherwise, when we expand and shrink the service scale (number of running containers for a given service), we will lose the state of the container being destroyed.
So let’s jump right in. Swarm supports two kinds of services. The first is a scalable service that you can control using a scale count. If you ask Swarm to scale a service to three instances, it will make sure that three instances of the container are running in the cluster. The second kind of service Swarm supports is called a global service. Launching a service as global causes Swarm to ensure that exactly one instance of the service runs on each node in the cluster. A global metrics collection service is therefore a perfect way to gather machine and container metrics from each node. If we can consolidate all of the collected data at one place and make decisions based on that, we should be all set.
Monitoring with TICK
Enter the TICK platform. TICK (Telegraf-Influxdb-Chronograf-Kapacitor) is a time series platform from InfluxData. The components in the stack allow for collecting metrics, sending them to a time-series database and graphing and processing the collected metrics. It’s relatively lightweight, is written in Go and runs nicely in Docker.
Here’s what the TICK stack looks like:
Telegraf is a small lightweight agent that runs on the monitored machine. It collects metrics and events from all kinds of sources and sends them to InfluxDB and Kapacitor. InfluxDB is a purpose-built time-series database. It’s the heart of the TICK stack and allows for data to be queried and graphed. Chronograf is the UI for the TICK stack. It allows for creating custom graphs and dashboards from data in InfluxDB. Kapacitor is the data-processing engine and is used for automatic alerting and notification, etc.
This is what our monitoring stack is going to look like when it’s deployed in Swarm:
All the code described in this article is available here. Please download and unzip the code somewhere.
The stack definition is in a file named TICK-stack.yml. A quick glance at TICK-stack.yml reveals that all containers with state are launched with a constraint set to master. This is to ensure that the stateful services always come up on the same machine and maintain their state on the tagged node. The state in this case is in the form of persistent volumes to store all the data. This includes InfluxDB, Kapacitor and Chronograf data. This is all fine it you have only one master, but if you have a larger cluster or more than one master, it’s recommended that you tag one of the master nodes as your state store node and set the constraints in the YAML file accordingly.
In a terminal or command prompt, change directories to the unzipped location and launch the TICK stack like so:
docker stack deploy -c TICK-stack.yml tick
This should spit out something like this:
Creating network tick_influx Creating secret tick_kapacitor.conf Creating secret tick_telegraf.conf Creating secret tick_influxdb.conf Creating service tick_chronograf Creating service tick_kapacitor Creating service tick_telegraf Creating service tick_influxdb
This launches a stack named tick. Wait for a while for the services in the stack to come up. The images are fairly small, so it shouldn’t take too much time. A docker service ls should show the current state of the tick stack. After all services start running, point your browser to one of the Swarm machines on port 8888. If you are running it on your local machine, then point to http://localhost:8888. That should bring up the Chronograf UI.
Feel free to browse around and explore the UI. While it’s relatively easy to create dashboards, there’s no easy way to save and restore dashboards. This makes it difficult to create and share dashboards among colleagues and friends. Fortunately, there’s a simple enough workaround for it. There’s a Python script in the unzipped code directory to manage Chronograf dashboards called dashboard.py. You will need a Python interpreter installed to run the code. To load a pre-created dashboard into Chronograf, do:
python dashboard.py load < dashboard.json
Browse to the dashboard tab on the left and select the Swarm dashboard. You should see something like this:
That’s it. You can expand on the dashboard and add metrics of interest, or create a new dashboard if you like. You can filter hosts by using the host filter at the top of the dashboard.
We haven’t explored how to use Kapacitor to configure and set up alerts, and this series of posts won’t cover that. Instead, we’ll focus on how to use the TICK stack to create scalable services that react to service load in real time. In the next post, we shall explore how to set up docker-flow-proxy to allow for automatic service discovery and routing, and how to track service request load using docker-flow-proxy and the TICK stack.
