Graphing tends to be undervalued in organisations. People often overlook the significance of quick access to historical box performance statistics. They realise the need for these sorts of tools only after production incidents, and they cannot get to the information they need. Instead, they rely on tools that provide frequent but not-quite-detailed-enough statistics, such as sar or atop.
Good, easy-to-read graphing, and people actively monitoring the information being logged can often reveal early signs of issues, such as a number of incorrect response codes returned by a web server suddenly spiking. It can also serve as an early warning for attacks on your systems. For example, when graphing connection counts or bandwidth on a firewall, a sudden spike could indicate that someone is “testing” your infrastructure for a DDOS, or starting an attack against you.
There are a number of tools that can be used for performing this function, such as Graphite. However, my favorite approach is using Grafana with a time series database like InfluxDB behind it.
Installation and configuration of InfluxDB
For the purposes of this article, I will assume that you are working from a CentOS 7 x64 server, which is fully up-to-date. Note also that all commands below should be run as root in the terminal.
The first thing we are going to do on our new server is add the repository for InfluxDB. This is simple, and can be done using the command below:
cat > /etc/yum.repos.d/influxdb.repo [influxdb] name = InfluxDB Repository - RHEL \$releasever baseurl = https://repos.influxdata.com/rhel/\$releasever/\$basearch/stable enabled = 1 gpgcheck = 1 gpgkey = https://repos.influxdata.com/influxdb.key EOF
Once we have added in our repository, we can go ahead and install and start InfluxDB.
yum -y install influxdb && systemctl enable influxdb && systemctl start influxdb
Once we have installed and started influxdb, we need to generate a config file. This can be done by running
influxd config > /etc/influxdb/influxdb.generated.conf
For the purpose of this guide, the default configuration values are fine. However, if you want to change variables such as where data is stored on the filesystem, bind address or port, you can do so using the configuration options outlined on the InfluxDB website.
Once you have customised the configuration file to your needs, you need to tell InfluxDB to use the new configuration file. This can be done by modifying /usr/lib/systemd/system/influxdb.service and adjusting the ExecStart line to point at your new configuration file.
Once this has been done, we need to reload the systemctl daemon and restart the InfluxDB service.
systemctl daemon-reload && systemctl restart influxdb
We now have influxdb configured and ready to go. The last thing we will do is connect to InfluxDB over the CLI interface and create ourselves a database, which we will use later for collectd.
influx Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring. Connected to http://localhost:8086 version 1.0.2 InfluxDB shell version: 1.0.2
Next, we will create our database for collectd.
< create database collectd > show databases name: databases --------------- name _internal collectd >
We have now created and listed the database, and can see our newly created database.
Installation and configuration of Grafana
We are now going to set up Grafana. This can be done on either a different server or on the same server. For the purpose of this guide, we are going to do this on the local server.
To install Grafana, we first need to add another repository for our system using the command below:
cat << EOF > /etc/yum.repos.d/grafana.repo [grafana] name=grafana baseurl=https://packagecloud.io/grafana/stable/el/7/\$basearch repo_gpgcheck=1 enabled=1 gpgcheck=1 gpgkey=https://packagecloud.io/gpg.key https://grafanarel.s3.amazonaws.com/RPM-GPG-KEY-grafana sslverify=1 sslcacert=/etc/pki/tls/certs/ca-bundle.crt EOF
We then want to install Grafana:
yum install grafana
And enable it to start at boot:
systemctl enable grafana-server
Finally, start it now (so we don’t have to wait for a reboot to launch it):
systemctl start grafana-server
Again, for the purpose of this guide, we will use the default settings. However, Grafana settings can be changed in the following two locations:
/etc/sysconfig/grafana-server /etc/grafana/grafana.ini
A full guide of configuration options for Grafana can be found on their website, here.
We now need to add firewall rules so we can access Grafana on its standard port (3000). This can be done by running the following.
firewall-cmd --zone=public --add-port=3000/tcp --permanent firewall-cmd --reload
We can now browse to Grafana’s web interface on http://<ip/host>:3000 and log in with admin/admin (the default credentials—which you should change, of course, for a production scenario).
Next, we need to add InfluxDB as a datasource in Grafana. We do this by logging into Grafana using the URL and credentials above, clicking the Grafana logo at the top left, and then selecting data sources from the dropdown. Then select “Add new data source.”
You should be greeted with a screen similar to the one above. Now we need to fill in some fields to get connectivity working.
Name: local influxdb (Can be whatever you like)
Type: InfluxDB
Access: proxy (Proxy means Grafana will talk to your backend datasource, direct will mean your client will talk to the backend data source—as we have not opened up the ports to the InfluxDB backed, we are using proxy here).
Http Auth: Basic Auth should be checked here.
User: root (the defaults)
Password: root (the defaults)
Database: collectd
User: root (the defaults)
Password: root (the defaults)
You can now click “add”, which will save and test the configuration. You should get a confirmation that the datasource is working.
We now need to get some data into InfluxDB to graph.
Getting our first data in using collectd
We now need to install collectd on our server. This can be done by running:
yum -y install epel-release yum -y install collectd
Once installed, we need to open up the collectd config file and tell it to write stats out to InfluxDB. To do that, open up /etc/collectd.conf and uncomment the following lines:
LoadPlugin network Server "ff18::efc0:4a42" "25826" While we are here we might as well update the server IP and port to our ones for InfluxDB. In our case, this is 127.0.0.1 and the default port. We can now go ahead and enable/start collectd:
systemctl enable collectd systemctl start collectd
Once enabled and started, we need to make InfluxDB listen on the correct port to receive this data. To do this, make the following changes in /etc/influxdb/influxdb.generated.conf:
Under [[collectd]] enabled = true Now restart InfluxDB: systemctl restart influxdb
We can now head back to Grafana in our web browser, click the dropdown which says “Home” at the top and create a new dashboard by clicking “Create New” at the bottom of the screen:
Next, on the left-hand side, click the tiny green indent on the page adn go to “Add Panel” > “Graph”.
It will then ask you to specify a datasource, which we added earlier, and a query, as in the following screenshot:
As you can see, we have defined our query and are now getting our load average graphed.
You can read more about the query language here.
Final Thoughts
We now have collectd, InfluxDB and Grafana set up and receiving metrics.
This has been a very quick run through to get you set up. You will need to play around with all aspects of the configuration to gain confidence in using the products. However, there are many good documentation resources available at the following websites:
I am also happy to answer any questions that you leave in the comments below.
