Grafana and InfluxDB allowing easy to read graphing

Getting Started With Grafana and InfluxDB

17948 VIEWS

· · · ·

Graphing tends to be undervalued in organisations. People often overlook the significance of quick access to historical box performance statistics. They realise the need for these sorts of tools only after production incidents, and they cannot get to the information they need. Instead, they rely on tools that provide frequent but not-quite-detailed-enough statistics, such as sar or atop.

Good, easy-to-read graphing, and people actively monitoring the information being logged can often reveal early signs of issues, such as a number of incorrect response codes returned by a web server suddenly spiking. It can also serve as an early warning for attacks on your systems. For example, when graphing connection counts or bandwidth on a firewall, a sudden spike could indicate that someone is “testing” your infrastructure for a DDOS, or starting an attack against you.

There are a number of tools that can be used for performing this function, such as Graphite. However, my favorite approach is using Grafana with a time series database like InfluxDB behind it.

Installation and configuration of InfluxDB

For the purposes of this article, I will assume that you are working from a CentOS 7 x64 server, which is fully up-to-date. Note also that all commands below should be run as root in the terminal.

The first thing we are going to do on our new server is add the repository for InfluxDB. This is simple, and can be done using the command below:

cat > /etc/yum.repos.d/influxdb.repo

[influxdb]

name = InfluxDB Repository - RHEL \$releasever

baseurl = https://repos.influxdata.com/rhel/\$releasever/\$basearch/stable

enabled = 1

gpgcheck = 1

gpgkey = https://repos.influxdata.com/influxdb.key

EOF

Once we have added in our repository, we can go ahead and install and start InfluxDB.

yum -y install influxdb && systemctl enable influxdb && systemctl start influxdb

Once we have installed and started influxdb, we need to generate a config file. This can be done by running

influxd config > /etc/influxdb/influxdb.generated.conf

For the purpose of this guide, the default configuration values are fine. However, if you want to change variables such as where data is stored on the filesystem, bind address or port, you can do so using the configuration options outlined on the InfluxDB website.

Once you have customised the configuration file to your needs, you need to tell InfluxDB to use the new configuration file. This can be done by modifying /usr/lib/systemd/system/influxdb.service and adjusting the ExecStart line to point at your new configuration file.

Once this has been done, we need to reload the systemctl daemon and restart the InfluxDB service.

systemctl daemon-reload && systemctl restart influxdb

We now have influxdb configured and ready to go. The last thing we will do is connect to InfluxDB over the CLI interface and create ourselves a database, which we will use later for collectd.

influx

Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.

Connected to http://localhost:8086 version 1.0.2

InfluxDB shell version: 1.0.2

Next, we will create our database for collectd.

< create database collectd > show databases
name: databases

---------------

name

_internal

collectd

>

We have now created and listed the database, and can see our newly created database.

Installation and configuration of Grafana

We are now going to set up Grafana. This can be done on either a different server or on the same server. For the purpose of this guide, we are going to do this on the local server.

To install Grafana, we first need to add another repository for our system using the command below:

cat << EOF > /etc/yum.repos.d/grafana.repo
[grafana]
name=grafana
baseurl=https://packagecloud.io/grafana/stable/el/7/\$basearch
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packagecloud.io/gpg.key https://grafanarel.s3.amazonaws.com/RPM-GPG-KEY-grafana
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF

We then want to install Grafana:

yum install grafana

And enable it to start at boot:

systemctl enable grafana-server

Finally, start it now (so we don’t have to wait for a reboot to launch it):

systemctl start grafana-server

Again, for the purpose of this guide, we will use the default settings. However, Grafana settings can be changed in the following two locations:

/etc/sysconfig/grafana-server
/etc/grafana/grafana.ini

A full guide of configuration options for Grafana can be found on their website, here.

We now need to add firewall rules so we can access Grafana on its standard port (3000). This can be done by running the following.

firewall-cmd --zone=public --add-port=3000/tcp --permanent
firewall-cmd --reload

We can now browse to Grafana’s web interface on http://:3000 and log in with admin/admin (the default credentials—which you should change, of course, for a production scenario).

Next, we need to add InfluxDB as a datasource in Grafana. We do this by logging into Grafana using the URL and credentials above, clicking the Grafana logo at the top left, and then selecting data sources from the dropdown. Then select “Add new data source.”

Grafana Data Sources Screenshots

You should be greeted with a screen similar to the one above. Now we need to fill in some fields to get connectivity working.

Name: local influxdb (Can be whatever you like)

Type: InfluxDB

Url: http://localhost:8086

Access: proxy (Proxy means Grafana will talk to your backend datasource, direct will mean your client will talk to the backend data source—as we have not opened up the ports to the InfluxDB backed, we are using proxy here).

Http Auth: Basic Auth should be checked here.

User: root (the defaults)

Password: root (the defaults)

Database: collectd

User: root (the defaults)

Password: root (the defaults)

You can now click “add”, which will save and test the configuration. You should get a confirmation that the datasource is working.

Grafana Data Source added

We now need to get some data into InfluxDB to graph.

Getting our first data in using collectd

We now need to install collectd on our server. This can be done by running:

yum -y install epel-release
yum -y install collectd

Once installed, we need to open up the collectd config file and tell it to write stats out to InfluxDB. To do that, open up /etc/collectd.conf and uncomment the following lines:

LoadPlugin network

Server "ff18::efc0:4a42" "25826"

While we are here we might as well update the server IP and port to our ones for InfluxDB. In our case, this is 127.0.0.1 and the default port.

We can now go ahead and enable/start collectd:
systemctl enable collectd

systemctl start collectd

Once enabled and started, we need to make InfluxDB listen on the correct port to receive this data. To do this, make the following changes in /etc/influxdb/influxdb.generated.conf:

Under [[collectd]]

enabled = true

Now restart InfluxDB:

systemctl restart influxdb

We can now head back to Grafana in our web browser, click the dropdown which says “Home” at the top and create a new dashboard by clicking “Create New” at the bottom of the screen:

Grafana dashboard

Next, on the left-hand side, click the tiny green indent on the page adn go to “Add Panel” > “Graph”.

It will then ask you to specify a datasource, which we added earlier, and a query, as in the following screenshot:

Screenshot Grafana dashboard

As you can see, we have defined our query and are now getting our load average graphed.

You can read more about the query language here.

Final Thoughts

We now have collectd, InfluxDB and Grafana set up and receiving metrics.

This has been a very quick run through to get you set up. You will need to play around with all aspects of the configuration to gain confidence in using the products. However, there are many good documentation resources available at the following websites:

I am also happy to answer any questions that you leave in the comments below.


Casey Rogers is an experienced Senior Infrastructure Engineer with more than 8 years’ experience working with some of the UK’s biggest companies and multinationals.Largely self-taught, Casey has long had an interest in IT and has been brought up around technology.


Discussion

  • InfluxDB is creating a visualization tool so you don’t need Grafana. Would that b of interest to you?

    Lawrence Hecht 11/09/2016 6:50 pm Reply
Click on a tab to select how you'd like to leave your comment

Leave a Comment

Your email address will not be published. Required fields are marked *

Menu