/ Influxdb

cAdvisor with StatsD, influxdb and Grafana

Why use cAdvisor with StatsD?

cAdvisor already have an influxDB backend and it works great, but, I would like to send data to StatsD and StatsD handle where I would like to send my data: InfluxDB, Zabbix or Grafite.

Another point, I don't really love the schema created by cAdvisor because, prior to InfluxDB 0.9, with a lot of data points, I need to filter on some metric, or node or container and it was sub performant.

It's why I made a really simple schema with StatsD and InfluxDB:
docker_node.container_name.metric with only one value. Yes, it's not the simplest schema for some aggregation on dashboard on the first view, but I will explain how I do it with Grafana later.

StatsD

Why I love StatsD? Because it uses UDP as transport, it aggregates stats before sending it to the backend, support many backends like Zabbix and InfluxDB and could easily scale.

Configuration

InfluxDB

For this blog post, we will keep all default configuration of InfluxDB 0.9. We will create a database and a user for our metrics. I will use the HTTP API, but you can do this directly on the web UI or with the influx command line.

On each curl request, you should add the basic authentication of an admin user: $ curl -u username:password

Create the database

$ curl -G 'http://localhost:8086/query' --data-urlencode "q=CREATE DATABASE cadvisordb"
> {"results":[{}]}

Create a retention policy

Now we need to set a retention policy, by default it's an infinite retention

$ curl -G 'http://localhost:8086/query' --data-urlencode "q=\
    CREATE RETENTION POLICY cadvisor\
      ON cadvisordb\
      DURATION 90d\
      REPLICATION 1\
      DEFAULT"'
> {"results":[{}]}

Create a user

$ curl -G 'http://localhost:8086/query' --data-urlencode "q=\
    CREATE USER cadvisor \
      WITH PASSWORD 'password'"
> {"results":[{}]}

Grant read and write to cadvisor on cadvisordb

$ curl -G 'http://localhost:8086/query' --data-urlencode "q=\
    GRANT ALL \
      ON cadvisordb \
      TO cadvisor"'
> {"results":[{}]}

StatsD

For statsd, we will use the influxdb backend. It's not a backend provided by default with StatsD, we need to install it manually.

$ git clone --depth=1 https://github.com/etsy/statsd.git
$ cd statsd
$ npm install statsd-zabbix-backend
{
  port: 8125,
  percentThresold: [95, 99],
  flushInterval: 10000,
  deleteIdleStats: true,
  backends: ["statsd-influxdb-backend"],
  influxdb: {...}
}
influxdb: {
  host: '<influxdb IP>',
  port: 8086,
  version: 0.9,
  database: 'dbname'
}

For a detailed list of option for InfluxDB, you can check on the influxdb backend repository

Grafana dashboard

grafana-dashboard

Julien Maitrehenry

Julien Maitrehenry

I specialize in DevOps, Agile practices and web development. I love sharing my knowledge for helping other people to go to the next level!

Read More