Setup Docker

Once you install the OS of your choosing, you will want to install Docker/Docker Compose. For this, we choose Centos 9 Stream, although the steps will be similar under any OS you choose.

Setup the Repository

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

Install the latest version

sudo yum install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Start and Enable Docker

sudo systemctl start docker
sudo systemctl enable docker

Setup the TIG Stack

Now that Docker and the docker compose plugin are installed, we can now setup our docker compose yaml file to get the basis of our setup.

Setup a folder for the TIG stack and navigate to the folder

mkdir tig
cd tig

Save this file in your tig directory as:

docker-compose.yml

Create the docker compose file

version: '3.6'
services:
  telegraf:
    image: telegraf
    container_name: telegraf
    restart: always
    volumes:
    - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro
    depends_on:
      - influxdb
    links:
      - influxdb
    ports:
    - '8125:8125'

  influxdb:
    image: influxdb:1.8-alpine
    container_name: influxdb
    restart: always
    environment:
      - INFLUXDB_DB=influx
      - INFLUXDB_ADMIN_USER=influx
      - INFLUXDB_ADMIN_PASSWORD=superflux
    ports:
      - '8086:8086'
    volumes:
      - influxdb_data:/var/lib/influxdb

  grafana:
    image: grafana/grafana
    container_name: grafana
    restart: always
    depends_on:
      - influxdb
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_INSTALL_PLUGINS=
      - GF_AUTH_LDAP_ENABLED=true
    links:
      - influxdb
    ports:
      - '3000:3000'
    volumes:
      - ./grafana/ldap.toml:/etc/grafana/ldap.toml:ro,Z
      - grafana_data:/var/lib/grafana
  caddy:
    image: caddy:2-alpine
    container_name: caddy
    restart: always
    depends_on:
      - grafana
    volumes:
      - ./grafana/grafana.crt:/etc/ssl/private/grafana.crt:ro,z
      - ./grafana/grafana.key:/etc/ssl/private/grafana.key:ro,z
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
    ports:
      - 80:80   # Allows for http redirection
      - 443:443
volumes:
  grafana_data: {}
  influxdb_data: {}

Please note there are a couple variables in the above code you will want to change the influx password and grafana password environment variables. The grafana password will be prompted to change upon first login, so that one is not as critical.

I have caddy involved here to allow Grafana over HTTPS. Editing grafana.ini to allow HTTPS is possible, but a reverse proxy like Caddy makes this far easier.

Additionally, I have setup Grafana for LDAP authentication.

Below are some important settings in the docker-compose.yml file above that you need to watch out for. 

# Grafana environment variable to enable LDAP authentication. Omit this variable if you do not want to use LDAP.
- GF_AUTH_LDAP_ENABLED=true

# Map the ldap.toml file to allow LDAP integration with Grafana.  
- ./grafana/ldap.toml:/etc/grafana/ldap.toml:ro,Z

# Map the certificate files to your caddy container.
- ./grafana/grafana.crt:/etc/ssl/private/grafana.crt:ro,z
- ./grafana/grafana.key:/etc/ssl/private/grafana.key:ro,z

Woah now, what certificate files?

Good eye, glad you’re paying attention!

We need to create some locally signed certificates for Caddy to use for HTTPS because without them, HTTPS isn’t a thing.

Create a grafana directory inside your tig directory

cd ~/tig
mkdir grafana

Now generate a new private key, a certificate signing request and finally sign that key.

openssl genrsa -out grafana.key 2048

openssl req -new -key grafana.key -out grafana.csr

openssl x509 -req -days 3650 -in grafana.csr -signkey grafana.key -out grafana.crt

Generating a new CSR will prompt you for a few answers. Just make sure to set the ‘common name’ to the URL you want to use in Caddy for serving up your Grafana page.

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:NY
Locality Name (eg, city) [Default City]:NY
Organization Name (eg, company) [Default Company Ltd]:MyCompany LTD
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:supergraf.example.com
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Setup Telegraf

Create the telegraf.conf file. This will gather some basic metrics about your server and toss them into InfluxDB.

[global_tags]
[agent]
  interval = "60s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  hostname = "10.243.165.15"
  omit_hostname = false
[[outputs.influxdb]]
  urls = ["http://influxdb:8086"]
  database = "influx"
  timeout = "5s"
  username = "influx"
  password = "superflux"
[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  report_active = false
[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]
[[inputs.mem]]
[[inputs.processes]]

Setup Caddy

Now we want to setup Caddy to allow for the HTTPS magic to happen.

supergraf.example.com {
    reverse_proxy grafana:3000
    encode gzip zstd

    tls /etc/ssl/private/grafana.crt /etc/ssl/private/grafana.key

    log {
      level error
    }
}

Setup Grafana LDAP

Thought we forgot to finish this off? Think again!
The final step here is to setup your ldap.toml file with the appropriate settings for your environment. In the example below I have Grafana setup to use an Active Directory server.

cd grafana
vi ldap.toml

Now configure the ldap.toml file

[[servers]]
# Ldap server host (specify multiple hosts space separated)
host = "10.10.10.1"
# Default port is 389 or 636 if use_ssl = true
port = 389
# Set to true if LDAP server should use an encrypted TLS connection (either with STARTTLS or LDAPS)
use_ssl = false
# If set to true, use LDAP with STARTTLS instead of LDAPS
start_tls = false
# set to true if you want to skip ssl cert validation
ssl_skip_verify = true

# Search user bind dn
bind_dn = "CN=ldap,OU=Service Accounts,DC=example,DC=com"
# Search user bind password
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
bind_password = """superbind"""

# Timeout in seconds (applies to each host specified in the 'host' entry (space separated))
timeout = 10

# User search filter, for example "(cn=%s)" or "(sAMAccountName=%s)" or "(uid=%s)"
search_filter = "(sAMAccountName=%s)"

# An array of base dns to search through
search_base_dns = ["dc=example,dc=com"]

# Specify names of the ldap attributes your ldap uses
[servers.attributes]
member_of = "memberOf"
email =  "email"

[[servers.group_mappings]]
group_dn = "CN=Domain Admins,CN=Users,DC=example,DC=com"
org_role = "Admin"

[[servers.group_mappings]]
# If you want to match all (or no ldap groups) then you can use wildcard
group_dn = "*"
org_role = "Viewer"

Finally, start your docker compose!

cd ~/tig
docker compose up -d

Conclusion

Once you have completed all the above steps, your grafana instance will be available at supergraf.example.com (you did change all the settings above to match your environment, right? RIGHT?)

To validate your InfluxDB is gathering data from Telegraf, you can either browse the InfluxDB directly, or install this dashboard into your grafana

https://grafana.com/grafana/dashboards/928-telegraf-system-dashboard/

If all went well, you will now be seeing system metrics for your new tig server populating into your telegraf dashboard within the next hour.