Using Prometheus and Grafana with Django

For my web app project I wanted to add some monitoring and dashboards so I decided to try out prometheus. Since I am using django I also decided to use django-prometheus. It ended up fine but there was a bump or two along the way.

The README tells you most of what you need to know. There’s a few tutorials out there on the web which basically duplicate this info.

This is all fine and easy to do. The trick is that it won’t work in general for any reasonably complex django install. Specifically it won’t handle the case where the backend has more than one process. This is clearer in the documentation for the official python prometheus client. There is also information in this django specific tutorial that doesn’t use django-prometheus.

The key thing you need to do is setting:

PROMETHEUS_MULTIPROC_DIR=/tmp

So that all the processes will share the same metrics counters. Until then you can waste a lot of time being confused about why the numbers are so wacky.

Getting prometheus going from the container is really easy:

You just need a docker-compose file or whatever your poison is.

services:
  monitor:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - /mnt/db/prom:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
    ports:
        - 9090:9090
    expose:
      - 9090

and then a prometheus.yml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: django # or whatever you like
    static_configs:
      - targets: ['your_app.example.com']

I have all this proxied behind apache so I protected the url for the prometheus server with basic auth.

Next you can just make a grafana cloud account and use the prometheus install as a data source.