Prometheus & Grafana: A Practical Monitoring Setup from Scratch

You can't fix what you can't see. Whether you're running three containers or three hundred nodes, you need a monitoring stack that gives you real answers, not just colorful graphs. Prometheus and Grafana have become the de facto standard for infrastructure monitoring — and for good reason. Prometheus scrapes and stores metrics; Grafana turns them into dashboards humans can actually read.

This guide walks you through setting up both from scratch with Docker Compose, connecting your first targets, writing useful PromQL queries, and building a dashboard that tells you something worth knowing.

Why Prometheus + Grafana?

Prometheus pulls metrics from targets on a schedule (pull-based), stores them in a time-series database, and lets you query them with PromQL — a functional query language designed specifically for metrics. It's lightweight, has no external dependencies, and was built for the kind of ephemeral infrastructure we deal with today.

Grafana is the visualization layer. It connects to Prometheus as a data source and lets you build dashboards with panels, alerts, and annotations. Together they cover the full pipeline: collect → store → query → visualize → alert.

Setting Up with Docker Compose

Create a docker-compose.yml with both services:

version: "3.8"
services:
  prometheus:
    image: prom/prometheus:v2.53.0
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=30d'

  grafana:
    image: grafana/grafana:11.1.0
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin123
    depends_on:
      - prometheus

  node-exporter:
    image: prom/node-exporter:v1.8.1
    ports:
      - "9100:9100"
    pid: host

volumes:
  prometheus_data:
  grafana_data:

The node-exporter exposes host-level metrics — CPU, memory, disk, network — from whatever machine it's running on. It's the most common first target for Prometheus.

Configuring Prometheus

Create prometheus.yml in the same directory:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']
        labels:
          env: 'production'

scrape_interval controls how often Prometheus polls each target. 15 seconds is a sensible default — low enough to catch problems quickly, high enough to not drown in data. The labels section lets you tag targets, which becomes essential when you're monitoring multiple environments.

Start everything:

docker compose up -d

Prometheus UI is at http://localhost:9090. Check Status → Targets — both endpoints should show "UP".

Essential PromQL Queries

PromQL looks intimidating at first, but the core idea is simple: every query selects a time series, then optionally transforms it. Here are queries you'll actually use:

CPU Usage Percentage

100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

This takes the idle CPU rate over 5 minutes, subtracts it from 100, and gives you actual utilization. The irate() function computes per-second rate from the last two data points in the range — it's more responsive than rate() for dashboards.

Memory Usage

(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

Disk Usage by Mountpoint

(1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes) * 100

The fstype!~"tmpfs|overlay" filter excludes virtual filesystems that would otherwise pollute your results.

HTTP Request Rate (for apps with a /metrics endpoint)

sum(rate(http_requests_total[5m])) by (method, status)

Building Your First Dashboard

Open Grafana at http://localhost:3000 (admin / admin123), then:

1. Add data source: Settings → Data Sources → Add → Prometheus. URL: http://prometheus:9090. Click "Save & Test".
2. Create dashboard: Dashboards → New → New Dashboard → Add visualization.
3. Add panels for CPU, memory, and disk using the PromQL queries above.

For each panel, set these options sensibly:

• Unit: percent (0-100) for CPU/memory/disk panels
• Thresholds: green below 70%, yellow at 70%, red at 90%
• Time range: default to last 1 hour for operational dashboards

Adding Alert Rules

Dashboards are for humans who are already looking. Alerts are for when nobody is. Create a rules file alert_rules.yml:

groups:
  - name: host_alerts
    rules:
      - alert: HighCPU
        expr: 100 - (avg by(instance)(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {% raw %}{{ $labels.instance }}{% endraw %}"

      - alert: DiskAlmostFull
        expr: (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes) * 100 > 90
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Disk > 90% on {% raw %}{{ $labels.instance }}{% endraw %}"

Add to your prometheus.yml:

rule_files:
  - 'alert_rules.yml'

The for: 5m clause prevents flapping — the condition must hold for 5 continuous minutes before the alert fires. This single line saves you from a lot of noise.

Service Discovery Beyond Static Configs

Static targets lists work for small setups, but they rot fast when services scale. Prometheus supports built-in service discovery for most platforms:

• Docker: docker_sd_configs — auto-discovers containers with specific labels
• Kubernetes: kubernetes_sd_configs — the standard for K8s, discovers pods, services, and nodes
• Consul: consul_sd_configs — pulls targets from Consul service registry
• File-based: file_sd_configs — JSON/YAML files that you update from any automation

If you're running containers, Docker service discovery is the obvious next step. Label your containers with prometheus.scrape=true and Prometheus handles the rest.

Practical Tips

• Retention matters: Default is 15 days. Set --storage.tsdb.retention.time=30d for production. Disk is cheap; not having historical data when you need it isn't.
• Recording rules: Pre-compute expensive queries into new time series. If a dashboard query takes seconds, make it a recording rule and query the result.
• Don't over-scrape: 15s is fine for most targets. Scraping every second creates storage pressure for negligible resolution gains.
• Label cardinality: Every unique label combination is a separate time series. Labels like user_id or request_id will explode your storage. Keep labels to things you'll actually query by.
• Use rate() for counters: Raw counter values are meaningless. Always use rate() or irate() to get per-second rates.

What to Monitor First

Don't try to monitor everything at once. Start with the four golden signals from Google's SRE book:

1. Latency: How long requests take
2. Traffic: How many requests per second
3. Errors: Rate of failed requests
4. Saturation: How "full" your resources are (CPU, memory, disk, queue depth)

These four metrics cover most of what you need to know about a system's health. Everything else is detail you add when a specific problem demands it.

Monitoring isn't about collecting data — it's about knowing when to act. Set up Prometheus and Grafana with these basics, and you'll have a foundation that grows with your infrastructure instead of fighting it.