Prometheus — Glossary

Prometheus is an open-source monitoring system built around one idea: it pulls numeric measurements (metrics) from your systems at fixed intervals and stores them as a time series. Instead of watching dashboards waiting for something to look wrong, you write rules that fire alerts when a measurement crosses a threshold.

The pull model

Application / Exporter  <-- Prometheus scrapes every 15s
        |                        via HTTP GET /metrics
        v
   time series DB  -->  Alertmanager  -->  Email / Chat / Pager
                    -->  Grafana       -->  Dashboards
                    -->  long-term storage (Thanos, VictoriaMetrics)

Key point: Prometheus PULLS. Your service does not need to know
where the monitoring lives - it just has to expose /metrics.

The four metric types

Counter — only increases (requests served, errors, bytes sent). Read it with rate(), never use the raw value.
Gauge — can go up and down (temperature, memory used, active connections).
Histogram — buckets of observed values, used for request durations. Gives you averages and percentiles.
Summary — pre-calculated quantiles on the client side. Cheaper to query but quantiles cannot be aggregated across instances.

The query language: PromQL

# HTTP requests per second, averaged over 5 minutes
rate(http_requests_total[5m])

# Only errors
sum(rate(http_requests_total{status=~"5.."}[5m]))

# Error ratio as a percentage
100 * sum(rate(http_requests_total{status=~"5.."}[5m]))
    / sum(rate(http_requests_total[5m]))

# 95th percentile response time
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# Memory usage below 5% free
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 5

# Predict disk exhaustion within 4 hours
predict_linear(node_filesystem_avail_bytes[1h], 4 * 3600) < 0

# Absent = the target stopped reporting (a metric that should exist)
absent(up{job="api"} == 1)

# A target that is down
up == 0

Alert rules

groups:
  - name: availability
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 5m                    # must be true for 5 minutes
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} is down"
          description: "Target has been unreachable for 5 minutes."

      - alert: HighErrorRate
        expr: |
          100 * sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
              / sum(rate(http_requests_total[5m])) by (job) > 5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.job }} error rate above 5%"

# THE most important rule, often forgotten:
      - alert: NoData
        expr: absent(up{job="api"})
        for: 10m
        labels: {severity: critical}
        annotations:
          summary: "No metrics arriving from api"

Alert on symptoms, not causes. "The API returns 5 % errors" is actionable. "CPU is above 80 %" is often noise. If CPU is high and users are happy, nobody should be paged.

Instrumenting your own service

# Python - the four golden signals, plus a business metric
from prometheus_client import Counter, Histogram, Gauge, start_http_server

anfragen = Counter("app_requests_total", "Requests", ["method", "path", "status"])
dauer    = Histogram("app_request_duration_seconds", "Duration", ["path"],
                     buckets=(.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5))
laufend  = Gauge("app_requests_in_progress", "In progress")
auftraege = Counter("app_orders_total", "Completed orders", ["status"])

@dauer.labels(path="/api/orders").time()      # decorator measures duration
def bestellungen_holen():
    laufend.inc()
    try:
        ...
        anfragen.labels("GET", "/api/orders", "200").inc()
    except Exception:
        anfragen.labels("GET", "/api/orders", "500").inc()
        raise
    finally:
        laufend.dec()

if __name__ == "__main__":
    start_http_server(9100)                   # /metrics on :9100
    ...

# Cardinaility warning: labels are multiplied together.
#   path="/api/orders/12345"  -> one series PER ORDER ID -> thousands
#   of useless series. Use route templates: path="/api/orders/:id".

Exporters for things that do not speak Prometheus

node_exporter — CPU, memory, disk, network of a Linux host.
cadvisor — per-container metrics.
postgres_exporter / mysqld_exporter — database internals.
blackbox_exporter — probes HTTP/TCP/ICMP endpoints from outside.
snmp_exporter — network devices.

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

alerting:
  alertmanagers:
    - static_configs: [{targets: ["localhost:9093"]}]

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["web01:9100", "web02:9100"]
  - job_name: api
    metrics_path: /metrics
    static_configs:
      - targets: ["api:8080"]

# Validate before reloading - catches YAML and PromQL errors
promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/*.yml
curl -X POST http://localhost:9090/-/reload

Limitations worth knowing

Not for long-term storage. Local storage is sized in weeks. For years of history add Thanos, Mimir or VictoriaMetrics.
Not for logs or traces. That is Loki, Elasticsearch or Tempo. Prometheus handles numbers only.
No exact per-request data. Metrics are aggregated; you cannot ask "show me the 5 slowest requests".
Cardinality is the killer. Every unique label combination is a separate series. User IDs, request IDs and raw URLs in labels will bring Prometheus down. This is the number one operational mistake.
Alert fatigue is a configuration problem, not a tool problem. Fewer, better alerts with meaningful thresholds and for: durations beat hundreds of noisy ones.

Alert on what users feel: error rate, latency, saturation, availability. Everything else supports the diagnosis, it does not deserve a page.