Grafana — Glossary

Grafana is a visualisation and dashboard platform. It does not collect or store your data — it queries other systems (Prometheus, Loki, PostgreSQL, Elasticsearch, InfluxDB and dozens more) and turns the results into graphs, tables and alerts.

What it is good at

Dashboards: Panels laid out on a grid, shared as code, embedded in other sites.
Ad-hoc exploration: Query a data source and see the result immediately without building a dashboard first.
Unified view: Metrics, logs and traces side by side, correlated by time and labels. This is the real strength — one place to look when something breaks.
Annotations: Mark deployments, incidents and maintenance windows on the timeline so you can see cause and effect.
Templating: Variables in the dashboard (server, environment, service) instead of hundreds of separate dashboards.

The dashboard you actually need

Start with four panels per service, based on the four golden signals:

1. Traffic      rate(requests_total[5m])            requests per second
2. Errors       5xx / total * 100                   error percentage
3. Latency      histogram_quantile(0.95, ...)       p95 response time
4. Saturation   cpu, memory, queue depth            how full is it

Additional, cheap and very useful:
5. Deployment annotations (releases marked on the timeline)
6. Instance availability: up{job="$service"}
7. Top-N slowest endpoints
8. Business metric (orders/min, signups/min)

Deliberately OUT of the top row: total requests since boot,
CPU per core, disk I/O detail. They belong lower down, if at all.

Variables instead of many dashboards

# A dashboard variable "instance" built from a query
label_values(up{job="node"}, instance)

# Then every panel uses it
rate(node_cpu_seconds_total{instance="$instance", mode="idle"}[5m])

# Extended: environment -> service -> instance, cascading
label_values(up, env)                              # env
label_values(up{env="$env"}, job)                  # job
label_values(up{env="$env", job="$job"}, instance) # instance

# Result: one dashboard serves every environment and service.
# Rule: never hard-code an instance or a hostname in a panel.

Alerting in Grafana vs Prometheus

Prometheus rules are the classic approach: rules live with the metrics, evaluated by Prometheus, routed by Alertmanager (grouping, silencing, inhibition, escalation).
Grafana alerting supports the same but works across any data source, including SQL. Useful when alerts depend on business data in a database.
Pragmatic choice: infrastructure alerts in Prometheus, business-logic alerts in Grafana. Do not duplicate the same alert in both systems — that guarantees double pages.

Dashboards as code

# Exporting a dashboard: the JSON is portable, but it is generated output.
# Keep the source in version control, not the exported JSON.

# Option A: provision dashboards from files
# /etc/grafana/provisioning/dashboards/default.yaml
apiVersion: 1
providers:
  - name: main
    type: file
    options:
      path: /var/lib/grafana/dashboards

# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      timeInterval: 15s

# Option B: generate dashboards with a tool so they can be reviewed in a PR
#   Grafonnet (Jsonnet) or Terraform's grafana provider
# That way "who changed which panel and why" is answerable from git history.

# Always verify provisioning took effect:
curl -s -u admin:geheim http://localhost:3000/api/search?query= | jq '.[].title'

Practical rules

A dashboard must answer a question. "How is the API doing?" — not "here are 60 numbers". If nobody can say what a panel means, delete it.
Top row = what a human should look at first during an incident.
Use consistent colours and units. Green good, red bad, everywhere. Percentages as percent, not as fractions.
Set a sensible default time range (last 6 hours, refresh 30 s). "Last 5 minutes" is too noisy for a first look.
Annotate deployments. Half of all "the graph changed" investigations are answered by seeing that a release went out at exactly that time.
Beware of averages. A 200 ms average can hide a very slow minority of users. Show percentiles (p95, p99).
Screenshots in runbooks age badly. Link to the dashboard with a time range instead.
Do not measure what nobody acts on. Every panel costs attention.

Permissions and access

# Grafana is not a security boundary - it holds read credentials
# for every data source. Treat it like a production system.
#   - Admin password not the default, and not "admin/admin"
#   - SSO / OIDC instead of local accounts
#   - Editor rights granted deliberately (an editor can query
#     any data source, which may include customer data)
#   - Anonymous access off
#   - Organisations/folders per team with minimum necessary access
#   - API keys / service accounts with short lifetimes for automation

# Typical mistake: exposing Grafana publicly "just for read".
# Read access to metrics often reveals internal topology,
# hostnames, traffic volumes and sometimes customer identifiers.