Grafana, Loki, and Alloy: centralized logs when you're the whole ops team

For about two years my debugging workflow was docker logs -f in one terminal and docker logs -f for a different container in another terminal, and squinting between them.

It works fine right up until it doesn't. A container restarts and takes its history with it. You want to know what happened at 2:14am and the buffer rolled over hours ago. Something breaks in a way that involves three services and you're trying to line up timestamps by eye across three scrollback buffers. Then you add a second machine and the whole approach quietly falls apart.

The fix is a log pipeline, and the Grafana stack version of it is genuinely pleasant to run at small scale. I want to write down how the pieces fit, because most of what's published about Loki assumes you're at a company with a platform team and a Kubernetes cluster.

Three components, three jobs

Loki stores the logs. The clever bit is that it indexes labels rather than log content. A traditional log store builds a full-text index over everything you send it, which is why those systems get expensive fast. Loki indexes a small set of key-value labels and keeps the actual log lines as compressed chunks. Queries filter by label first, then grep the chunks that survive. Cheaper to run, and the tradeoff only bites if you routinely search unstructured text across enormous time ranges.

Alloy collects and ships. It replaced Promtail and the old Grafana Agent, speaks OpenTelemetry, and handles discovery, relabeling, parsing, and delivery. One binary, one config file.

Grafana queries and draws. LogQL for search, panels for dashboards, alert rules that fire on log patterns.

You can run all three on a single box with Docker Compose. Mine live on one machine in my house that I refer to as the datacenter, which gets funnier the more people assume I mean a real one.

Applications should not know Loki exists

This is the design decision everything else depends on, and it's easy to get wrong in the other direction by reaching for a logging library with a Loki backend.

Every app writes structured JSON to stdout. That's the entire logging integration.

{"level":"info","time":"2026-08-21T14:32:01.442Z","msg":"probe_complete","target":"api","ms":412}

No shipping library, no network calls from the app, no credentials, no retry buffer, nothing that can fail and take a request path down with it. The container runtime captures stdout. Alloy reads it out of Docker. The app has no idea any of this is happening and runs identically on my laptop with nothing collecting at all.

The payoff shows up when you change something. Swapping Loki for a different store, adding a second destination, rerouting one service's logs somewhere else--all of that is collector config. You never touch application code, and you never redeploy a service to change where its logs go.

Opt-in through Docker labels

Alloy discovers containers through the Docker socket. Rather than maintaining a list of what to collect, I let containers volunteer:

services:
  some-service:
    image: some-service:latest
    labels:
      logging: "alloy"
      app: "some-service"

Two labels and a container joins the pipeline. No collector restart, no config edit. Anything without the logging label gets ignored, which keeps noisy third-party containers out by default instead of requiring me to exclude them one at a time.

The Alloy side:

discovery.docker "containers" {
  host             = "unix:///var/run/docker.sock"
  refresh_interval = "15s"
}

discovery.relabel "containers" {
  targets = discovery.docker.containers.targets

  rule {
    source_labels = ["__meta_docker_container_label_logging"]
    regex         = "alloy"
    action        = "keep"
  }

  rule {
    source_labels = ["__meta_docker_container_label_app"]
    target_label  = "app"
  }

  rule {
    source_labels = ["__meta_docker_container_name"]
    regex         = "/(.*)"
    target_label  = "container"
  }
}

loki.source.docker "containers" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.relabel.containers.output
  labels     = { env = "prod", host = "basement" }
  forward_to = [loki.process.structured.receiver]
}

loki.process "structured" {
  stage.json {
    expressions = { level = "level", ts = "time" }
  }
  stage.labels {
    values = { level = "" }
  }
  stage.timestamp {
    source = "ts"
    format = "RFC3339"
  }
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

Read top to bottom: find containers, keep the ones that opted in, copy their Docker labels into log labels, read their stdout, parse the JSON, promote level to a label, use the app's own timestamp instead of ingestion time, ship.

That timestamp stage matters more than it looks. Without it, every line gets stamped when Loki received it, and a collector hiccup smears twenty minutes of events into one spike that lines up with nothing.

The mistake that wrecks your instance

Loki's performance model rests entirely on label cardinality, and this is where people ruin their setup in the first week.

Labels are for things with few distinct values. app, env, host, level, container. Maybe a few dozen combinations total.

Every unique combination of label values creates a separate stream, and each stream gets its own chunks. Add request_id as a label and you've created one stream per request. Add user_email and you get one per user. Memory climbs, queries crawl, and eventually ingestion starts rejecting writes.

The good news is you lose nothing by keeping high-cardinality fields in the log line, because LogQL parses JSON at query time:

{app="api", env="prod"} | json | request_id="d41d8cd9"

That filters to the api streams by label, then scans those chunks for the request ID. Fast enough in practice, and your label space stays tiny.

Rule of thumb I use: if I can't guess every possible value of a field off the top of my head, it does not become a label.

The laptop runs the same collector

Here's the part that surprised me by how much I liked it. I run a second Alloy instance on my development machine, collecting from local Docker, shipping to the same Loki.

The only meaningful difference in its config is one label:

loki.source.docker "containers" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.relabel.containers.output
  labels     = { env = "dev", host = "laptop" }
  forward_to = [loki.process.structured.receiver]
}

env = "dev" instead of env = "prod". That one label does a lot of work, and forgetting it would poison every dashboard and alert with development noise, so put it in before anything else.

What I get out of it:

Local debugging uses the same queries as the server. I'm not context-switching between docker logs plus grep on the laptop and LogQL on the box. One query language, one set of muscle memory, both places.

Bad log shape gets caught before it ships anywhere. If a service emits a timestamp Alloy can't parse, or forgets to include level, or wraps everything in an extra data object, I find out while I'm writing it. Before this, log format problems surfaced when I went looking for something during an actual incident, which is the worst possible time.

Dashboards built against dev data work unchanged against prod. Same labels, same JSON fields, same panels. I build a panel while developing a service and it's already correct for the deployed one.

Comparing the two environments becomes a query. Drop the env filter and both show up side by side, which is a fast way to confirm that something misbehaving on the server also misbehaves locally.

On Docker Desktop the socket mount works the same as on Linux, so the collector config is nearly identical across a Windows laptop and a Linux server. I keep both in the same repo with the environment-specific labels in a small overlay.

The cost is a container idling on my laptop and some log volume from services I'm actively poking at. Retention on env="dev" can be short, and it's a small fraction of the total.

LogQL is the payoff

Everything above is plumbing. This is the part you actually use.

Errors across everything in production:

{env="prod"} | json | level="error"

Error rate by service, which makes a decent first dashboard panel:

sum by (app) (count_over_time({env="prod"} | json | level="error" [5m]))

Slow operations, filtering on a numeric field parsed out of the JSON:

{app="api"} | json | ms > 500 | line_format "{{.msg}} took {{.ms}}ms"

line_format is underrated. Raw JSON lines are miserable to read in a dashboard, and rewriting them into something human costs one clause.

Following one request across services:

{env="prod"} | json | request_id="d41d8cd9"

That last one is the reason to do any of this. Reconstructing a sequence of events across several services used to mean opening three terminals and comparing clocks. Now it's a query, and the answer arrives in the order it happened.

Alerting without much ceremony

Grafana alert rules run LogQL on a schedule, so an alert is a query plus a threshold. The first one worth having:

sum(count_over_time({env="prod"} | json | level="error" [10m])) > 0

Crude, and correct for a small setup. If you only produce errors when something is wrong, then any error deserves a look. Refine it later when a chatty service forces you to.

Retention is a Loki config setting, and disk is the constraint. Compressed chunks are small enough that a homelab box holds months of logs from a handful of services without anyone noticing. Set a retention period on day one anyway, because the failure mode of not setting one is discovering it when the disk fills.

Things that bit me

Multi-line stack traces arrive as one line each. Alloy has a stage.multiline for stitching them back together, and you want it configured before the first exception rather than after.

Clock skew between machines makes correlation lie to you. NTP on everything.

The Docker log driver has to be one Alloy can read. json-file or local. If you've pointed a container at some other driver, Alloy sees nothing and gives you no particular reason why.

Chatty containers will happily bury everything else. Databases and reverse proxies at debug level generate remarkable volume. The opt-in label approach handles this by default, which is most of why I use it.

Query while forgetting a label filter and you scan every chunk in the range. Start narrow.

Worth it at what size?

Two containers on one machine, probably not. docker logs is right there.

The line for me was somewhere around a handful of services on more than one machine, plus wanting to answer questions about the past rather than the present. Once "what happened last Tuesday" is a question you need answered, you need something that kept last Tuesday.

Setup is an afternoon. The part that took longer was learning to think in labels instead of grep, and putting the same collector on my laptop was what made that click, because I was writing LogQL every day instead of only when something was already on fire.