Prometheus: The Pull-Based Time-Series Database Powering Self-Hosted Observability

Prometheus: The Pull-Based Time-Series Database Powering Self-Hosted Observability

1. What Prometheus Actually Is

Prometheus is an open-source systems monitoring and alerting toolkit that has, over the last decade, become the de facto standard for metrics in the cloud-native world. It was originally built at SoundCloud in 2012 to solve the gaps left by then-dominant but rigid monitoring stacks, and it was the second project ever to graduate from the Cloud Native Computing Foundation (CNCF) after Kubernetes itself. As of late 2026 it carries roughly 66,000 GitHub stars, is licensed entirely under Apache 2.0, and is governed by an open community rather than a single vendor. For the self-hoster, that independence matters: no company can suddenly change the licensing terms or hold your observability data hostage behind a paywall. At its heart, Prometheus is a time-series database (TSDB) with a built-in scraping engine, a powerful query language called PromQL, and a local storage layer that keeps every sample you collect on disks you control. Unlike many SaaS monitoring products that ship your telemetry to a vendor's cloud for analysis, Prometheus pulls metrics from your own services over HTTP and stores them locally. That single design decision is the reason it fits so naturally into a self-hosting or data-sovereignty mindset. The project's longevity is not accidental. Prometheus reached its 3.0 milestone in November 2024, and by late 2026 it sits at the 3.15.x line, with releases shipping every few weeks. The 3.15.0 release in September 2026, for example, added the long-requested ability to scrape targets over Unix Domain Sockets, meaning you no longer need to expose a TCP port just so Prometheus can reach a local exporter. For a homelabber who dislikes open ports, that is a meaningful privacy win. Prometheus architecture

2. The Pull Model vs the Push Model

Most legacy monitoring tools, and many modern SaaS ones, use a push model: your application sends data outward to a collector. Prometheus deliberately inverts this. The Prometheus server pulls metrics from instrumented targets by making an HTTP GET request to a /metrics endpoint that each target exposes. This inversion has profound implications for self-hosters. First, it means your monitored services never need outbound internet access. A web server in your basement can sit behind a firewall with no egress rules, and Prometheus will still collect its metrics as long as the server can reach it on the internal network. Second, it makes debugging trivial: if Prometheus cannot scrape a target, the failure is almost always a networking or configuration issue on your side, not a dropped packet somewhere in a vendor's pipeline. Third, it forces a clean, text-based exposition format that you can curl yourself to see exactly what is being collected. The trade-off is that Prometheus must be able to reach every target. In a dynamic environment with dozens of containers spinning up and down, you need service discovery to tell Prometheus where they are. In a static homelab, a hand-written list of scrape targets is usually enough.

3. The Exposition Format and Instrumentation

When Prometheus scrapes a target, it expects a simple text format. A typical response looks like this: ``

HELP http_requests_total Total HTTP requests.

TYPE http_requests_total counter

http_requests_total{method="get",code="200"} 1027 http_requests_total{method="post",code="500"} 3

HELP node_cpu_seconds_total Seconds of CPU time consumed.

TYPE node_cpu_seconds_total counter

node_cpu_seconds_total{mode="idle"} 41233.9
` Each line is a sample: a metric name, an optional set of key-value labels, and a float value, stamped with the scrape timestamp. This multidimensional model is the secret sauce of Prometheus. Because every metric carries labels, you can slice and aggregate along any dimension—per host, per HTTP method, per status code—without pre-defining those groupings. Instrumenting your own code is straightforward in Go, Python, Java, and most other languages, thanks to official and community client libraries. For off-the-shelf software that does not expose Prometheus metrics natively, you use an exporter: a small sidecar process that translates a system's native statistics into the exposition format. Node Exporter turns Linux host statistics into metrics. Blackbox Exporter probes endpoints. There are hundreds of exporters, which is part of why the ecosystem feels bottomless.

4. PromQL: The Query Language

PromQL is the language you use to extract meaning from raw samples. It is not SQL. It is designed around vectors of time-series data and functions that operate on them. A few examples illustrate its character:
  • rate(http_requests_total[5m]) computes the per-second average request rate over the last five minutes, automatically handling counter resets.
  • histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) estimates the 95th percentile latency.
  • sum by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) aggregates idle CPU per instance.
PromQL's learning curve is real, but it rewards study. The same query that powers a dashboard also powers an alerting rule, so there is no separate alert DSL to learn. For a self-hoster who wants to ask questions like "which of my containers used the most memory at 3 a.m.?" PromQL answers them directly. PromQL query in the expression browser

5. The Local TSDB: How Your Data Is Stored

Prometheus does not delegate storage to an external database. It writes samples to a custom TSDB on the local filesystem, organized into two-hour blocks. Recent data lives in a write-ahead log and an in-memory head block; older data is compacted into immutable block files on disk. This design is what makes Prometheus so self-contained—and so sovereign. The storage engine is remarkably efficient. Each sample typically consumes only one to two bytes on disk after compression. A common rule of thumb: scrape 10,000 active series at a 15-second interval and you will write roughly 57 million samples per day, which compresses down to somewhere between 50 and 200 megabytes per day depending on cardinality and churn. Over a 30-day retention window, expect on the order of 3 to 6 gigabytes of disk. That is small enough to fit on a Raspberry Pi's SD card for a modest homelab, though you would not want production data on an SD card for long. Retention is configured with the
--storage.tsdb.retention.time flag (or the equivalent config). The default is 15 days. You can set it to anything from a few hours to multiple years, limited only by your disk budget. The 3.15.x line also stabilized chunk-encoding improvements and faster head initialization, which matter when you restart an instance that holds a large in-memory head.

6. Service Discovery and Scraping Targets

For a static setup, your
scrape_configs might list three targets by IP. For anything dynamic, Prometheus supports dozens of service-discovery mechanisms: Kubernetes, Consul, DNS, Docker, file-based, AWS, Azure, GCP, and more. In a Docker Compose homelab, the file-based or DNS SD is often enough: Prometheus resolves container names via the Docker network's internal DNS and scrapes them. Relabeling is the advanced technique that ties it all together. Relabel configs let you rewrite, drop, or add labels before a target is scraped, and again before a series is stored. You can use it to drop high-cardinality labels that would blow up your storage, or to tag every series with the environment it came from. It is fiddly, but mastering it is the difference between a tidy metrics store and a disk that fills up overnight.

7. Alerting with Alertmanager

Prometheus itself only records and queries. Alerting is delegated to a companion component called Alertmanager. You define alerting rules in Prometheus—for example, "fire an alert if the 5-minute error rate exceeds 1%"—and when the condition holds, Prometheus sends the alert to Alertmanager. Alertmanager then handles grouping, silencing, inhibition, and routing to notification channels: email, Slack, Discord, webhooks, PagerDuty, and many more. For a self-hoster, this is where the project pays for itself. You can have Alertmanager ping your phone via a self-hosted ntfy instance or a Discord webhook the moment a disk is about to fill or a service goes down. No monthly subscription to a status-page vendor is required. The whole notification path can remain inside your own network.

8. Visualization: Grafana and the Native UI

Prometheus ships with a bare-bones expression browser, adequate for poking at queries but not for dashboards. The standard companion is Grafana, itself open source, which connects to Prometheus as a data source and renders rich dashboards. Many community dashboards exist for Node Exporter, Docker, and common services, so you can import a polished view in minutes. Grafana, interestingly, began as a Prometheus-specific front end and still treats it as a first-class citizen. In our own catalog, we have covered Grafana directly, and the two tools are so commonly paired that learning one implicitly teaches you about the other. Prometheus also exposes its own data over HTTP APIs that any compatible panel or custom script can consume. Grafana dashboard fed by Prometheus

9. Exporters: The Long Tail of the Ecosystem

The exposition format spawned an entire economy of exporters. Some of the most useful for a self-hoster:
  • Node Exporter for host CPU, memory, disk, and network.
  • cAdvisor for container resource usage.
  • Blackbox Exporter for probing HTTP, HTTPS, DNS, and TCP endpoints from the outside.
  • Postgres Exporter, MySQL Exporter, Redis Exporter for databases.
  • WireGuard Exporter, Unbound Exporter, and countless others for network gear.
This breadth means that almost anything you run can be made observable with a small amount of glue. The flip side is operational surface area: every exporter is another process to update and secure.

10. Installing Prometheus on a Single Node

A minimal Docker Compose deployment looks roughly like this:
`yaml services: prometheus: image: prom/prometheus:v3.15.0 volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml - prom_data:/prometheus command: - --config.file=/etc/prometheus/prometheus.yml - --storage.tsdb.retention.time=30d - --web.enable-lifecycle ports: - "127.0.0.1:9090:9090" restart: unless-stopped volumes: prom_data: ` A minimal prometheus.yml might scrape Prometheus itself and a Node Exporter: `yaml global: scrape_interval: 15s scrape_configs: - job_name: prometheus static_configs: - targets: ['localhost:9090'] - job_name: node static_configs: - targets: ['node-exporter:9100'] ` Binding the port to 127.0.0.1 rather than 0.0.0.0 is a small but real security habit: it keeps the UI off the public internet and behind your reverse proxy, where you can enforce authentication.

11. Configuring Scrape Intervals and Relabeling

Global and per-job scrape intervals let you trade granularity for storage. A critical security camera might be scraped every 5 seconds; a rarely changing backup job might be scraped every 5 minutes. Shorter intervals multiply your sample count linearly, so they should be reserved for metrics where the detail is actually used. Relabeling deserves its own paragraph because it is where beginners lose data or waste disk. The
action: labeldrop directive can strip a noisy label; action: keep can discard entire targets that fail a regex; action: labelmap can promote service-discovery labels into your series. Used well, relabeling keeps your TSDB lean. Used carelessly, it can silently drop the one dimension you needed.

12. Retention, Compaction, and Disk Footprint

Disk planning is the unglamorous core of running Prometheus well. As a practical formula, estimate active series, multiply by samples per series per day (86,400 divided by your scrape interval in seconds), and multiply by your per-sample byte cost. For a homelab scraping 2,000 series every 15 seconds with 30-day retention, you are looking at roughly 1 to 2 gigabytes—trivial. For a small business scraping 200,000 series every 10 seconds, you are in the hundreds of gigabytes per month and should strongly consider a long-term storage tier. Compaction runs in the background and merges smaller blocks into larger ones, improving query speed and compression. You will occasionally see CPU and disk spikes during compaction; that is normal. If you run on spinning disks, compaction can bottleneck, which is why SSDs are recommended for anything beyond a toy deployment.

13. High Availability and Federation

A single Prometheus instance is a single point of failure for your alerting. The standard HA pattern is to run two identical Prometheus servers scraping the same targets; only one connects to Alertmanager at a time, and the duplicate simply provides redundancy. True horizontal sharding—splitting different jobs across different servers—requires either manual division or a project like Thanos or Grafana Mimir sitting on top. Federation lets one Prometheus pull aggregated time series from another. A common topology: a per-datacenter Prometheus collects everything locally, and a global Prometheus federates only the pre-aggregated totals it needs. For a single homelab, federation is overkill, but it is good to know the scaling path exists without leaving the open-source stack.

14. Remote Write and Long-Term Storage

Prometheus's local TSDB is not designed to be queried across years of history from a single node. For long-term retention, you configure
remote_write to ship samples to a backend such as Thanos, Grafana Mimir, Cortex, or VictoriaMetrics. This is where data sovereignty gets nuanced: remote write sends your metrics off the Prometheus box to whatever endpoint you configure. If that endpoint is your own object storage bucket on a provider you trust, you retain control. If it is a third-party hosted service, you have traded local ownership for convenience. It is worth stating plainly: the default, out-of-the-box behavior of Prometheus ships nothing anywhere. Remote write is opt-in and explicit. That is the right default for a privacy-minded operator.

15. Real Self-Hosting Costs

Let us put actual numbers on the table, because the landing page will not. Scenario A: Homelab, zero incremental cost. You already run a mini PC or NAS. Prometheus runs as a container using 200–500 MB of RAM and a few gigabytes of disk. Electricity for a 15-watt mini PC at $0.15/kWh is about $20 per year. So your incremental cost is effectively $0 to $20/year, plus the amortized hardware you alreadyOwn. Scenario B: Tiny VPS. A Hetzner CX21 (2 vCPU, 4 GB RAM, 40 GB disk) runs about €5.83/month, roughly $6–7/month. That comfortably handles a few thousand series with 30-day retention. Add a 20 GB volume for $1–2/month if you need more retention. Scenario C: Small business / community. Suppose you scrape 50,000 series every 15 seconds and want 90-day retention. You need roughly 4 vCPU, 8–16 GB RAM, and 300–500 GB SSD. A Hetzner CX41 (4 vCPU, 16 GB) is about €11.90/month plus a 500 GB volume around €4.76/month, totaling roughly $18–20/month. The equivalent managed Prometheus on a cloud provider would easily run $150–400/month for the same ingest, because managed services charge per million samples. Scenario D: Long-term storage tier. Adding Thanos or Mimir backed by S3-compatible storage for a year of history might add $5–15/month in object storage, depending on volume. Still a fraction of any SaaS alternative. The headline: for the vast majority of self-hosters, Prometheus costs less than a single streaming subscription per month, and often nothing at all beyond hardware you already run.

16. Where Your Data Goes: Data Sovereignty

This is the section the marketing copy skips. By default, every metric Prometheus collects is written to a directory on the machine running the server—typically
/prometheus in the container. No sample leaves that host unless you explicitly configure remote write, federation, or a notification path. Your CPU usage, your endpoint latencies, your disk fill rates: all of it sits on your disk, under your jurisdiction. There are, however, three sovereignty caveats worth internalizing. First, metrics frequently contain sensitive labels. A request counter labeled with user_id, ip, or email means personally identifiable information is now sitting in your TSDB. Scrub such labels with relabeling before they are stored. Second, federation and remote write do export data. If you point remote write at a vendor bucket, that vendor now holds your metrics. Third, Alertmanager notifications contain the alert payload; route them through channels you trust. None of these are Prometheus flaws—they are consequences of how you wire it up. The point is that with Prometheus, you make those decisions, not a vendor's default.

17. Security: Authentication, TLS, and Exposure

The Prometheus server has no built-in user authentication. Anyone who can reach port 9090 can query everything. Therefore the standard, non-negotiable practice is: 1. Bind to localhost or an internal network only. 2. Place it behind a reverse proxy (Caddy, Nginx Proxy Manager, or Traefik all work) that terminates TLS and enforces authentication—HTTP basic auth, or better, your single sign-on via Authelia or Authentik. 3. Never expose the admin
/-/reload or /-/quit endpoints publicly; guard them or disable the lifecycle API. 4. If you enable remote write, use TLS and credentials to the backend. The 3.15 line also added a runtime-reloadable log level, so you can crank up debug logging temporarily without restarting—useful, but keep logs out of places untrusted eyes can read. Reverse proxy in front of Prometheus

18. Performance and Resource Planning

Prometheus is written in Go and is generally efficient, but it is not magic. The two resources that matter are RAM (for the in-memory head and the active chunks) and disk I/O (for the TSDB and compaction). CPU matters mostly during heavy query bursts and compaction. A good planning heuristic: budget 1–2 bytes of RAM per active time series for the head, plus overhead. So 100,000 active series needs roughly 200–400 MB just for the head, before query buffers. Query performance degrades if you ask for huge time ranges over high cardinality, so pre-aggregate with recording rules for dashboards you load often. Recording rules are evaluated on a schedule and store the result as a new series, turning an expensive query into a cheap lookup.

19. Honest Limitations

No tool is perfect, and Prometheus has real edges.
  • Not a long-term store by default. Single-node Prometheus is happiest with retention measured in weeks or a few months. Years of history require an external tier.
  • Pull model struggles with ephemeral, short-lived jobs. If a job lives for less than one scrape interval, Prometheus may never see it. The Pushgateway exists for this, but it introduces its own pitfalls (stale metrics that never expire).
  • Cardinality is a sharp knife. A label with high variance—say, a user ID or a UUID in a URL—can explode your series count from thousands to millions, filling disks and blowing up RAM. This is the single most common way people break Prometheus.
  • No native clustering. HA requires running duplicates; horizontal sharding needs Thanos/Mimir. It is not turnkey.
  • Alerting depends on Alertmanager. If Alertmanager is down, alerts are lost, and Prometheus will not replay them once it recovers unless configured carefully.
  • Global queries across many shards are complex. You need additional components for a true multi-tenant, multi-region view.
  • It is metrics-only. Logs and traces are out of scope; you pair it with Loki or Tempo (or Grafana's stack) for full observability.
  • It demands a learning investment. PromQL, relabeling, and recording rules are not intuitive on day one. Teams that expect a drop-in appliance often struggle until someone sits down with the query documentation. Budget a weekend of experimentation before it feels natural, and treat the first dashboards as disposable practice.
Naming these up front spares you the surprise at 2 a.m. when a dashboard is blank because a target disappeared.

20. Prometheus vs the Alternatives

  • VictoriaMetrics: a drop-in-compatible, often more resource-efficient TSDB that handles higher cardinality and longer retention on less hardware. We have covered it in depth; many operators adopt it specifically when Prometheus outgrows a single node.
  • InfluxDB: a push-based time-series database with its own query language. Strong for application-emitted metrics and events, weaker on the pull-and-alert story that Prometheus nails.
  • Netdata: real-time, per-second metrics with almost zero configuration, excellent for a single host but not built for long-term multi-host querying the way Prometheus is.
  • Grafana Mimir / Thanos: not alternatives so much as scaling layers on top of Prometheus, turning many Prometheuses into one long-term store.
The honest take: if you want a self-hosted, pull-based, label-dimensional metrics system with a massive ecosystem and no vendor lock-in, Prometheus is the default for good reasons. If you outgrow a single node, the escape hatch is also open source. Start small, scrape one host, and let the graphs earn their place on your homescreen.

21. Common Pitfalls

  • Setting a 15-second scrape on 500,000 series because "more data is better," then watching the disk fill.
  • Forgetting to bind the port to localhost and accidentally exposing the unauthenticated UI.
  • Using rate() on a gauge instead of a counter, producing nonsense.
  • Storing PII in labels and later realizing the TSDB is now a compliance liability.
  • Treating the Pushgateway as a Queue, leaving dead metrics forever.
  • Neglecting backups of the TSDB directory, then losing a month of history to a corrupted block.

22. Backup and Disaster Recovery

Because the TSDB is just files, backup is conceptually simple: stop writes (or snapshot), copy the data directory. For live systems, you can use the Admin APIs or file-system snapshots. Many operators back up the volume at the block level daily. The bigger question is whether you need the history—for a homelab, losing 30 days of metrics is annoying but rarely catastrophic, so a weekly snapshot may suffice. For a business using metrics for SLA evidence, treat the TSDB like any production database: scheduled snapshots plus an offsite copy.

23. Upgrading and Version Policy

Prometheus follows semantic versioning with frequent minor releases. The 3.x line is mature and stable; upgrading within 3.x is usually a matter of bumping the image tag and reading the release notes for config changes. Major version jumps (2.x to 3.x) did introduce breaking changes around storage and config, so read the migration guide. Because the project ships as a single static binary or a small container, rolling back is as easy as re-tagging the image—another sovereignty benefit: you are never forced into an upgrade you did not choose.

24. Use Cases: From Homelab to Production

  • Homelab health: scrape your NAS, routers, and containers; alert when disk space drops or a service dies.
  • Self-hosted SaaS monitoring: watch response times and error rates for the other tools you run (Nextcloud, Vaultwarden, your reverse proxy).
  • IoT and sensor networks: pull metrics from ESPHome or MQTT bridges exposed via exporters.
  • Small business SLA: track uptime and latency for customer-facing services without paying per-host monitoring fees.
  • Research and batch jobs: record resource usage of long simulations to understand their profile.
In every case, the data stays yours, the cost is predictable, and the stack is entirely open.

25. Recording Rules and Pre-Aggregation

Recording rules are one of the most underused features in the Prometheus toolbox, and learning them is the difference between a snappy dashboard and one that times out. A recording rule is a PromQL expression evaluated on a fixed schedule—say, every minute—with the result stored back into the TSDB as a new time series. Instead of computing
sum(rate(http_requests_total[5m])) on every page load for every viewer, you compute it once per minute and let the dashboard read the pre-baked series. The syntax lives in a rule file referenced from your config: `yaml groups: - name: request_rates interval: 1m rules: - record: job:http_requests:rate5m expr: sum by (job) (rate(http_requests_total[5m])) - alert: HighErrorRate expr: job:http_requests:rate5m{code="500"} > 1 for: 5m labels: severity: page annotations: summary: "High 500 rate on {{ $labels.job }}" ` Notice that the same file mixes recording rules and alerting rules. The convention of naming recorded series with a level:metric:window pattern (like job:http_requests:rate5m) is a community standard that keeps your metrics list readable. For any dashboard panel that is viewed frequently and queries a wide time range, recording rules are almost always worth the small upfront effort.

26. A Working Multi-Container Stack

Theory is useful, but most people learn by copying a working example. Below is a compact but realistic Compose stack that runs Prometheus, Node Exporter, and Alertmanager together, with data on a named volume and the UI bound only to localhost:
`yaml services: prometheus: image: prom/prometheus:v3.15.0 volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml - ./rules:/etc/prometheus/rules - prom_data:/prometheus command: - --config.file=/etc/prometheus/prometheus.yml - --storage.tsdb.retention.time=30d - --web.enable-lifecycle ports: ["127.0.0.1:9090:9090"] restart: unless-stopped alertmanager: image: prom/alertmanager:v0.28.0 volumes: - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml ports: ["127.0.0.1:9093:9093"] restart: unless-stopped node-exporter: image: prom/node-exporter:v1.9.0 pid: host volumes: - /:/host:ro,rslave command: - --path.rootfs=/host restart: unless-stopped volumes: prom_data: ` Point your scrape_configs at node-exporter:9100 and prometheus:9090, route alerts to alertmanager:9093, and within minutes you have a functioning, fully local monitoring system. Nothing about this stack phones home, and every byte of telemetry lives in prom_data.

27. Mapping Exporters to the Apps You Already Run

If your homelab already runs the usual self-hosted suspects, a wealth of metrics is one exporter away. Scrape your Traefik reverse proxy directly—it exposes a Prometheus endpoint natively—and you can chart request rates and 4xx/5xx ratios per router. Point an Nginx Exporter or the built-in stub status at your web tier. Use the Postgres or MySQL exporter for the databases behind Nextcloud, Vaultwarden, or your wiki. Add cAdvisor to see per-container CPU and memory. WireGuard Exporter reveals tunnel handshakes and traffic. Even your Unbound or Pi-hole DNS resolver can export query counts, which is how you discover that one IoT device is hammering a tracker a thousand times an hour. The practical pattern is incremental: start with Node Exporter on the host, confirm the scrape works, then add one exporter per week as curiosity dictates. Because each exporter is independent, a misbehaving one never takes down the others—Prometheus simply marks its target as down and keeps collecting the rest. This graceful degradation is exactly what you want at 3 a.m. when one container has fallen over.

28. A Pre-Flight Checklist Before You Go Live

Before you declare monitoring "done," walk through a short checklist. First, confirm the UI is not reachable from the public internet; a quick external port scan from your phone's cellular connection should fail. Second, verify that alert routing actually works by triggering a deliberate, low-severity test alert and confirming it arrives in your notification channel. Third, set a retention window you can afford on disk, then check the disk usage graph after a week to validate the estimate. Fourth, scrub any PII from labels using relabel configs. Fifth, take one real backup of the data volume and practice restoring it to a throwaway container, so you know the process before an emergency. Sixth, write down the exact command to reload or stop Prometheus, because the day you need it, you do not want to be reading docs. None of these steps is glamorous, but together they convert a demo into a dependable system.

29. A Concrete Homelab Walkthrough

To make all of this tangible, consider a realistic single-home-server setup. The machine is a mini PC with an Intel N100, 16 GB of RAM, and a 512 GB SSD, already running Docker and a Traefik reverse proxy for a handful of services: Nextcloud, Vaultwarden, a Minecraft server, and a personal wiki. The operator installs Prometheus, Node Exporter, cAdvisor, and Alertmanager as four small containers, all on the same Docker network. Scrape interval is 15 seconds for the host and containers, 30 seconds for the slower application endpoints. Within an evening, the operator has dashboards showing total RAM usage creeping toward 11 GB after the Minecraft server loads a new world, the SSD's SMART-predicted wear level, and a spike in 5xx errors from Nextcloud every time a large upload coincides with a backup job. A single recording rule pre-computes per-service request rates so the overview dashboard loads instantly. Two alerting rules watch disk space and container restarts. When the wiki's container crashes at 2 a.m., Alertmanager fires a webhook to a self-hosted ntfy topic, the operator's phone buzzes, and by the time they are awake the wiki has already restarted via Docker's restart policy. Total incremental cost: zero, because the monitoring containers share the existing host. Total data leaving the house: none, except the ntfy push if ntfy is hosted externally—or, if ntfy is also self-hosted, none at all. This is the quiet superpower of Prometheus in a self-hosting context. It turns "I think the server is slow" into "the P95 latency on the upload endpoint tripled at 01:47 and here is the graph," and it does so entirely on hardware you already own, under a license that will never be revoked. The same operator later discovers a subtler lesson that is worth passing on. The first version of their config scraped every container every 10 seconds and stored a
container_label_ for each Docker label, which inflated the series count from a manageable 4,000 to nearly 40,000 overnight and doubled disk usage. A single relabel rule dropping the unused label prefix—labeldrop: [container_label_]`—brought it back in line within minutes. Experiences like this are why the community repeats the cardinality mantra so often: Prometheus will happily let you shoot yourself in the foot, and the only safeguard is understanding what you are storing before it fills the disk. The good news is that the feedback loop is fast. The targets page in the web UI shows exactly how many series each job produces, so you can catch an explosion on the same day you introduce it rather than three weeks later when the volume is full.

30. Related Reading

If you want to go deeper on the surrounding ecosystem, these pieces from our own catalog pair naturally with Prometheus:

31. Bottom Line

Prometheus earns its place as the backbone of self-hosted observability because it does one thing—collect, store, and query time-series metrics—and does it on hardware you own, with a license that can never be taken from you. It pulls rather than phones home, stores everything locally by default, and scales from a Raspberry Pi to a multi-region federation through open-source companions. The costs are modest: often nothing beyond hardware you already run, or a few dollars a month on a small VPS, versus hundreds for a managed equivalent. The limitations are real but well-understood—watch your cardinality, plan your retention, put authentication in front of the UI, and decide deliberately before you enable remote write. Do those things, and you get a monitoring system whose data sovereignty is a property of the architecture, not a promise in a terms-of-service document.

Comments (0)

No comments yet. Be the first to comment!

Leave a Comment