★ Enterprise Edition — not included in the Community Edition. Compare editions

Prometheus¶

Prometheus scrapes metrics from every Mailyte worker and stores them as time-series data — the foundation for dashboards and alerts.

How It Fits In¶

Prometheus (prom/prometheus:v2.51.0) runs as a container in the main compose stack. Every 15 seconds it pulls metrics from each target's /metrics endpoint. Grafana reads from Prometheus. Alertmanager receives alerts from Prometheus.

graph LR
    RS[Rspamd :11334/metrics] --> PR[Prometheus :9090]
    API[api :8080/metrics] --> PR
    WK[17 worker /metrics endpoints] --> PR
    DB[mysql-exporter :9104] --> PR
    RD[redis-exporter :9121] --> PR
    PR --> GR[Grafana :3000]
    PR --> AM[Alertmanager :9093]

Scrape Configuration¶

The config lives in monitoring/prometheus/prometheus.yml and is mounted read-only into the container. Targets use internal container ports, not the host-mapped ports:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 10s

rule_files:
  - "/etc/prometheus/rules/*.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Mail services
  # (the rspamd job is commented out in the real file since 2026-08-31 —
  # verified live that this rspamd build's controller serves no /metrics
  # at all (404; /stat is auth-gated), so the job scraped nothing)

  # Worker services (a selection -- see the file for the full list)
  - job_name: 'api'
    metrics_path: /metrics
    static_configs:
      - targets: ['api:8080']        # container listens on 8080, host-mapped to 8083

  - job_name: 'monitoring'
    metrics_path: /metrics
    static_configs:
      - targets: ['monitoring:8085']

  - job_name: 'archiver'
    metrics_path: /metrics
    static_configs:
      - targets: ['archiver:8083']   # container listens on 8083, host-mapped to 8089

  # Infrastructure exporters
  - job_name: 'mysql'
    static_configs:
      - targets: ['mysql-exporter:9104']

  - job_name: 'redis'
    static_configs:
      - targets: ['redis-exporter:9121']

The full file covers 20 worker jobs: api, webhooks, rate_limiter, monitoring, tracking, analytics, dashboard, archiver, encryption, queue_manager, rag, storage_usage, delivery_optimizer, jmap, migration, autoconfig, caldav, templates, url_protection, and oauth — every target on its container port (the analytics and rag jobs used to point at their host-mapped ports 8087/8091 and scraped nothing; fixed 2026-08-30 to 8085/8090).

What the Workers Expose¶

Every FastAPI worker uses the shared collector in shared/metrics.py. All metric names are prefixed with the service name, and histograms are exposed as summaries (_count, _sum, and quantile series — not _bucket).

Common series for each service <svc>:

Service-specific series worth knowing:

Infrastructure exporters¶

mysql-exporter and redis-exporter expose the standard upstream series:

Useful PromQL Queries¶

# Which targets are up right now?
up

# API request rate (per second)
rate(api_http_requests_total[5m])

# API error ratio
sum(rate(api_http_errors_total[5m])) / sum(rate(api_http_requests_total[5m]))

# API p95 response time (summary quantile -- pre-computed, no histogram_quantile needed)
api_http_request_duration_seconds{quantile="0.95"}

# MySQL connections as a fraction of max
mysql_global_status_threads_connected / mysql_global_variables_max_connections

# Redis cache hit rate
rate(redis_keyspace_hits_total[5m])
/ (rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m]))

# Backup freshness (should stay under 93600 for type="full")
monitoring_backup_age_seconds

# Archive spool backlog (should be 0)
archiver_archive_spool_depth

Data Retention¶

The compose file pins retention to 30 days:

prometheus:
  command:
    - '--config.file=/etc/prometheus/prometheus.yml'
    - '--storage.tsdb.path=/prometheus'
    - '--storage.tsdb.retention.time=30d'
    - '--web.enable-lifecycle'

--web.enable-lifecycle is on, so a config change can be applied without recreating the container:

curl -X POST http://localhost:9090/-/reload

Data lives in the prometheus_data named volume.

Checking Prometheus Health¶

# Is Prometheus up?
curl http://localhost:9090/-/healthy

# Check scrape targets and their status
curl -s http://localhost:9090/api/v1/targets | python3 -m json.tool

# Run a quick query
curl -s 'http://localhost:9090/api/v1/query?query=up' | python3 -m json.tool

If a target shows as DOWN, first check whether it's one of the two jobs with no deployed exporter (postfix, dovecot) — that is expected. Otherwise check that the container is running and healthy. See Troubleshooting for more.