Monitoring

Relic exports Prometheus metrics, OpenTelemetry traces and structured logs. deploy/observability/ holds seven Grafana dashboards and the alert rules, which the Helm chart and the Compose stack can install: see Kubernetes: Monitoring and Compose: Observability profile. Settings are in the configuration reference.

Metrics

The server and the worker serve /metrics on a separate port (RELIC_METRICS_LISTEN_ADDRESS, default :8091), never on the API port. It has no authentication: keep it inside your network. RELIC_PROFILING_ENABLED=true adds Go's profiler at /debug/pprof/ on the same port.

Series start with relic_; the metrics reference lists them. They cover:

Labels never hold a user, path, digest, token or address. RELIC_METRICS_REPOSITORY_LABEL=true adds the repository name to the artifact, upstream, cache and replication series.

Traces

Set OTEL_EXPORTER_OTLP_ENDPOINT (OTLP over HTTP) to trace every request except health probes, down to database, object storage, upstream and LDAP calls. Queued jobs get a span linked to the request. Relic honours and forwards traceparent.

OTEL_TRACES_SAMPLER sets which traces are kept. The Helm chart keeps 10% of new traces and follows a traced caller; Compose keeps all.

Logs

Relic writes JSON lines to stderr at RELIC_LOG_LEVEL; RELIC_LOG_FORMAT=text suits a terminal. OTEL_LOGS_EXPORTER=otlp also sends records over OTLP.

• Request lines carry request_id, also returned as X-Request-Id, and, when traced, trace_id and span_id.
• A request still running after 10 seconds logs a slow request warning.
• A failed telemetry export logs telemetry export failed and counts in relic_telemetry_errors_total.

Dashboards

Each has datasource, namespace and instance variables. A latency panel's exemplar opens its trace in Tempo.

Alerts

Runbooks

Each alert links to its section below.

RelicDown

1. Check the pods or containers: kubectl get pods -l app.kubernetes.io/component=server or docker compose ps relic-server.
2. A server in a restart loop usually cannot reach PostgreSQL or object storage, or cannot apply a migration. Read its last log lines.
3. If it runs, check that Prometheus reaches port 8091 and that the ServiceMonitor selects the Service.

RelicHighErrorRate

1. On Relic / Overview, find the failing routes in 5xx ratio and Slowest routes, and open an exemplar's trace.
2. Filter the log panel by the trace ID; the error field names the cause.
3. If RelicDBPoolExhausted or RelicStorageErrors fire, start there.

RelicDownloadLatencySLO

Critical burns the budget fast (1 hour of data); warning burns it slowly (6 hours).

1. On Relic / Packages and repositories, check whether slow downloads come from remote repositories (upstream latency) or local ones.
2. On Relic / Storage and garbage collection, check get latency.
3. On Relic / Database, check pool waits and slow queries.

RelicUpstreamErrors

Relic still serves cached content.

1. Check that the upstream answers from Relic's network: egress, proxy, DNS.
2. While an upstream is down, you can set its remote repositories offline.

RelicDBPoolExhausted

1. On Relic / Database, find the queries holding connections in Time spent by query.
2. RELIC_DATABASE_STATEMENT_TIMEOUT cuts long queries. Check PostgreSQL locks and I/O on the CloudNativePG dashboard.
3. Add server replicas, or raise RELIC_DATABASE_MAX_CONNECTIONS (default 20). Replicas × pool size, plus the worker's pool, must fit PostgreSQL's max_connections.

RelicJobBacklog

1. Check that a worker runs (relic_build_info{component="worker"}).
2. On Relic / Jobs and worker, check which kinds wait and whether they fail.
3. Long scans occupy the scan queue. Add worker replicas if the backlog grows.

RelicScanBacklog

The cause label says which condition fired.

1. Backlog: see Relic / Relic Scan. Each worker runs at most two scans at a time. Add worker replicas, or check the worker logs for scan failures.
2. Old database: the worker updates it every 12 hours, which needs outbound access or, air-gapped, an imported archive. Check the scan_db job's errors on the System page.

RelicCVEFeedStale

1. The hourly cve_refresh job reads the EPSS and KEV feeds; its last error is on the System page.
2. Air-gapped sites point RELIC_CVE_EPSS_URL and RELIC_CVE_KEV_URL at a generic repository. Update the files there.

RelicGCFailing

1. The gc job's error is on the System page and in the worker's logs.
2. Usually object storage refuses deletes (credentials or bucket policy). See RelicStorageErrors.

RelicReplicationFailing

1. The last error is on the repository's replication page.
2. Check the target's URL and credentials, and that the worker reaches it.

RelicStorageErrors

Uploads and downloads fail with these operations.

1. On Relic / Storage and garbage collection, check which operations fail.
2. Check the S3 endpoint (SeaweedFS pods or the provider's status), the credentials and free space.

RelicContentIntegrity

Relic refuses to serve missing or damaged content until it is restored.

1. A read logs stored content is missing or stored content does not match its checksum, with the SHA-256, repository and path.
2. Run the storage check and print its report, which lists each missing or damaged object:

   relic storage check --wait
   

3. Upload the file again with any client; Relic replaces the stored object.
4. If many objects are affected, check the store: a lost volume, a lifecycle rule deleting from artifacts, or a restore from an older backup.

RelicDiskSpaceLow

The volume holds the SQLite database (volume="database") or the filesystem object store (volume="object_storage"). Writes fail when it fills.

1. relic_db_sqlite_file_bytes is the database's share; the rest is content.
2. Empty the trash and lower RELIC_TRASH_RETENTION; garbage collection then reclaims unreferenced content after RELIC_GARBAGE_COLLECTION_GRACE_PERIOD.
3. Grow the volume (in Helm, data.size on an expandable storage class), or move to S3 and PostgreSQL with relic server database copy.

RelicServerCertificateExpiring

source="operator" is your certificate (RELIC_TLS_CERTIFICATE_FILE, RELIC_TLS_KEY_FILE); source="generated" is one Relic issued.

1. Renew your certificate and replace the files; the server reloads them without a restart.
2. With cert-manager, check the Certificate's status and the issuer's events.
3. Relic renews a generated certificate 30 days before expiry. If the alert fires for one, the server cannot write RELIC_TLS_DIRECTORY: check the logs and its permissions.