Operations

See also Installation, the configuration reference and Storage.

Backup

Back up PostgreSQL and the artifacts bucket. The temporary bucket and Grype's database need no backup. For a single node with SQLite or the filesystem store, see Limits and backups.

1. Back up PostgreSQL first: a snapshot, a base backup with WAL, or pg_dump.
2. Copy the artifacts bucket with an S3 sync tool, such as rclone sync.

Garbage collection deletes only content unreferenced for RELIC_GARBAGE_COLLECTION_GRACE_PERIOD (default 1 hour). If the bucket copy takes longer, stop the worker while it runs (kubectl scale deploy/relic-worker --replicas=0). Extra blobs in the copy are removed by garbage collection after a restore.

Keep a copy of RELIC_ENCRYPTION_KEY apart from the backups.

Restore

1. Stop the server and the worker.
2. Restore PostgreSQL to the chosen point.
3. Restore the artifacts bucket from a copy taken after that point. Create an empty temporary bucket.
4. Start the server, then the worker.

Upgrades

Migrations run at startup and only go forward. The server, the worker and relic server migrate log each migration they apply ("database migration applied", with its version and name). Upgrade the server first, then the worker. Downgrades are not supported: a release older than the database's schema refuses to start and names both schema versions, so restore from a backup instead. For Helm, see Upgrades.

Routine operations

Behind a reverse proxy

The server serves HTTPS by default. Behind a proxy that terminates TLS, start it with --no-tls, or let the proxy connect over HTTPS (TLS).

Relic trusts X-Forwarded-Proto and X-Forwarded-For only from the addresses in RELIC_TRUSTED_PROXIES and strips them from other clients. List your proxy there; otherwise Relic generates http:// URLs, sets cookies without Secure, and sees every client as the proxy. Over HTTPS, responses carry Strict-Transport-Security: max-age=31536000.

A client address that sends more than 20 wrong passwords within 10 minutes must wait before its next attempt: one second at first, doubling up to five minutes. Meanwhile it gets 429 with Retry-After. Tokens are not throttled.

Memory and concurrent uploads

In a container with a memory limit, Relic sets Go's soft memory limit to 90% of it, unless GOMEMLIMIT is set. Each upload in flight to object storage holds buffers: up to 40 MiB for S3-compatible storage and Azure Blob Storage, 32 MiB for Google Cloud Storage. Relic runs (soft limit − 256 MiB) ÷ that size uploads at once, between 4 and 64; without a limit, 16. A 1 GiB limit allows 16 to S3 or Azure and 20 to Google Cloud Storage. Uploads to a local directory are not limited. Size server.resources.limits.memory in the chart for the uploads you expect.

Relic Scan's scanners run in the worker's container, so a worker with Syft and Grype sets aside a scan memory budget (RELIC_SCAN_MEMORY_BUDGET; by default half the container's limit, at most 1 GiB) and takes 90% of the rest as its own soft limit, at least 128 MiB. The scanners run one at a time for each 192 MiB of the budget, at most two, and each gets an equal share of it as its GOMEMLIMIT. A vulnerability database update runs alone with the whole budget, never while a scanner or the malicious-package import runs.

The caches Relic keeps in memory size themselves from its soft limit: a sixteenth for each of the Debian, RPM and Helm index caches (at most 256 MiB each), an eighth for rendered npm package documents (at most 128 MiB) and a thirty-second each for npm package indexes and NuGet manifests (at most 32 MiB and 16 MiB).

Give the single-node image (relic server --worker) at least 512 MiB: it keeps about 230 MiB for itself and runs one scanner with 256 MiB. With 1 GiB it runs two scanners of 256 MiB each and keeps about 460 MiB.

Secrets at rest

With RELIC_ENCRYPTION_KEY set, Relic seals these secrets in the database with AES-256-GCM:

• token signing keys
• key pairs, such as the ones that sign Alpine, Debian and RPM indexes and release manifests
• remote repository and replication passwords
• LDAP manager passwords
• OIDC client secrets and SAML keys
• webhook secrets

Secrets stored before the key was set are sealed at startup. On SQLite, Relic overwrites the space deleted rows leave and empties the write-ahead log after sealing, so the plaintext does not stay in the database files; copies of the files made before then still hold it. Set the same key on the server and the worker, and keep a copy apart from the backups. The server does not start if it finds sealed signing keys and has no key or a different one. Without a key, secrets are stored unencrypted and Relic logs a warning.

Signing keys

An Alpine, Debian or RPM repository signs its indexes with the key pair its signingKey names. Alpine and Debian indexes are also signed with the one secondarySigningKey names, so apk and apt accept them with either key. dnf reads only one signature, so repomd.xml.asc holds the signingKey's alone. Both public keys are published: relic package setup apk and apt install both, and repomd.xml.key, the gpgkey that relic package setup dnf writes, holds both. This lets a key be replaced without breaking clients.

To replace debian-signing with debian-signing-2:

1. Add the new key pair as secondarySigningKey and apply the repository. Relic creates the key pair on first use.
2. Run relic package setup again on every apk and apt client, so each one trusts both keys. dnf clients whose gpgkey is the repository's repomd.xml.key URL need nothing.
3. Swap the two names (signingKey: debian-signing-2, secondarySigningKey: debian-signing) and apply. Alpine and Debian clients that still trust only the old key keep working. dnf finds repomd.xml.asc signed by a key it has not imported, downloads gpgkey again and imports the new key, asking first unless it runs with -y. A dnf client whose gpgkey is a local file needs that file replaced with repomd.xml.key before this step.
4. When no apk or apt client trusts only the old key, remove secondarySigningKey and apply.

This command lists the key pairs:

relic api key-pairs

This command deletes a key pair. Relic refuses to delete one that signs a repository, as its signingKey, its secondarySigningKey or the default, and release-bundles, which signs release manifests:

relic api -X DELETE key-pairs/debian-signing

A repository without signingKey signs with relic-rsa (Alpine) or relic-gpg (Debian and RPM). A key pair name is up to 64 letters, digits, ., _ and -, and starts with a letter or digit. Alpine repositories sign with RSA key pairs, Debian and RPM repositories with OpenPGP ones: applying a repository refuses a key pair that is stored as, or that another repository signs with as, the other kind.

Health

• /healthz: the process is alive.
• /readyz: the database and object storage answer, and the database has not refused a write for lack of space in the last minute. It answers 503 with the reason: "database unavailable", "database is full" or "object storage unavailable".
• Administration → System shows both, plus storage, jobs and scanners.

Troubleshooting

Usage reports

Relic counts downloads and deploys per repository and user, and records repository sizes every six hours, for two years. Administration → Usage shows them to administrators and viewers.

Print the last 90 days:

relic usage --days 90