Storage

Other S3-compatible services

Checklist and settings for Ceph RGW, Wasabi, DigitalOcean Spaces and other S3-compatible stores.

Checklist

Path-style requests (<endpoint>/<bucket>/<key>).
Signature Version 4 with a static access key and secret key.
Multipart uploads with parts of 5 MiB or more, all but the last of one size.
Server-side copy between two buckets with one key pair: CopyObject, and UploadPartCopy over 5 GiB.
Range reads on GetObject.
ListObjectsV2 on both buckets.
HeadObject on a missing key answers 404, not 403.
The AWS SDK's default upload checksums (CRC32), or RELIC_OBJECT_STORAGE_CHECKSUM=when_required.
A way to expire temporary objects after a day.
Deletes free space, or a rule removes old versions.
A publicly trusted certificate on https://, or http:// on a private network.

Common settings

Service	`RELIC_OBJECT_STORAGE_ENDPOINT`	`RELIC_OBJECT_STORAGE_REGION`
Ceph Object Gateway (RGW)	`http://<rgw-host>:7480`, or your TLS front end	The zonegroup's API name
Wasabi	`https://s3.<region>.wasabisys.com`	The bucket's region, such as `eu-central-1`
DigitalOcean Spaces	`https://<region>.digitaloceanspaces.com`	The region, such as `fra1`
Garage	`http://<garage-host>:3900`	The `s3_region` from Garage's configuration

Create two buckets and a key pair with read, write, list and delete access to both, then set:

RELIC_OBJECT_STORAGE_ENDPOINT=<endpoint>
RELIC_OBJECT_STORAGE_REGION=<region>
RELIC_OBJECT_STORAGE_ARTIFACTS_BUCKET=<artifacts-bucket>
RELIC_OBJECT_STORAGE_TEMPORARY_BUCKET=<temporary-bucket>
RELIC_OBJECT_STORAGE_ACCESS_KEY=<access-key>
RELIC_OBJECT_STORAGE_SECRET_KEY=<secret-key>

Helm:

seaweedfs:
  enabled: false
existingCredentialsSecret: relic          # holds the S3 key pair, adminPassword and encryptionKey
objectStorage:
  endpoint: <endpoint>
  region: <region>
  artifactsBucket: <artifacts-bucket>
  temporaryBucket: <temporary-bucket>

Testing a new service

/readyz answers ok.
Upload a small file and one of a few hundred megabytes with relic artifact upload. Download both with relic artifact download and compare checksums.
Delete a file and empty it from the trash can. After the grace period (RELIC_GARBAGE_COLLECTION_GRACE_PERIOD, which you can shorten for the test) and the next garbage collection run, its object is gone from the artifacts bucket.
If you store files over 5 GiB, upload one to test UploadPartCopy.

When a step fails, the server log shows the S3 error.

Edit this pageorReport an issue

Backblaze B2

Backblaze B2 as object storage, through its S3-compatible API.

Filesystem

Store artifact content in a local directory instead of an S3 service, on a single node.