Persistent storage with the buckets sidecar

Tinfoil offers two ways to persist data. The buckets sidecar (this page) is object storage: encrypted objects in S3, reachable from any number of enclaves and clients, encrypted under keys your end users hold. Encrypted volumes are block storage: a filesystem attached to one enclave at a time, suited to large working sets such as build trees, datasets, and fine-tuning checkpoints, encrypted under a key the deployment owner holds.

By default, Tinfoil Containers have no persistent disk: the enclave filesystem is a ramdisk, and everything on it is lost on every deploy or update. The standard way to persist data is object storage like Amazon S3, but writing plaintext to S3 would hand your users’ data to the storage provider and break the enclave’s privacy guarantees. The tinfoil-buckets-sidecar solves this. It runs as a second container inside your enclave and exposes an S3-compatible API on the enclave’s internal network. On every write it encrypts the object with AES-256-GCM before forwarding it to S3; on every read it fetches the ciphertext, decrypts it, and verifies the authentication tag. Plaintext and keys only ever exist inside the secure enclave. S3 (and anyone with access to the bucket) sees only ciphertext.

End users hold the keys

You operate the enclave, but the data belongs to your end users, so each user’s data is sealed under a key only that user holds. Keys cannot live inside the enclave: it has no persistent disk, so a key generated and kept there would be lost on the first deploy or update, making everything stored under it unrecoverable. Instead, each end user generates a 32-byte AES-256 key on their own device (or derives one from a passphrase using a key derivation function such as HKDF) and sends it along with each request to your app. Your app authenticates the user, then forwards the key to the sidecar in a per-request header. The key exists inside the enclave only for the lifetime of the request and is never persisted, so neither you as the operator nor Tinfoil can decrypt the stored data.

Each user’s key is the confidentiality boundary for their data. A user who loses their key permanently loses everything stored under it, so client applications must store keys durably (device keychain, passkey-wrapped backup, or similar).

Tutorial

This walks through adding encrypted S3 storage to an existing container deployment. For a complete runnable deployment, see the persistent storage example.

1. Create a bucket and IAM credentials

Create an S3 bucket and an IAM user whose credentials the sidecar will use. IAM is what controls which buckets are accessible, so attach a policy scoped to the bucket(s) the sidecar should reach:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": ["arn:aws:s3:::YOUR-BUCKET/*"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:GetBucketLocation"
      ],
      "Resource": ["arn:aws:s3:::YOUR-BUCKET"]
    }
  ]
}

Add the credentials as two secrets: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Use tinfoil secret create SECRET_NAME --value-file ./path for organization scope. For repository scope, add them in the repository’s Repository secrets section under Containers → Repositories in the dashboard. These credentials only ever touch ciphertext.

2. Add the sidecar to your config

Run the sidecar alongside your app in multitenant mode and put both on a shared network. The sidecar needs egress to reach S3; your app reaches it at http://buckets:9000 by container name:

tinfoil-config.yml

cvm-version: "<CVM_VERSION>" # TODO: Use latest version from https://github.com/tinfoilsh/cvmimage
cpus: 4
memory: 8192

networks:
  storage:
    egress: open

containers:
  - name: "app"
    image: "ghcr.io/myorg/my-app:v1.0.0@sha256:abc123..."
    networks: [storage]
    env:
      - BUCKETS_URL: "http://buckets:9000"

  - name: "buckets"
    image: "ghcr.io/tinfoilsh/tinfoil-buckets-sidecar@sha256:def456..."
    networks: [storage]
    env:
      - PORT: "9000"
      - AWS_REGION: "us-east-2"
      - MULTITENANT: "true"
    secrets:
      - AWS_ACCESS_KEY_ID
      - AWS_SECRET_ACCESS_KEY

shim:
  upstream-port: 8080
  paths:
    - /health
    - /api/*

3. Have each end user generate a key

Client applications generate 32 random bytes per user, base64-encode them, and store the key durably on the user’s side. The equivalent of:

openssl rand -base64 32

Alternatively, derive the key from a passphrase the user already holds, so there is nothing extra to store. The client sends this key to your app with each request that reads or writes stored data. The connection to the enclave is attested and encrypted end-to-end, so the key is only ever visible inside the enclave.

4. Make per-user requests from your app

For each authenticated request, your app forwards two headers to the sidecar: the user’s key, and a tenant id your app derives from the user’s verified identity (never from user input). Any S3 SDK works, configured with the sidecar as the endpoint, path-style addressing, and sequential single-request transfers; credentials can be any throwaway values, since SDKs refuse to send unsigned requests:

import boto3
from botocore.config import Config
from boto3.s3.transfer import TransferConfig

s3 = boto3.client(
    "s3",
    endpoint_url="http://buckets:9000",
    region_name="us-east-2",
    aws_access_key_id="tinfoil-sidecar",
    aws_secret_access_key="tinfoil-sidecar",
    config=Config(s3={"addressing_style": "path"}),
)

# Attach the authenticated user's tenant id and key to every request
def tenant_headers(request, **kwargs):
    request.headers["X-Tinfoil-Tenant-Id"] = f"user-{verified_user_id}"
    request.headers["X-Tinfoil-Encryption-Key"] = user_key_base64

s3.meta.events.register("before-send.s3", tenant_headers)

# Force sequential multipart uploads and disable ranged-GET downloads
tc = TransferConfig(max_concurrency=1, multipart_threshold=5 * 1024**4)

s3.upload_file("./upload.bin", "YOUR-BUCKET", "docs/report.pdf", Config=tc)
s3.download_file("YOUR-BUCKET", "docs/report.pdf", "./roundtrip.pdf", Config=tc)

The bucket comes from the request path (s3://bucket/key), so one sidecar can serve multiple buckets; reachability is enforced by the IAM policy from step 1. Equivalent Java and AWS CLI setups are in the sidecar’s guides.

How multitenancy works

Objects are transparently namespaced under <tenantId>/ in the backing bucket, so one tenant can never address another’s objects. Listing and reads within a tenant see plain object keys with the prefix stripped.
The sidecar trusts the tenant ID and key it receives in the headers; your app is the authentication layer. Derive the tenant ID from the user’s verified identity, never from client input.
Key and tenant id are decoupled, so a user can rotate keys without moving data. The caller tracks which key decrypts which object; the wrong key returns 400 DecryptionFailed.

If the enclave stores only its own operational data rather than per-user data, the sidecar also runs in single-key mode: omit MULTITENANT and supply one ENCRYPTION_KEY secret instead. See the sidecar README.

Differences from normal S3

Because the sidecar maintains streaming AES-GCM cipher state, a few S3 behaviors differ:

Behavior	Constraint
Addressing	Path-style only (`forcePathStyle` or equivalent)
Multipart uploads	Sequential parts (`max_concurrency=1`); non-last parts must be multiples of 16 bytes
Ranged GETs	Not supported; download whole objects
GET responses	Buffered in memory to verify the authentication tag; 1 GiB default, configurable via `BUFFER_SIZE` up to 64 GiB
ETags	Synthetic, so not usable for client-side integrity checks

High-level SDK transfer managers satisfy the multipart constraints with the configuration shown above. For objects larger than the buffer cap, the sidecar supports a streaming mode (DANGEROUS_DELAYED_AUTH) that requires a verifying client; see the design notes before using it.

Relevant repositories

tinfoil-buckets-sidecar: the sidecar itself, with CLI and SDK guides
tinfoil-persistent-storage-example: a minimal deployable example

Getting Started

Model catalog

Tinfoil SDKs

Tinfoil Chat

Local Proxy

Tinfoil Containers

Guides

Verification & Attestation

Tutorials

Admin API

Resources

Persistent storage with the buckets sidecar

End users hold the keys

Tutorial

1. Create a bucket and IAM credentials

2. Add the sidecar to your config

3. Have each end user generate a key

4. Make per-user requests from your app

How multitenancy works

Differences from normal S3

Relevant repositories

​End users hold the keys

​Tutorial

​1. Create a bucket and IAM credentials

​2. Add the sidecar to your config

​3. Have each end user generate a key

​4. Make per-user requests from your app

​How multitenancy works

​Differences from normal S3

​Relevant repositories

End users hold the keys

Tutorial

1. Create a bucket and IAM credentials

2. Add the sidecar to your config

3. Have each end user generate a key

4. Make per-user requests from your app

How multitenancy works

Differences from normal S3

Relevant repositories