---
title: "S3 Storage"
description: "Store redacted logs in AWS S3, Cloudflare R2, or MinIO and inspect them in Studio with bounded reads."
canonical_url: "https://www.blyp.dev/docs/s3"
markdown_url: "https://www.blyp.dev/docs/s3.md"
last_updated: "1970-01-01"
x_farming_labs_generated_preamble: true
---

# S3 Storage
URL: /docs/s3
LLM index: /llms.txt
Description: Store redacted logs in AWS S3, Cloudflare R2, or MinIO and inspect them in Studio with bounded reads.

# S3 Storage

Added in `@blyp/core@0.1.35`, `destination: "s3"` writes logs to S3-compatible object storage. Use AWS S3, Cloudflare R2, or MinIO as a primary destination, with outbound [connectors](/docs/connectors) enabled alongside it when needed.

The sink requires Node.js 20+ or Bun. Browser and edge clients forward through a server using Blyp's existing ingestion APIs.

## Install and configure

Install Core and the optional S3 SDK:

```bash
bun add @blyp/core@0.1.35 @aws-sdk/client-s3
```

```ts
// blyp.config.ts
import { defineConfig } from "@blyp/core/config";

export default defineConfig({
  destination: "s3",
  s3: {
    bucket: "production-logs",
    prefix: "orders/production",
    region: "eu-west-1",
    delivery: { buffer: "memory" },
  },
});
```

Create the bucket before starting the logger. Credentials use the SDK's server-side chain, including environment variables, profiles, SSO, and workload roles. Executable configuration can also provide a `credentials` object or an asynchronous credential provider. Keep credentials in environment variables or a server-side provider.

| Provider | Configuration |
| --- | --- |
| AWS S3 | Set the bucket region or use SDK environment/profile defaults. Use a general-purpose bucket with directly readable objects. |
| Cloudflare R2 | Set `region: "auto"` and `endpoint: "https://<account-id>.r2.cloudflarestorage.com"`. Supply R2 credentials on the server. |
| MinIO | Set its S3 API endpoint, a region such as `"us-east-1"`, and `forcePathStyle: true`. Local development can use HTTP. |

Other providers must support SigV4, ordered `ListObjectsV2` pagination, conditional `GetObject` with `If-Match`, and conditional `PutObject` with `If-None-Match: *`.

## Batching and delivery

Records pass through Blyp redaction before storage. Batches split by UTC event hour and seal at the first configured threshold:

| Delivery option | Default |
| --- | --- |
| `flushIntervalMs` | 30,000 ms |
| `maxBatchBytes` | 1 MiB decoded data |
| `maxBatchRecords` | 5,000 |
| `maxQueueBytes` | 16 MiB |
| `maxRecordBytes` | 1 MiB, including the newline |
| `concurrency` | 2 |
| `requestTimeoutMs` | 10,000 ms |
| `maxAttempts` | 5 total attempts |
| `initialBackoffMs` / `maxBackoffMs` | 500 / 30,000 ms |
| `buffer` | `"memory"` |

Set these options under `s3.delivery`. Each healthy batch needs one PUT. Blyp writes immutable gzip NDJSON objects, without reading old objects to append or creating one object per log. More writer processes and frequent explicit flushes create more small objects.

Retries use exponential backoff with jitter and retain the original key and compressed payload. Conditional creation makes ambiguous successful uploads safe to retry. Throttling and transient network/server errors retry; permanent errors pause dispatch. Oversized records and queue overflow reject new records.

Ordinary logging calls do not throw. Inspect `result.destinations.sink` from `logger.flush()` for delivery failures. Failures remain latched for the logger's lifetime so a later flush cannot falsely acknowledge lost records. Correct the destination and recreate the logger after a failure.

Memory buffering loses unsent data when the process exits. Framework adapters preserve batching rather than flushing after every request. Serverless handlers must explicitly await `logger.flush({ timeoutMs: ... })` before freezing or exiting. A caller's flush timeout leaves pending work intact; each HTTP attempt has its own timeout.

Selecting S3 avoids file logging and project-bootstrap writes. The default connector SQLite queue is disabled unless explicitly enabled; S3's own buffer is configured separately.

## Recover with a persistent spool

Use SQLite buffering and a path on persistent storage for restart recovery:

```ts
import { defineConfig } from "@blyp/core/config";

export default defineConfig({
  destination: "s3",
  s3: {
    bucket: "production-logs",
    prefix: "orders/production",
    region: "eu-west-1",
    delivery: {
      buffer: "sqlite",
      spoolPath: "/persistent/blyp/s3.db",
      maxSpoolBytes: 512 * 1024 * 1024,
    },
  },
});
```

SQLite mode requires Bun's built-in SQLite or a Node runtime with `node:sqlite`. The default path is `~/.blyp/s3/<target-hash>.db`. Durability begins when the worker commits an accepted record; `write()` is asynchronous, and `flush()` waits for accepted work. The spool stores redacted records and compressed batches. Its 512 MiB default logical payload limit excludes SQLite and WAL overhead.

A restart recovers unsealed records and pending batches. Expiring leases let abandoned uploads become eligible again, and failed batches remain on disk.

```ts
import { getS3SpoolStatus, retryS3Spool } from "@blyp/core/s3";

console.log(await getS3SpoolStatus("/persistent/blyp/s3.db"));
// { records, pending, failed, bytes }
await retryS3Spool("/persistent/blyp/s3.db");
```

Recreate a failed logger after correcting the cause and retrying the spool to resume delivery and clear its failure latch.

## Read logs in Studio

Use `@blyp/cli@0.1.57` or newer. Open a project with S3 configuration, or supply a source directly:

```bash
blyp studio ./my-project
blyp studio --source s3://production-logs/orders/production --region eu-west-1
```

For R2, pass `--region auto --endpoint https://<account-id>.r2.cloudflarestorage.com`. For MinIO, pass its endpoint and `--force-path-style`. Direct source mode needs no local project; source-code context requires an attached project. Stop an existing Studio server before changing server-side source options.

Opening Studio makes no bucket requests. Select a time range of at most seven days, then choose **Load logs**. Studio lists only the selected UTC hour prefixes. Each initial load or **Load more** adds one read grant:

| Allowance per grant | Default |
| --- | --- |
| Object downloads | 16 MiB |
| LIST attempts | 20 |
| GET attempts | 64 |
| Analyzed records | 20,000 |
| Decoded data | 25 MiB |
| Maximum grants | 5 |

Override these through `studio.s3` using `downloadBytes`, `listAttempts`, `getAttempts`, `records`, `decodedBytes`, and `maxGrants`. Retries count toward the allowances. Reaching a limit marks results partial; **Load more** resumes without duplicates. Charts, search, details, facets, and AI context share the loaded snapshot, so their counts reflect loaded data until it is complete.

Automatic refresh shares the existing allowance and stops when it runs out. Hidden pages do not poll. Immutable objects are cached; download counters describe observed payload bytes rather than a provider bill. Studio limits each object to 25 MiB compressed and decoded, so keep custom batch sizes within the decoded limit.

Credentials and raw S3 configuration stay on the server. An S3 source URL does not make the bucket public or share credentials with teammates.

## Permissions and object format

Writers need `s3:PutObject` on `<prefix>/v1/*`. Readers need `s3:ListBucket` restricted to that prefix and `s3:GetObject` on those objects. Use separate writer and reader roles. Blyp does not create buckets, delete logs, or change policies or lifecycle rules during normal operation; configure retention and encryption through the provider.

```text
<prefix>/v1/YYYY/MM/DD/HH/<reverse-max-ms>-<min-ms>-<max-ms>-<batch-id>.ndjson.gz
```

Objects contain newline-terminated `LogRecord` JSON values plus metadata for format version, timestamp bounds, record count, and decoded bytes. Studio reads this versioned format, without arbitrary NDJSON discovery. Archived objects requiring restore and S3 directory buckets are outside the supported format.

The v0.1.35 release passed local MinIO write/read validation. Hosted AWS and R2 validation was unavailable. See the [upstream S3 guide](https://github.com/Blyphq/blyp/blob/v0.1.35/docs/connectors/s3.md) for the provider validation script and full storage contract.

## Sitemap

See the full [sitemap](/sitemap.md) for all pages.
Docs-scoped sitemap: [/docs/sitemap.md](/docs/sitemap.md).
Well-known sitemap: [/.well-known/sitemap.md](/.well-known/sitemap.md).