mirror of
https://github.com/misenhower/splatoon3.ink.git
synced 2026-11-18 16:58:18 -06:00
A credential renewal now counts as failed only when it ends without a new bullet token. A step that failed and was made up for by a later one (znca briefly unavailable, say) is kept as a warning and no longer goes to Sentry. Everything that runs on its own now has a switch in the admin panel: scheduled renewals (one switch for every region) and the daily archive compression join the lanes' scheduling switches. The compressor's starts from AUTOMATIC_SCHEDULING_ENABLED: on in dev, off in production, so the production compressor can be deployed before spectre stops. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
splatoon3.ink archive compressor Worker
Once a day, bundles each finished day of data snapshots in the archive Space (DigitalOcean
Spaces) into YYYY/MM/YYYY-MM-DD.tar.zst plus a .manifest.json, using the same code as
npm run data:archive:compress (app/data/ArchiveCompressor.mjs). The updater Worker writes
the snapshots; this Worker never touches R2.
How it runs
- A cron trigger at 00:30 UTC asks the
CompressorDurable Object (namedarchive) to start a run. Days still being written (today, or anything changed in the last 30 minutes) are skipped, and a run compresses at most 10 days. - The work happens in a Cloudflare Container (
Dockerfile, entry pointcontainer/compress.mjs) because it needstarandzstd. It runs onstandard-1(½ vCPU, 4 GiB, 8 GB disk); a production day (~430 MB of snapshots) takes under a minute. - What the container prints (the
[Archive Compressor]lines) goes to stdout, which is in Workers Logs (notwrangler tail). The container also serves it on port 8080 (container/logServer.mjs), and once the work is done, its exit code too. - While a run goes on, an alarm every three seconds fetches the new lines (up to 500 are kept with each run) and checks on the container. The run ends when the container reports its exit code; the container waits (up to two minutes) until that report has been fetched, then exits. The object keeps the last 30 runs (exit code, error, timing, log), and only one run happens at a time.
- Nothing depends on the Durable Object staying in memory: Cloudflare can evict it mid-run (it did, before the alarm took over), and the next alarm carries on from what is stored.
- A container that stops without reporting (a crash, running out of memory) fails the run with "The container stopped before it reported how the run went"; its log shows how far it got. A run past three hours is stopped.
- The updater's admin panel shows the runs (timeline and Controls), can start one, and turns
the daily run on or off, through its
COMPRESSORbinding to this Worker's Durable Object. Until it is turned on or off there, theAUTOMATIC_SCHEDULING_ENABLEDvar decides: on in dev, off in production.
The compressor doesn't delete the source snapshots; npm run data:archive:prune does that,
by hand.
Setup (once per environment)
- Docker or a Docker-compatible CLI (OrbStack) must be running:
wrangler deploybuilds the image from the repository root and pushes it to Cloudflare's registry. - Secrets (
npx wrangler secret put NAME --config workers/compressor/wrangler.jsonc --env dev):RUN_TOKEN,AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY(a Spaces key for the archive bucket), optionalSENTRY_DSN(the Durable Object reports each failed run to Sentry; the container itself does not). - The bucket is a var: dev uses
splatoon3ink-archive-testand must never point at the productionsplatoon3ink-archive. - Production starts with the daily run off, because spectre also compresses at 00:30 UTC. Turn it on from the admin panel once spectre has stopped.
Commands
npm run compressor:deploy:dry-run # build the image and bundle the dev Worker
npm run compressor:deploy # deploy dev
npm run compressor:deploy:production # deploy production
npm run compressor:tail # stream dev Worker logs (not the container's output)
With RUN_TOKEN in workers/compressor/.dev.vars:
AUTH="Authorization: Bearer $RUN_TOKEN"
BASE=https://splatoon3-ink-dev-compressor.ike.workers.dev
curl -H "$AUTH" "$BASE/status" # the run in progress and recent runs
curl -H "$AUTH" "$BASE/run?id=RUN_ID" # one run with its log lines
curl -X POST -H "$AUTH" "$BASE/run" # compress now (202, or 409 if running)
curl -X POST -H "$AUTH" "$BASE/run?dryRun=true" # list what would be compressed
curl -X POST -H "$AUTH" "$BASE/run?maxDays=30" # catch up on more days
curl -X POST -H "$AUTH" "$BASE/schedule?enabled=false" # turn the daily run off (or on)
To try the image locally (OrbStack), against dev's bucket only:
docker build --platform linux/amd64 -f workers/compressor/Dockerfile -t splatoon3ink-compressor .
docker run --rm --platform linux/amd64 -e AWS_S3_ENDPOINT -e AWS_REGION -e AWS_ACCESS_KEY_ID \
-e AWS_SECRET_ACCESS_KEY -e AWS_S3_ARCHIVE_BUCKET=splatoon3ink-archive-test -e DRY_RUN=true \
splatoon3ink-compressor