Files
splatoon3.ink/workers/updater/README.md
Matt Isenhower 883ffc0f02 Configure the production updater's archive and compressor
Production writes data snapshots to the archive Space the Node server used
(splatoon3ink-archive) and binds the production compressor for the admin
panel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 14:54:48 -08:00

182 lines
11 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# splatoon3.ink updater Worker
Runs the shared backend code in `app/` (updaters, social posters) on Cloudflare. The local
Node runner (`npm run splatnet`, `npm run social`, …) uses the same code with `dist/` and
`storage/`; this Worker uses R2 instead (see `src/runtime.mjs`).
## How it runs
- **Fast lane** (`Scheduler` Durable Object named `fast`): schedules, gear, co-op, festivals
(including results), and stages, then social posts. Same cadence as the old Node cron:
quick runs at :00–:04, full runs at :05, :10, :15, :20, :30, :45 (15 seconds past the
minute). Quick runs skip translations (`translations: false`), the run's only option.
- **Slow lane** (`Scheduler` named `slow`): X-Rank (one request at a time to save memory),
Splatfest rankings, and seasonal avatar updates, hourly at :20:45. It has no run options.
A run stops starting new work after 10 minutes; if work is left (a large backfill), it queues
a continuation a minute later, up to ten in a row.
- **RegionAuth** (one Durable Object per region, `US`/`EU`/`JP`/`AP`): the only place that logs
in to Nintendo for its region. Both lanes get bullet tokens from it, so they share one token
and never log in at the same time. It renews credentials hourly at :35–:38, unless scheduled
renewals are turned off in the admin panel (tokens are then obtained when an update needs
one). A renewal counts as failed only if it ends without a new bullet token; a step that
failed and was made up for by a later one is kept as a warning.
- **iPad token uploads** (`PUT /bullet-token`): splat-mitmproxy on ghost captures the bullet
token the iPad's SplatNet 3 app receives (an hourly Shortcuts automation opens it) and
uploads it to both the dev and production Workers. It replaces the cached token of the
`TOKEN_UPLOAD_REGION` account (`US`, the North America account) until the next renewal or
upload, and shows on the admin timeline's Nintendo auth row as "iPad upload". It is an extra
source of tokens: nothing depends on it.
- A cron trigger every five minutes only re-arms missing alarms. It never enables a lane.
Failed runs are not retried: the lane's next slot runs again, 1–15 minutes later for the
fast lane. Each lane runs one thing at a time, and one run per alarm; the lanes run
independently of each other. Run limits
(concurrent SplatNet requests, image downloads) are created per run, never shared between
Durable Objects (see `app/data/RunLimits.mjs`).
Before social posts, the fast lane checks that `SITE_URL` serves the data it just published
(screenshots render the public site). A mismatch fails the run, and the next slot tries again.
With `CLOUDFLARE_CACHE_PURGE_*` configured, data JSON is purged from the edge after writes.
With a `SENTRY_DSN` secret, errors go to Sentry (`@sentry/cloudflare`): a failed updater,
social post or archive snapshot (reported by the shared `app/` code through `@sentry/core`), a
lane run that fails outright, a credential renewal that ends without a new bullet token, and
anything uncaught. The Node
runner does not report to Sentry.
## Setup (once per environment)
1. R2 buckets: dev uses `splatoon3-ink-dev-assets` and `splatoon3-ink-dev-private`;
production uses `splatoon3-ink-assets` and `splatoon3-ink-private` (create the private
bucket before the first production deploy; never make it public).
2. Routing: images are served from each bucket's custom domain (`assets.splatoon3.ink`,
`dev-assets.splatoon3.ink`). Cloud Connector rules send `/data` and `/status-screenshots`
on the site hostname to its bucket (production: "Data Assets" for `splatoon3.ink`; dev:
"Dev Data Assets" for `dev.splatoon3.ink`), or the fast lane's published-data check fails.
3. Secrets (`npx wrangler secret put NAME --config workers/updater/wrangler.jsonc --env dev`):
`NINTENDO_TOKEN_US`, `NINTENDO_TOKEN_EU`, `NINTENDO_TOKEN_JP`, `NINTENDO_TOKEN_AP`,
`NXAPI_CLIENT_ID`, `NXAPI_SHARED_SECRET`, `USER_AGENT`, `RUN_TOKEN`, `TOKEN_UPLOAD_TOKEN`
(see iPad token uploads), `ACCESS_TEAM_DOMAIN`
and `ACCESS_AUD` (see Admin panel), `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` (see
Archive), optional `SENTRY_DSN` and
`CLOUDFLARE_CACHE_PURGE_ZONE_ID` / `CLOUDFLARE_CACHE_PURGE_API_TOKEN`. Social account
credentials go on production only (leave them unset in dev).
4. Deploy, check status, run once by hand, then turn on automatic scheduling.
New schedulers start with automatic scheduling off; the setting is stored in the Durable
Object and survives deployments.
## Commands
```sh
npm run updater:test # Worker tests in workerd (also part of npm test)
npm run updater:deploy:dry-run # bundle the dev Worker without deploying
npm run updater:deploy # deploy dev
npm run updater:deploy:production # deploy production
npm run updater:tail # stream dev logs
```
## Control endpoints
All require `Authorization: Bearer $RUN_TOKEN` (they are disabled without `RUN_TOKEN`).
```sh
BASE=https://splatoon3-ink-dev-updater.<subdomain>.workers.dev
AUTH="Authorization: Bearer $RUN_TOKEN"
curl -H "$AUTH" "$BASE/status"
curl -X POST -H "$AUTH" "$BASE/run?lane=fast" # queue a run (202)
curl -X POST -H "$AUTH" "$BASE/run?lane=fast&translations=false&social=false" # quick, data only
curl -X POST -H "$AUTH" "$BASE/run?lane=slow"
curl -X POST -H "$AUTH" "$BASE/schedule?lane=all&enabled=true" # automatic scheduling
curl -X POST -H "$AUTH" "$BASE/schedule?lane=all&enabled=false" # stop it
curl -X POST -H "$AUTH" "$BASE/auth/refresh?region=US" # renew credentials now
curl -X POST -H "$AUTH" "$BASE/arm" # what the cron does
curl -X POST -H "$AUTH" "$BASE/sentry-test" # two test events
```
Manual runs are queued and executed by the lane's alarm; poll `/status` for the result.
A manual run interrupted mid-way is reported as failed rather than replayed, so social posts
are never repeated automatically.
## iPad token uploads
`PUT /bullet-token` takes the body of a SplatNet 3 `/api/bullet_tokens` response, with
`Authorization: Bearer $TOKEN_UPLOAD_TOKEN`. That token only allows uploads (it cannot reach
the control endpoints), and uploads are disabled without it. splat-mitmproxy uses the same
value for both environments, so set the same secret on dev and production.
It answers `{ ok, region, expires }`, or 400 for a body without a `bulletToken`. Do not test
it with a made-up token: whatever is uploaded replaces the region's working one.
## Archive
With `ARCHIVE_DATA` set, every data file an updater publishes is also written to the archive
bucket on DigitalOcean Spaces, at the same keys the Node runner uploads to
(`YYYY/MM/DD/<date>.<time>.<name>.json`, publicly readable). The Worker has no disk, so the
snapshot goes straight to Spaces as part of the update (`app/common/storage/S3ArchiveStorage.mjs`).
- Settings: the vars `ARCHIVE_DATA`, `AWS_S3_ENDPOINT`, `AWS_REGION` and `AWS_S3_ARCHIVE_BUCKET`
in `wrangler.jsonc`, and a Spaces key that can write to that bucket as the secrets
`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
- Dev writes to its own Space (`splatoon3ink-archive-test`). Environments write the same keys,
so they must never share a bucket.
- The fast lane holds its snapshots until after social posts (even failed ones), then uploads
the newest copy of each, so a slow archive bucket never delays a post. The slow lane uploads
as it goes, to avoid holding large X-Rank files in memory.
- A snapshot that cannot be written is a warning in the run's log (and a Sentry event), never
a failed update. Each write is tried three times. Most snapshots are named for the hour and
written again by every run in it, so a short Spaces outage costs nothing; X-Rank snapshots
are stamped to the second and written once, so one that fails all three tries is lost.
- Production writes to the archive Space the Node server used (`splatoon3ink-archive`).
## Admin panel
A Vue app (`admin/`) served by this Worker on its own hostname (`admin.dev.splatoon3.ink`,
`admin.splatoon3.ink`), behind a Cloudflare Access application. The Worker checks the Access
token itself on every request, so nothing is served without it, and the panel is not reachable
on any other hostname.
- **Timeline**: runs of both lanes, credential renewals, social posts and archive
compressions on a time axis that scrolls back through the last 48 hours, and the same things as a list (with what is coming
up next). The two scroll together. Selecting anything in either shows its details and log
lines. A combined log sits below.
- **Controls**: per lane, run now and a switch for automatic scheduling; credential renewal per
region, and one switch for scheduled renewals in every region; archive compression now, or as
a dry run, and a switch for the daily run. Everything that runs on its own has a switch here
(social accounts have theirs on the Social page).
- **Social**: the post log (what was posted, what failed), a posting switch per account, when
each status last went to each account, and a preview of any status with a confirmed post by
hand. A manual run from the panel never posts unless "Post to social" is ticked.
- **Data**: the published data files with their age, a check that the site serves them, and a
cache purge.
Setup, once per environment:
1. Create a Cloudflare Access self-hosted application for the whole admin hostname, allowing
only the owner.
2. Set the secrets `ACCESS_TEAM_DOMAIN` (`<team>.cloudflareaccess.com`) and `ACCESS_AUD` (the
application's audience tag). Without both, every admin request is refused.
3. Deploy. The deploy scripts build the app first (`npm run admin:build`), and the first
deploy attaches the admin hostname as a Worker custom domain.
```sh
npm run admin:dev # the panel with a simulated API on http://localhost:5180 (no Worker)
npm run admin:build # build to admin/dist, which the Worker serves
```
What the panel stores:
- Each lane's `Scheduler` keeps its runs and their log lines for 48 hours (`HISTORY_MS` in
`src/schedule.mjs`).
- Social posts and failures are kept in the fast lane's `Scheduler` (the last 300).
- Each `RegionAuth` keeps its renewals and their log lines for the same 48 hours.
- Account posting switches are in private storage at `settings/social.json`; the Node runner
honours them too.
Daily archive compression (each day's snapshots into one `tar.zst`) is the compressor Worker's
job (`workers/compressor/`), since it needs the `zstd` binary and local disk. The archive rows
and card come from that Worker through the `COMPRESSOR` Durable Object binding. Without the
binding (production, until its compressor is deployed) the panel leaves the archive out. In the Worker tests, `test/fakeCompressor.mjs`
stands in for that Worker.