783 Commits

Author SHA1 Message Date
Matt Isenhower
aa71bf2eda Hold a lane run while an earlier one may still be going; log data changes
Some checks failed
Build frontend / build (24.x) (push) Has been cancelled
Lint / lint (push) Has been cancelled
Tests / test (24.x) (push) Has been cancelled
Cloudflare replaced the fast lane's Durable Object during the 00:00 UTC run.
The alarm restarted on a new instance while the old one carried on posting,
and the old run's results were lost because it could no longer write to its
own storage. Like the old cron, a run now takes a lock in the private R2
bucket (which a replaced instance can still write), refreshes it every 10
seconds and deletes it at the end; a run that finds a fresh lock waits and
checks again in 15 seconds. A scheduled run that never saved its results is
recorded as cut off.

Each updater now reports whether its published file changed, in its log
line and in the run's results, to see when fresh data really arrives at the
top of the hour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 16:23:40 -08:00
Matt Isenhower
883ffc0f02 Configure the production updater's archive and compressor
Production writes data snapshots to the archive Space the Node server used
(splatoon3ink-archive) and binds the production compressor for the admin
panel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 14:54:48 -08:00
Matt Isenhower
66602492a6 Merge remote-tracking branch 'origin/main' into cloudflare-workers 2026-11-15 14:44:33 -08:00
Matt Isenhower
cf9c30ad63 Accept bullet tokens uploaded from the iPad and show them on the timeline
splat-mitmproxy now PUTs the captured /api/bullet_tokens response to
/bullet-token on each updater Worker, authenticated with its own
TOKEN_UPLOAD_TOKEN. The Worker caches it for the TOKEN_UPLOAD_REGION
account (US) and records an "iPad upload" event on the Nintendo auth row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 14:32:05 -08:00
Matt Isenhower
25f4794f7e Drop lane retries and queue continuations like manual runs
A failed run is no longer retried: the lane's next slot runs again (1-15
minutes later for the fast lane). A run that stops at its time budget with
work left queues a continuation in the same queue as manual runs, whether it
was scheduled or manual, so a lane's state is just its switch, next slot and
last run. Each alarm runs one thing, a due slot before a queued run, so a run
always has the alarm's whole time limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 13:58:55 -08:00
Matt Isenhower
66cb97cd6c Replace lane pause with the scheduling switch, and use toggles in the panel
Pause and turning scheduling off did nearly the same thing, so pause is gone:
a lane with scheduling off runs only when started by hand. Every on/off control
in the admin panel (lanes, renewals, the daily archive run, social accounts) is
now a toggle switch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 13:48:30 -08:00
Matt Isenhower
8806239511 Remove the server's scheduler, cache warming and archive uploader
The Worker's lanes, RegionAuth renewals and direct archive writes replace
app/cron.mjs, warmCaches and DataArchiver, which only spectre used. Local
runs keep every updater, social, screenshot and archive tool; there is no
local scheduler, and local snapshots stay in storage/archive/.

ARCHIVE_DATA now has to be "true": any value, including the "false" in
.env.example, used to turn archiving on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:51:51 -08:00
Matt Isenhower
a406f73622 Add switches for renewals and compression; judge renewals by their result
A credential renewal now counts as failed only when it ends without a new
bullet token. A step that failed and was made up for by a later one (znca
briefly unavailable, say) is kept as a warning and no longer goes to
Sentry.

Everything that runs on its own now has a switch in the admin panel:
scheduled renewals (one switch for every region) and the daily archive
compression join the lanes' scheduling switches. The compressor's starts
from AUTOMATIC_SCHEDULING_ENABLED: on in dev, off in production, so the
production compressor can be deployed before spectre stops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:08:10 -08:00
Matt Isenhower
4ff47cbeee Save festivals.json one region at a time
The four Splatfest regions all save into festivals.json. The lock only
covered reading and merging the file, so regions could write at once
(R2 rejects concurrent puts to one object) or read it before another
region's write landed and drop that region. The lock now covers the
whole save.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 17:00:32 -08:00
Matt Isenhower
1480138e15 Report Worker errors to Sentry and drop Sentry from the Node CLI
Both Workers and their Durable Objects are wrapped with @sentry/cloudflare, so the shared
code's @sentry/core calls (failed updaters, social posts, archive snapshots) reach Sentry.
A lane run that fails outright, a failed credential renewal and a failed archive compression
are reported too. Durable Object RPC methods use instrumentPrototypeMethods; without it they
run with no Sentry client and their events are dropped.

The Node runner and the compressor's container no longer use Sentry; @sentry/node is gone.
POST /sentry-test sends one test event from the updater Worker and one from a Durable Object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:05:01 -08:00
Matt Isenhower
47f532f578 Show archive compressions in the admin panel
Compressor runs get an Archive row and filter on the timeline, a details view with the
run's log lines, and a Controls card with Compress now and Dry run. The updater reaches the
compressor through a COMPRESSOR binding to its Durable Object (dev only for now); without the
binding the panel leaves the archive out. Worker tests use a stand-in compressor Worker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 13:11:29 -08:00
Matt Isenhower
c053372361 Compress archive days in a Cloudflare Container
A separate compressor Worker runs app/data/ArchiveCompressor.mjs in a container (standard-1)
once a day at 00:30 UTC, against the archive Space. Its Durable Object starts the container
and keeps the last 30 runs with their log lines. An alarm every few seconds collects the
container's output from its log server and ends the run when the container reports its exit
code, so a run survives the object being evicted from memory.

Only dev is deployed; production waits until spectre stops compressing at the same time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 13:11:24 -08:00
Matt Isenhower
ed8b5491d8 Log when archive snapshots start uploading and how many were written
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:29:35 -08:00
Matt Isenhower
73f4d403bd Replace the quick/default/all update modes with lane run options
The modes mostly chose which lane's updaters ran, which the lanes now decide. The fast
lane keeps one option, translations (quick runs at :00-:04 skip them); the slow lane has
none. Runs, slots and manual requests carry the option instead of a mode, and the admin
panel shows runs as quick or full, with an "Include translations" box for manual fast
runs. Slots and runs saved with a mode run with the defaults.

The Node runner follows the same cadence: npm run splatnet:quick is the fast lane without
translations, splatnet the fast lane, and splatnet:all both lanes. The unused disableXRank
and disableFestivalRankings settings are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:39:16 -08:00
Matt Isenhower
008db9b459 Hold deferred archive snapshots in a map keyed by archive path
Each region's FestivalUpdater saves the combined festivals.json, so a run saves it several
times. A newer copy now replaces the held one instead of being added alongside it, so each
snapshot is held and uploaded once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:39:08 -08:00
Matt Isenhower
e7388fce6a Archive data snapshots from the Worker to DigitalOcean Spaces
With ARCHIVE_DATA set, the Worker writes each published data file to the
archive Space, at the same YYYY/MM/DD/<date>.<time>.<name>.json keys, ACL,
and content type as the Node uploader (DataArchiver):

- S3ArchiveStorage signs PUTs with aws4fetch, tries three times with
  backoff, and stops early on errors that will not improve.
- A snapshot that cannot be written is a warning and a Sentry event, never
  a failed update.
- The fast lane holds its snapshots until after social posts (even failed
  ones) and then uploads the newest copy of each key, so a slow archive
  bucket never delays a post. The slow lane uploads as it goes, to avoid
  holding large X-Rank files in memory.
- Dev writes to its own Space (splatoon3ink-archive-test); production has
  no archive settings until cutover. Daily compression is still to come.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 22:47:38 -08:00
Matt Isenhower
84e2845a06 Merge pull request #140 from misenhower/buffer-channel-check-cache
Some checks failed
Update query hashes / check-and-create-pr (push) Has been cancelled
Build frontend / build (24.x) (push) Has been cancelled
Deploy / deploy-backend (push) Has been cancelled
Lint / lint (push) Has been cancelled
Tests / test (24.x) (push) Has been cancelled
Check the Buffer channel once a day instead of every run
2026-11-10 14:35:41 -08:00
Matt Isenhower
0e4b5af455 Check the Buffer channel once a day instead of every run
Every Buffer API request counts against the quota, queries included. The
Free plan allows 2,000 requests per 30 days across both channels, and the
per-run channel check made up about 26 of the roughly 67 requests a day.

A successful check is now kept in a ValueCache for 24 hours, keyed by
channel ID, so each channel is checked about once a day. Failed checks are
not cached, and a channel locked within the day still fails at createPost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 14:34:25 -08:00
Matt Isenhower
ae117404a4 Make the admin timeline and its list one scrollable view
The Timeline page showed four kinds of things on the time axis but only
lane runs in the list, and credential renewals could not be opened. Now
both draw from one set of items (lib/activity.js): lane runs, credential
renewals, social posts, and what is coming up next.

- The timeline scrolls back through the last 48 hours by dragging (with a
  short glide after a flick), and loads earlier history six hours at a
  time. Marks that would cover each other are stacked so each can be
  clicked.
- The timeline and the list follow one cursor, so scrolling either moves
  the other. A marker shows the moment the list is lined up with, and
  "Now" returns both to the present. The list is windowed, with upcoming
  items above a "now" line.
- Everything opens in the details panel: renewals with their errors and
  log lines, posts with each account's result, upcoming items with when
  and why. A selected upcoming or running item stays selected as it
  starts and finishes. The selection is ?item=<key> in the address.
- Lane and credential controls move to a new Controls page, which links
  back to the timeline. The Timeline page notes a paused or unscheduled
  lane.
- The list filters by row and by problems; the timeline dims the rest.

Worker side:

- Scheduler keeps runs for 48 hours (HISTORY_MS) instead of the last 50,
  with log lines stored apart from the run summaries so listing stays
  cheap. New history({ from, to }) RPC; run() finds a run directly when
  given its start time; posts() takes a time span.
- RegionAuth keeps 48 hours of renewals, each with its trigger and log
  lines, and serves them by time span and one at a time.
- New GET /admin/api/activity (by time span) and /admin/api/auth/event.
  Status carries only the newest activity, and log lines from renewals
  name the renewal that printed them.

Checked in the local preview against the simulated API; not yet run
against a deployed Worker.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 13:57:38 -08:00
Matt Isenhower
c2666b27ca Add an admin panel to the updater Worker
A small Vue + Tailwind app in workers/updater/admin, served by the updater
Worker on its own hostname (admin.splatoon3.ink, admin.dev.splatoon3.ink)
behind Cloudflare Access:

- Timeline: both lanes, credential renewals, and social posts on one time
  axis, a filterable run list, run details with per-updater timings and
  the run's own log lines, lane controls (run now, pause, scheduling),
  credential renewal, and a combined text log.
- Social: the post log with errors, per-account posting switches, when
  each status last went to each account, and a preview that can post one
  status by hand after a confirmation.
- Data: freshness of the core data files, the published file list, a
  check that the site serves the new data, and a cache purge.

Worker side:

- src/admin: the Access JWT check (jose; fails closed and only answers on
  ADMIN_HOSTNAME) and the API. Actions must be same-origin JSON POSTs, and
  the app is served with a strict CSP from the ADMIN_ASSETS binding.
- Runs capture their console output (AsyncLocalStorage), so log lines
  belong to the run that printed them. The Scheduler keeps a post log and
  serves run details and log lines; RegionAuth keeps its last 24 renewals.
- A manual run from the panel only posts to social when asked to.

Shared code:

- StatusGeneratorManager records what was posted or failed, and skips
  accounts turned off in private settings/social.json (the Node runner
  honours this too). app/social/admin.mjs previews and posts one status.
- Updater results include their duration.

Also: npm run admin:dev runs the panel against a simulated API, deploy
scripts build the panel first, the dev environment's assets move to
dev-assets.splatoon3.ink, and the lockfile drops entries nothing depends
on any more.

Not yet run against a deployed Worker or a real Access application.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 12:59:36 -08:00
Matt Isenhower
1aee4053c6 Give the slow lane a time budget and continue unfinished work
Durable Object alarms are stopped after 15 minutes, and a full X-Rank or
Splatfest rankings update can take longer than that:

- RunLimits takes an optional deadline. X-Rank season details, Splatfest
  rankings, and the slow lane's updater loop stop starting new work once
  it passes and mark the run incomplete; what is left updates in a later
  run. The current X-Rank season goes first.
- The slow lane runs with a 10-minute budget. An incomplete run is
  followed a minute later by a continuation (up to 10, never past the
  lane's next regular slot), which does not count as a retry. Manual runs
  continue the same way through a queued follow-up.
- Ignore Wrangler's local secrets files (.dev.vars*).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 12:59:36 -08:00
Matt Isenhower
db2e68b847 Add the Cloudflare updater Worker with fast and slow lanes
Run the shared app/ updaters and social posters from workers/updater:

- Scheduler Durable Object lanes: "fast" keeps the Node cron cadence
  (quick at :00-:04, default at :05/:10/:15/:20/:30/:45) and posts to
  social after checking SITE_URL serves the new data; "slow" runs X-Rank,
  Splatfest rankings, and avatar updates hourly. Bounded retries, queued
  manual runs, pause, and persisted automatic-scheduling state.
- RegionAuth Durable Object per region: the single owner of each
  Nintendo login, serving bullet tokens to both lanes and renewing them
  hourly at :35-:38.
- A five-minute cron watchdog that only re-arms alarms, bearer-token
  control endpoints, and dev/production environments. DO migrations keep
  Scheduler at v1 to match the earlier dev deployment.
- Per-run limits (RunLimits) and per-account NsoClient locks instead of
  module-level limiters, since Durable Objects sharing an isolate cannot
  share in-flight requests. Updaters without localization rules no longer
  rewrite locale files.
- 20 workerd tests (npm run updater:test, included in npm test) with all
  network access blocked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 22:13:30 -08:00
Matt Isenhower
8a02e2bfdc Make the shared backend code run in Cloudflare Workers
Replace the remaining Node-only dependencies so the updaters and social
posters bundle and run in workerd, while the Node runner keeps working:

- Replace nxapi with app/splatnet/coral.mjs (fetch + WebCrypto), aligned
  with nxapi 1.6.1-next.257: znca compatibility 3.5.0, the current client
  version, and the slimmer Login parameter. Use one znca-api client per
  Nintendo account, since its token binds to the first account it serves.
  Cache the remote config in private storage instead of restarting daily.
  Sessions cached by the nxapi package trigger a fresh login.
- Convert images through #image-converter: sharp in Node, the Images
  binding in Workers. Read Bluesky aspect ratios from the PNG header.
- Replace twitter-api-v2 with OAuth 1.0a signed fetch; TwitterClient now
  only updates the seasonal X profile (posts go through Buffer).
- Load avatars through #avatars (bundled as Data modules in Workers) and
  keep the local Chrome driver out of Worker bundles via
  #local-screenshot-driver.
- prefixedConsole forwards to the global console; shared code reports to
  Sentry through @sentry/core.

Verified live: full credential refresh for all four regions in Node, and
the complete Coral → bullet token → GraphQL chain for EU inside workerd.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 16:51:54 -08:00
Matt Isenhower
21a4f5d85c Route backend file access through storage and auth seams
Prepare the shared updater and social code to run against R2 in a Worker
while keeping the local Node runner working:

- Add app/common/storage with public, private, and archive areas keyed like
  the R2 objects. FilesystemStorage maps them onto dist/ and storage/ for
  Node; BucketStorage wraps R2 bindings. Storage is configured once per
  process; only image existence is cached, so JSON reads are always fresh.
- Move ValueCache, DataUpdater (data, calendars, archive snapshots),
  ImageProcessor, LocalizationProcessor, festival updaters, status
  generators, social translations, ImageWriter, and Buffer/Threads image
  hosting onto storage. Query hashes are a JSON import.
- Add app/splatnet/auth.mjs: the existing bullet-token code behind auth(),
  so the Worker can route token requests to per-region Durable Objects.
  warmCaches uses auth().refresh().
- Remove the S3/R2 sync, vfs, the dual upload, the sync commands, and
  s3-sync-client. Local runs no longer read from or write to remote buckets.
- Give every test fresh in-memory storage via app/test/setup.mjs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 16:03:33 -08:00
Matt Isenhower
6df93806c2 Prepare the backend for Cloudflare Workers
Groundwork for running the existing updaters and social posters in a Worker:

- Derive IDs with @noble/hashes shake256; the Workers runtime's node:crypto
  lacks shake256. A test pins an ID produced by node:crypto.
- Split updaters into fast (schedules, gear, co-op, festivals, stages; run
  concurrently) and slow (X-Rank, Splatfest rankings; run one at a time).
  The cron posts after the fast updaters, before the slow ones.
- Fetch X-Rank seasons and detail types sequentially, cutting peak heap per
  division from ~85 MB to ~38 MB.
- Move Splatfest rankings into FestivalRankingsUpdater, driven by the
  published festivals data, so results never wait on ranking pagination.
- Coalesce concurrent downloads of the same SplatNet image.
- Bring over the Browser-binding screenshot work from develop (f7293b6):
  Cloudflare captures use the BROWSER binding with retries, Browserless is
  removed, and workers/screenshots offers capture-only local testing. The
  local driver keeps puppeteer-core with an installed Chrome, always serves
  dist/, and fails on page-reported screenshot errors.
- Remove the Docker image, compose files, and GHCR deploy workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 15:40:51 -08:00
Matt Isenhower
09ce415f38 Merge pull request #139 from misenhower/codex/serialize-bullet-token
Some checks failed
Update query hashes / check-and-create-pr (push) Has been cancelled
Build frontend / build (24.x) (push) Has been cancelled
Deploy / deploy-backend (push) Has been cancelled
Lint / lint (push) Has been cancelled
Tests / test (24.x) (push) Has been cancelled
Serialize bullet token acquisition per account
2026-11-08 08:14:58 -08:00
Matt Isenhower
a2403c0680 Serialize bullet token acquisition per account 2026-11-08 07:56:02 -08:00
Matt Isenhower
244bb487c6 Merge pull request #138 from misenhower/codex/tricolor-social-layout
Some checks failed
Update query hashes / check-and-create-pr (push) Has been cancelled
Build frontend / build (24.x) (push) Has been cancelled
Deploy / deploy-backend (push) Has been cancelled
Lint / lint (push) Has been cancelled
Tests / test (24.x) (push) Has been cancelled
Preserve card proportions in Tricolor social images
2026-10-31 09:54:27 -07:00
Matt Isenhower
c4c1d55ea2 Preserve card proportions in Tricolor social images 2026-10-31 09:45:53 -07:00
Matt Isenhower
d0e094e55a Add Splatfest preview without Tricolor 2026-10-31 09:21:48 -07:00
Matt Isenhower
07ba37540c Merge pull request #137 from misenhower/codex/update-social-links
Some checks failed
Update query hashes / check-and-create-pr (push) Has been cancelled
Build frontend / build (24.x) (push) Has been cancelled
Deploy / deploy-backend (push) Has been cancelled
Lint / lint (push) Has been cancelled
Tests / test (24.x) (push) Has been cancelled
Update social account links and add Japanese follow buttons
2026-10-18 13:35:52 -07:00
Matt Isenhower
5a7b24b744 Match follow buttons to the main menu styling 2026-10-18 13:33:24 -07:00
Matt Isenhower
0109241d01 Add Japanese social follow buttons to the site header 2026-10-18 13:32:06 -07:00
Matt Isenhower
47c473ba86 Update social links for English and Japanese accounts 2026-10-17 16:45:10 -07:00
Matt Isenhower
2c10505504 Merge pull request #136 from misenhower/codex/japanese-social-posts
Some checks failed
Update query hashes / check-and-create-pr (push) Has been cancelled
Build frontend / build (24.x) (push) Has been cancelled
Deploy / deploy-backend (push) Has been cancelled
Lint / lint (push) Has been cancelled
Tests / test (24.x) (push) Has been cancelled
Add Japanese social posts for Bluesky and Twitter
2026-10-17 13:22:30 -07:00
Matt Isenhower
341601d075 Simplify social account configuration and shared time messages 2026-10-17 13:17:59 -07:00
Matt Isenhower
564331b212 Remove redundant screenshot time watcher 2026-10-17 13:15:04 -07:00
Matt Isenhower
e29ef53750 Enable Japanese social clients through their credentials 2026-10-17 13:03:29 -07:00
Matt Isenhower
1016708901 Keep Japanese account profiles unchanged 2026-10-17 13:01:21 -07:00
Matt Isenhower
61fb95df77 Make the social post comparison reproducible 2026-10-17 12:55:55 -07:00
Matt Isenhower
23a6416ef2 Include both Splatoon hashtags on Japanese posts 2026-10-17 12:51:39 -07:00
Matt Isenhower
286e3fc3fe Use Japanese hashtags for post categories 2026-10-17 12:50:36 -07:00
Matt Isenhower
20f65894f8 Describe image details and preserve post filtering tags 2026-10-17 12:44:04 -07:00
Matt Isenhower
74231fffb2 Keep countdowns and timely announcements in Japanese posts 2026-10-17 12:33:22 -07:00
Matt Isenhower
91d3ed0be1 Keep relative expiry times in Japanese gear images 2026-10-17 12:13:47 -07:00
Matt Isenhower
3f462ce6f4 Show absolute times in Japanese social images 2026-10-17 12:09:06 -07:00
Matt Isenhower
9eaf7ad84e Remove social post character counting 2026-10-17 12:04:24 -07:00
Matt Isenhower
058f23cfc1 Add separate Japanese social accounts and local post previews 2026-10-17 11:57:28 -07:00
Matt Isenhower
23474c742f Localize social screenshots and keep rotation times fixed 2026-10-17 11:57:22 -07:00
Matt Isenhower
a81fc2b2a5 Generate Japanese social text with localized game names and JST dates 2026-10-17 11:49:02 -07:00