84 Commits

Author SHA1 Message Date
Matt Isenhower
aa71bf2eda Hold a lane run while an earlier one may still be going; log data changes
Some checks failed
Build frontend / build (24.x) (push) Has been cancelled
Lint / lint (push) Has been cancelled
Tests / test (24.x) (push) Has been cancelled
Cloudflare replaced the fast lane's Durable Object during the 00:00 UTC run.
The alarm restarted on a new instance while the old one carried on posting,
and the old run's results were lost because it could no longer write to its
own storage. Like the old cron, a run now takes a lock in the private R2
bucket (which a replaced instance can still write), refreshes it every 10
seconds and deletes it at the end; a run that finds a fresh lock waits and
checks again in 15 seconds. A scheduled run that never saved its results is
recorded as cut off.

Each updater now reports whether its published file changed, in its log
line and in the run's results, to see when fresh data really arrives at the
top of the hour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 16:23:40 -08:00
Matt Isenhower
8806239511 Remove the server's scheduler, cache warming and archive uploader
The Worker's lanes, RegionAuth renewals and direct archive writes replace
app/cron.mjs, warmCaches and DataArchiver, which only spectre used. Local
runs keep every updater, social, screenshot and archive tool; there is no
local scheduler, and local snapshots stay in storage/archive/.

ARCHIVE_DATA now has to be "true": any value, including the "false" in
.env.example, used to turn archiving on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:51:51 -08:00
Matt Isenhower
4ff47cbeee Save festivals.json one region at a time
The four Splatfest regions all save into festivals.json. The lock only
covered reading and merging the file, so regions could write at once
(R2 rejects concurrent puts to one object) or read it before another
region's write landed and drop that region. The lock now covers the
whole save.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 17:00:32 -08:00
Matt Isenhower
73f4d403bd Replace the quick/default/all update modes with lane run options
The modes mostly chose which lane's updaters ran, which the lanes now decide. The fast
lane keeps one option, translations (quick runs at :00-:04 skip them); the slow lane has
none. Runs, slots and manual requests carry the option instead of a mode, and the admin
panel shows runs as quick or full, with an "Include translations" box for manual fast
runs. Slots and runs saved with a mode run with the defaults.

The Node runner follows the same cadence: npm run splatnet:quick is the fast lane without
translations, splatnet the fast lane, and splatnet:all both lanes. The unused disableXRank
and disableFestivalRankings settings are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:39:16 -08:00
Matt Isenhower
008db9b459 Hold deferred archive snapshots in a map keyed by archive path
Each region's FestivalUpdater saves the combined festivals.json, so a run saves it several
times. A newer copy now replaces the held one instead of being added alongside it, so each
snapshot is held and uploaded once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:39:08 -08:00
Matt Isenhower
e7388fce6a Archive data snapshots from the Worker to DigitalOcean Spaces
With ARCHIVE_DATA set, the Worker writes each published data file to the
archive Space, at the same YYYY/MM/DD/<date>.<time>.<name>.json keys, ACL,
and content type as the Node uploader (DataArchiver):

- S3ArchiveStorage signs PUTs with aws4fetch, tries three times with
  backoff, and stops early on errors that will not improve.
- A snapshot that cannot be written is a warning and a Sentry event, never
  a failed update.
- The fast lane holds its snapshots until after social posts (even failed
  ones) and then uploads the newest copy of each key, so a slow archive
  bucket never delays a post. The slow lane uploads as it goes, to avoid
  holding large X-Rank files in memory.
- Dev writes to its own Space (splatoon3ink-archive-test); production has
  no archive settings until cutover. Daily compression is still to come.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 22:47:38 -08:00
Matt Isenhower
1aee4053c6 Give the slow lane a time budget and continue unfinished work
Durable Object alarms are stopped after 15 minutes, and a full X-Rank or
Splatfest rankings update can take longer than that:

- RunLimits takes an optional deadline. X-Rank season details, Splatfest
  rankings, and the slow lane's updater loop stop starting new work once
  it passes and mark the run incomplete; what is left updates in a later
  run. The current X-Rank season goes first.
- The slow lane runs with a 10-minute budget. An incomplete run is
  followed a minute later by a continuation (up to 10, never past the
  lane's next regular slot), which does not count as a retry. Manual runs
  continue the same way through a queued follow-up.
- Ignore Wrangler's local secrets files (.dev.vars*).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 12:59:36 -08:00
Matt Isenhower
db2e68b847 Add the Cloudflare updater Worker with fast and slow lanes
Run the shared app/ updaters and social posters from workers/updater:

- Scheduler Durable Object lanes: "fast" keeps the Node cron cadence
  (quick at :00-:04, default at :05/:10/:15/:20/:30/:45) and posts to
  social after checking SITE_URL serves the new data; "slow" runs X-Rank,
  Splatfest rankings, and avatar updates hourly. Bounded retries, queued
  manual runs, pause, and persisted automatic-scheduling state.
- RegionAuth Durable Object per region: the single owner of each
  Nintendo login, serving bullet tokens to both lanes and renewing them
  hourly at :35-:38.
- A five-minute cron watchdog that only re-arms alarms, bearer-token
  control endpoints, and dev/production environments. DO migrations keep
  Scheduler at v1 to match the earlier dev deployment.
- Per-run limits (RunLimits) and per-account NsoClient locks instead of
  module-level limiters, since Durable Objects sharing an isolate cannot
  share in-flight requests. Updaters without localization rules no longer
  rewrite locale files.
- 20 workerd tests (npm run updater:test, included in npm test) with all
  network access blocked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 22:13:30 -08:00
Matt Isenhower
8a02e2bfdc Make the shared backend code run in Cloudflare Workers
Replace the remaining Node-only dependencies so the updaters and social
posters bundle and run in workerd, while the Node runner keeps working:

- Replace nxapi with app/splatnet/coral.mjs (fetch + WebCrypto), aligned
  with nxapi 1.6.1-next.257: znca compatibility 3.5.0, the current client
  version, and the slimmer Login parameter. Use one znca-api client per
  Nintendo account, since its token binds to the first account it serves.
  Cache the remote config in private storage instead of restarting daily.
  Sessions cached by the nxapi package trigger a fresh login.
- Convert images through #image-converter: sharp in Node, the Images
  binding in Workers. Read Bluesky aspect ratios from the PNG header.
- Replace twitter-api-v2 with OAuth 1.0a signed fetch; TwitterClient now
  only updates the seasonal X profile (posts go through Buffer).
- Load avatars through #avatars (bundled as Data modules in Workers) and
  keep the local Chrome driver out of Worker bundles via
  #local-screenshot-driver.
- prefixedConsole forwards to the global console; shared code reports to
  Sentry through @sentry/core.

Verified live: full credential refresh for all four regions in Node, and
the complete Coral → bullet token → GraphQL chain for EU inside workerd.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 16:51:54 -08:00
Matt Isenhower
21a4f5d85c Route backend file access through storage and auth seams
Prepare the shared updater and social code to run against R2 in a Worker
while keeping the local Node runner working:

- Add app/common/storage with public, private, and archive areas keyed like
  the R2 objects. FilesystemStorage maps them onto dist/ and storage/ for
  Node; BucketStorage wraps R2 bindings. Storage is configured once per
  process; only image existence is cached, so JSON reads are always fresh.
- Move ValueCache, DataUpdater (data, calendars, archive snapshots),
  ImageProcessor, LocalizationProcessor, festival updaters, status
  generators, social translations, ImageWriter, and Buffer/Threads image
  hosting onto storage. Query hashes are a JSON import.
- Add app/splatnet/auth.mjs: the existing bullet-token code behind auth(),
  so the Worker can route token requests to per-region Durable Objects.
  warmCaches uses auth().refresh().
- Remove the S3/R2 sync, vfs, the dual upload, the sync commands, and
  s3-sync-client. Local runs no longer read from or write to remote buckets.
- Give every test fresh in-memory storage via app/test/setup.mjs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 16:03:33 -08:00
Matt Isenhower
6df93806c2 Prepare the backend for Cloudflare Workers
Groundwork for running the existing updaters and social posters in a Worker:

- Derive IDs with @noble/hashes shake256; the Workers runtime's node:crypto
  lacks shake256. A test pins an ID produced by node:crypto.
- Split updaters into fast (schedules, gear, co-op, festivals, stages; run
  concurrently) and slow (X-Rank, Splatfest rankings; run one at a time).
  The cron posts after the fast updaters, before the slow ones.
- Fetch X-Rank seasons and detail types sequentially, cutting peak heap per
  division from ~85 MB to ~38 MB.
- Move Splatfest rankings into FestivalRankingsUpdater, driven by the
  published festivals data, so results never wait on ranking pagination.
- Coalesce concurrent downloads of the same SplatNet image.
- Bring over the Browser-binding screenshot work from develop (f7293b6):
  Cloudflare captures use the BROWSER binding with retries, Browserless is
  removed, and workers/screenshots offers capture-only local testing. The
  local driver keeps puppeteer-core with an installed Chrome, always serves
  dist/, and fails on page-reported screenshot errors.
- Remove the Docker image, compose files, and GHCR deploy workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 15:40:51 -08:00
Matt Isenhower
5dd2bbf172 Resolve npm audit vulnerabilities 2026-08-30 16:43:01 -07:00
Matt Isenhower
7125f30bfc Remove inline base64 image embedding from iCal output
No major calendar client renders these inline binary attachments.
The ATTACH URL is kept, which links to the publicly hosted image.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 09:44:04 -08:00
Matt Isenhower
41c9f9a315 Add VFS layer to avoid downloading files only needed for existence checks
Uses S3 ListObjectsV2 to build an in-memory file listing at startup,
allowing exists() and olderThan() to resolve from the listing instead
of requiring files on disk. sync:download now skips assets/splatnet/,
data/xrank/, data/festivals.ranking.*, and status-screenshots/.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 09:26:05 -08:00
Matt Isenhower
60c61e6231 Fix bug with invalid data on disk 2026-02-17 21:35:21 -08:00
Matt Isenhower
af526403a1 Update cron to v4 and ical-generator to v10
cron v4: No code changes needed (positional constructor args still supported).
ical-generator v10: Removed `new` keyword (now a factory function) and
switched from `attachments` property to `createAttachment()` method.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 16:09:19 -08:00
Matt Isenhower
8b11900905 Revert jsonpath-plus back to jsonpath
jsonpath v1.2.1 (published Feb 2025) fixed the security vulnerability
that prompted the switch to jsonpath-plus. Reverting to the original
package simplifies the code by removing the jsonpathQuery/jsonpathApply
wrapper functions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 15:56:05 -08:00
Matt Isenhower
7adb9c313e Add mkdirp utility to avoid verbose fs.mkdir calls
Added mkdirp() helper to app/common/fs.mjs that wraps
fs.mkdir(dir, { recursive: true }). Updated all call sites to use it.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 11:09:02 -08:00
Matt Isenhower
8780463c66 Replace deprecated and unmaintained packages
- Removed mkdirp in favor of native fs.mkdir({ recursive: true })
- Replaced ecstatic (unmaintained since 2021) with sirv
- Replaced jsonpath (security vulnerability) with jsonpath-plus
  - Added jsonpathQuery/jsonpathApply helpers in app/common/util.mjs
- Updated sharp: 0.32.6 → 0.34.5
- Updated puppeteer-core: 23.8.0 → 24.37.3

Vulnerabilities reduced from 40 to 1 (only remaining: axios in
threads-api transitive dependency).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 11:05:49 -08:00
Matt Isenhower
011430f344 Fix code inconsistencies across frontend and backend
Use strict equality in TricolorBox, add async keyword to
CoopUpdater/GearUpdater getData() for consistency, and add
missing HMR accept for useCoopGearStore.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-31 20:04:05 -08:00
Matt Isenhower
ba44aec52a Make the data archive optional
Some checks failed
Deploy / deploy-frontend (push) Has been cancelled
Deploy / deploy-backend (push) Has been cancelled
2024-11-29 14:35:52 -05:00
Matt Isenhower
71965990bd Make festival update atomic
Some checks are pending
Deploy / deploy-frontend (push) Waiting to run
Deploy / deploy-backend (push) Waiting to run
2024-11-21 22:09:16 -08:00
Matt Isenhower
7d4b9cc038 Skip "no need to update" message 2024-11-20 21:46:45 -08:00
Matt Isenhower
fbdea0f3c2 Update X-Rank data in parallel 2024-11-20 21:39:39 -08:00
Matt Isenhower
002ca2e34a Update festival data in parallel 2024-11-20 21:39:12 -08:00
Matt Isenhower
10c082a1c2 Retrieve localizations in parallel 2024-11-20 21:13:40 -08:00
Matt Isenhower
34c2d65b15 Download images concurrently 2024-11-17 21:16:24 -08:00
Matt Isenhower
c726f25910 Fix X-Rank updater 2024-09-12 10:36:29 -07:00
Matt Isenhower
2736996f05 Tweak Splatfest updater to handle new paginated query 2024-09-05 08:01:12 -07:00
Matt Isenhower
37ee0c4dc9 Prefer single quotes and enforce semicolons 2024-03-17 10:20:17 -07:00
Matt Isenhower
e87a4b241d lint-fix 2024-03-17 10:15:18 -07:00
Matt Isenhower
8b55a5deba Fix linting issues 2024-03-16 22:31:20 -07:00
Matt Isenhower
cce839e3bd Tweak festival updater 2023-12-02 11:13:54 -08:00
Matt Isenhower
6173693520 Make updaters more configurable
This should reduce the number of queries needed during high-load times (like waiting for Splatfest results).
2023-12-02 11:06:45 -08:00
Matt Isenhower
6909cfb864 Cleanup 2023-12-02 10:21:29 -08:00
Matt Isenhower
77c1680bb9 Use new boss data 2023-11-29 20:09:20 -08:00
Matt Isenhower
10ee1f862a Fix festival ranking updater 2023-08-13 19:46:16 -07:00
Matt Isenhower
51348effc8 Prevent issues with nulls 2023-07-23 08:36:43 -07:00
Matt Isenhower
edfeb9b51c Retrieve translations for events 2023-06-01 09:32:42 -07:00
Matt Isenhower
840d45373a Move archives back to storage dir and use precise time for X-Rank 2023-05-30 19:33:37 -07:00
Matt Isenhower
69ea3962cb Fix a mistake 2023-05-23 17:51:46 -07:00
Matt Isenhower
ba54d94e09 Make data archives more accessible, and store only one file per hour 2023-05-23 17:48:30 -07:00
Matt Isenhower
ce838a7652 Fix broken X-Rank images 2023-04-29 09:24:45 -07:00
Matt Isenhower
4b142c890e Only update X-Rank details when needed 2023-04-22 14:33:51 -07:00
Matt Isenhower
142f2a82c5 Fix a bug with outdated festival data 2023-04-22 14:27:55 -07:00
Matt Isenhower
46fdbbe8c7 Clean up the way we detect whether an update is needed 2023-04-22 14:05:59 -07:00
Matt Isenhower
cda854de0b Add a utility function to check whether a file exists 2023-04-22 13:33:04 -07:00
Matt Isenhower
b7d47a297b Retrieve data for previous X-Rank seasons as well as current ones 2023-04-22 12:41:28 -07:00
Matt Isenhower
14f89d3eb8 Add stages.json 2023-04-20 09:49:42 -07:00
Matt Isenhower
03c2837a85 Don't add King Salmonid predictions for Eggstra Work schedules 2023-04-13 10:40:55 -07:00