42 Commits

Author SHA1 Message Date
Matt Isenhower
8806239511 Remove the server's scheduler, cache warming and archive uploader
The Worker's lanes, RegionAuth renewals and direct archive writes replace
app/cron.mjs, warmCaches and DataArchiver, which only spectre used. Local
runs keep every updater, social, screenshot and archive tool; there is no
local scheduler, and local snapshots stay in storage/archive/.

ARCHIVE_DATA now has to be "true": any value, including the "false" in
.env.example, used to turn archiving on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:51:51 -08:00
Matt Isenhower
1480138e15 Report Worker errors to Sentry and drop Sentry from the Node CLI
Both Workers and their Durable Objects are wrapped with @sentry/cloudflare, so the shared
code's @sentry/core calls (failed updaters, social posts, archive snapshots) reach Sentry.
A lane run that fails outright, a failed credential renewal and a failed archive compression
are reported too. Durable Object RPC methods use instrumentPrototypeMethods; without it they
run with no Sentry client and their events are dropped.

The Node runner and the compressor's container no longer use Sentry; @sentry/node is gone.
POST /sentry-test sends one test event from the updater Worker and one from a Durable Object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:05:01 -08:00
Matt Isenhower
e7388fce6a Archive data snapshots from the Worker to DigitalOcean Spaces
With ARCHIVE_DATA set, the Worker writes each published data file to the
archive Space, at the same YYYY/MM/DD/<date>.<time>.<name>.json keys, ACL,
and content type as the Node uploader (DataArchiver):

- S3ArchiveStorage signs PUTs with aws4fetch, tries three times with
  backoff, and stops early on errors that will not improve.
- A snapshot that cannot be written is a warning and a Sentry event, never
  a failed update.
- The fast lane holds its snapshots until after social posts (even failed
  ones) and then uploads the newest copy of each key, so a slow archive
  bucket never delays a post. The slow lane uploads as it goes, to avoid
  holding large X-Rank files in memory.
- Dev writes to its own Space (splatoon3ink-archive-test); production has
  no archive settings until cutover. Daily compression is still to come.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 22:47:38 -08:00
Matt Isenhower
8a02e2bfdc Make the shared backend code run in Cloudflare Workers
Replace the remaining Node-only dependencies so the updaters and social
posters bundle and run in workerd, while the Node runner keeps working:

- Replace nxapi with app/splatnet/coral.mjs (fetch + WebCrypto), aligned
  with nxapi 1.6.1-next.257: znca compatibility 3.5.0, the current client
  version, and the slimmer Login parameter. Use one znca-api client per
  Nintendo account, since its token binds to the first account it serves.
  Cache the remote config in private storage instead of restarting daily.
  Sessions cached by the nxapi package trigger a fresh login.
- Convert images through #image-converter: sharp in Node, the Images
  binding in Workers. Read Bluesky aspect ratios from the PNG header.
- Replace twitter-api-v2 with OAuth 1.0a signed fetch; TwitterClient now
  only updates the seasonal X profile (posts go through Buffer).
- Load avatars through #avatars (bundled as Data modules in Workers) and
  keep the local Chrome driver out of Worker bundles via
  #local-screenshot-driver.
- prefixedConsole forwards to the global console; shared code reports to
  Sentry through @sentry/core.

Verified live: full credential refresh for all four regions in Node, and
the complete Coral → bullet token → GraphQL chain for EU inside workerd.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 16:51:54 -08:00
Matt Isenhower
21a4f5d85c Route backend file access through storage and auth seams
Prepare the shared updater and social code to run against R2 in a Worker
while keeping the local Node runner working:

- Add app/common/storage with public, private, and archive areas keyed like
  the R2 objects. FilesystemStorage maps them onto dist/ and storage/ for
  Node; BucketStorage wraps R2 bindings. Storage is configured once per
  process; only image existence is cached, so JSON reads are always fresh.
- Move ValueCache, DataUpdater (data, calendars, archive snapshots),
  ImageProcessor, LocalizationProcessor, festival updaters, status
  generators, social translations, ImageWriter, and Buffer/Threads image
  hosting onto storage. Query hashes are a JSON import.
- Add app/splatnet/auth.mjs: the existing bullet-token code behind auth(),
  so the Worker can route token requests to per-region Durable Objects.
  warmCaches uses auth().refresh().
- Remove the S3/R2 sync, vfs, the dual upload, the sync commands, and
  s3-sync-client. Local runs no longer read from or write to remote buckets.
- Give every test fresh in-memory storage via app/test/setup.mjs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 16:03:33 -08:00
Matt Isenhower
6df93806c2 Prepare the backend for Cloudflare Workers
Groundwork for running the existing updaters and social posters in a Worker:

- Derive IDs with @noble/hashes shake256; the Workers runtime's node:crypto
  lacks shake256. A test pins an ID produced by node:crypto.
- Split updaters into fast (schedules, gear, co-op, festivals, stages; run
  concurrently) and slow (X-Rank, Splatfest rankings; run one at a time).
  The cron posts after the fast updaters, before the slow ones.
- Fetch X-Rank seasons and detail types sequentially, cutting peak heap per
  division from ~85 MB to ~38 MB.
- Move Splatfest rankings into FestivalRankingsUpdater, driven by the
  published festivals data, so results never wait on ranking pagination.
- Coalesce concurrent downloads of the same SplatNet image.
- Bring over the Browser-binding screenshot work from develop (f7293b6):
  Cloudflare captures use the BROWSER binding with retries, Browserless is
  removed, and workers/screenshots offers capture-only local testing. The
  local driver keeps puppeteer-core with an installed Chrome, always serves
  dist/, and fails on page-reported screenshot errors.
- Remove the Docker image, compose files, and GHCR deploy workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 15:40:51 -08:00
Matt Isenhower
5dd2bbf172 Resolve npm audit vulnerabilities 2026-08-30 16:43:01 -07:00
Matt Isenhower
e5a7c44ac1 Fix unused catch bindings 2026-08-18 23:31:17 -07:00
Matt Isenhower
96725eecbb Use prefixedConsole for VFS logging
Lazily initialized to avoid console-stamp issues in test environment.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 10:11:40 -08:00
Matt Isenhower
941a6eb258 Remove unused catch binding variables in fs.mjs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 10:09:17 -08:00
Matt Isenhower
6dac044a69 Make VFS load lazily
VFS now loads on first has()/getMtime() call rather than requiring
explicit initialization in sync/index.mjs. This ensures it works
regardless of entry point (sync:download, splatnet:quick, etc.).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 09:55:13 -08:00
Matt Isenhower
e6e96ac8f9 Check local disk before VFS, removing need for manual vfs.track()
exists() and olderThan() now check the local filesystem first and only
consult the VFS as a fallback. This means freshly written files are
found naturally without callers needing to register them.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 09:27:00 -08:00
Matt Isenhower
41c9f9a315 Add VFS layer to avoid downloading files only needed for existence checks
Uses S3 ListObjectsV2 to build an in-memory file listing at startup,
allowing exists() and olderThan() to resolve from the listing instead
of requiring files on disk. sync:download now skips assets/splatnet/,
data/xrank/, data/festivals.ranking.*, and status-screenshots/.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 09:26:05 -08:00
Matt Isenhower
7cfe3775fe Remove unused variable in util test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 15:51:53 -08:00
Matt Isenhower
6fba6f3076 Remove stale jsonpath tests from util.test.mjs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 16:55:28 -08:00
Matt Isenhower
6e4b57f6a7 Add unit tests for fs utilities and ValueCache
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 16:45:20 -08:00
Matt Isenhower
7257c26869 Add unit tests for app/common/util
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 16:45:20 -08:00
Matt Isenhower
8b11900905 Revert jsonpath-plus back to jsonpath
jsonpath v1.2.1 (published Feb 2025) fixed the security vulnerability
that prompted the switch to jsonpath-plus. Reverting to the original
package simplifies the code by removing the jsonpathQuery/jsonpathApply
wrapper functions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 15:56:05 -08:00
misenhower
f92267d0bf Fix styling 2026-02-15 22:04:22 +00:00
Matt Isenhower
7adb9c313e Add mkdirp utility to avoid verbose fs.mkdir calls
Added mkdirp() helper to app/common/fs.mjs that wraps
fs.mkdir(dir, { recursive: true }). Updated all call sites to use it.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 11:09:02 -08:00
Matt Isenhower
8780463c66 Replace deprecated and unmaintained packages
- Removed mkdirp in favor of native fs.mkdir({ recursive: true })
- Replaced ecstatic (unmaintained since 2021) with sirv
- Replaced jsonpath (security vulnerability) with jsonpath-plus
  - Added jsonpathQuery/jsonpathApply helpers in app/common/util.mjs
- Updated sharp: 0.32.6 → 0.34.5
- Updated puppeteer-core: 23.8.0 → 24.37.3

Vulnerabilities reduced from 40 to 1 (only remaining: axios in
threads-api transitive dependency).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 11:05:49 -08:00
Matt Isenhower
f45a4654d1 Extract shared cache expiry calculation to utility function
Deduplicate identical _calculateCacheExpiry methods from
SplatNet3Client and NsoClient into a shared calculateCacheExpiry
function in util.mjs.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-31 20:04:23 -08:00
Matt Isenhower
37ee0c4dc9 Prefer single quotes and enforce semicolons 2024-03-17 10:20:17 -07:00
Matt Isenhower
c6091d822e Fix import errors 2024-03-17 10:12:13 -07:00
Matt Isenhower
3e2c31eaa7 Remove v-html 2024-03-16 22:41:13 -07:00
Matt Isenhower
38dd66b207 Fix <br /> in challenge posts 2023-11-23 08:31:11 -08:00
Matt Isenhower
9894ec30b1 Add Sentry 2023-08-03 08:52:01 -07:00
Matt Isenhower
e4e6051999 Run linter 2023-06-25 20:59:43 -07:00
Matt Isenhower
ba54d94e09 Make data archives more accessible, and store only one file per hour 2023-05-23 17:48:30 -07:00
Matt Isenhower
142f2a82c5 Fix a bug with outdated festival data 2023-04-22 14:27:55 -07:00
Matt Isenhower
46fdbbe8c7 Clean up the way we detect whether an update is needed 2023-04-22 14:05:59 -07:00
Matt Isenhower
cda854de0b Add a utility function to check whether a file exists 2023-04-22 13:33:04 -07:00
Matt Isenhower
2c1a77e63a Improve readability for X-Rank season IDs 2023-02-12 09:22:36 -08:00
Matt Isenhower
9719317a53 Consolidate festival ID parsing functions 2023-02-11 14:25:52 -08:00
Matt Isenhower
f689f13d9c Tweak derived ID logic to ensure consistency
This now strips out the domain and query string from the URL before hashing it. This way, we can help ensure the hashes will stay the same when those parameters change.
2022-10-20 16:16:46 -07:00
Matt Isenhower
133ed8ce56 Use shorter derived ID values 2022-10-17 20:55:20 -07:00
Matt Isenhower
47d0abb747 Retrieve localized strings from SplatNet 2022-10-17 20:32:38 -07:00
Matt Isenhower
212f05c88c Add gear icon to regular gear tweet 2022-10-04 16:26:41 -07:00
Matt Isenhower
668da0c1a9 Move "last tweet" tracking to the tweet generator 2022-09-20 20:32:51 -07:00
Matt Isenhower
231681bdac Improve logging 2022-09-10 12:56:21 -07:00
Matt Isenhower
314d61ecb4 Move cache files to their own directory 2022-09-09 14:55:30 -07:00
Matt Isenhower
9bd16d2e41 Add nxapi-powered NSO client 2022-09-09 14:41:39 -07:00