3 Commits

Author SHA1 Message Date
Matt Isenhower
8806239511 Remove the server's scheduler, cache warming and archive uploader
The Worker's lanes, RegionAuth renewals and direct archive writes replace
app/cron.mjs, warmCaches and DataArchiver, which only spectre used. Local
runs keep every updater, social, screenshot and archive tool; there is no
local scheduler, and local snapshots stay in storage/archive/.

ARCHIVE_DATA now has to be "true": any value, including the "false" in
.env.example, used to turn archiving on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:51:51 -08:00
Matt Isenhower
e7388fce6a Archive data snapshots from the Worker to DigitalOcean Spaces
With ARCHIVE_DATA set, the Worker writes each published data file to the
archive Space, at the same YYYY/MM/DD/<date>.<time>.<name>.json keys, ACL,
and content type as the Node uploader (DataArchiver):

- S3ArchiveStorage signs PUTs with aws4fetch, tries three times with
  backoff, and stops early on errors that will not improve.
- A snapshot that cannot be written is a warning and a Sentry event, never
  a failed update.
- The fast lane holds its snapshots until after social posts (even failed
  ones) and then uploads the newest copy of each key, so a slow archive
  bucket never delays a post. The slow lane uploads as it goes, to avoid
  holding large X-Rank files in memory.
- Dev writes to its own Space (splatoon3ink-archive-test); production has
  no archive settings until cutover. Daily compression is still to come.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 22:47:38 -08:00
Matt Isenhower
21a4f5d85c Route backend file access through storage and auth seams
Prepare the shared updater and social code to run against R2 in a Worker
while keeping the local Node runner working:

- Add app/common/storage with public, private, and archive areas keyed like
  the R2 objects. FilesystemStorage maps them onto dist/ and storage/ for
  Node; BucketStorage wraps R2 bindings. Storage is configured once per
  process; only image existence is cached, so JSON reads are always fresh.
- Move ValueCache, DataUpdater (data, calendars, archive snapshots),
  ImageProcessor, LocalizationProcessor, festival updaters, status
  generators, social translations, ImageWriter, and Buffer/Threads image
  hosting onto storage. Query hashes are a JSON import.
- Add app/splatnet/auth.mjs: the existing bullet-token code behind auth(),
  so the Worker can route token requests to per-region Durable Objects.
  warmCaches uses auth().refresh().
- Remove the S3/R2 sync, vfs, the dual upload, the sync commands, and
  s3-sync-client. Local runs no longer read from or write to remote buckets.
- Give every test fresh in-memory storage via app/test/setup.mjs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 16:03:33 -08:00