Files
splatoon3.ink/app/common/storage/index.mjs
Matt Isenhower e7388fce6a Archive data snapshots from the Worker to DigitalOcean Spaces
With ARCHIVE_DATA set, the Worker writes each published data file to the
archive Space, at the same YYYY/MM/DD/<date>.<time>.<name>.json keys, ACL,
and content type as the Node uploader (DataArchiver):

- S3ArchiveStorage signs PUTs with aws4fetch, tries three times with
  backoff, and stops early on errors that will not improve.
- A snapshot that cannot be written is a warning and a Sentry event, never
  a failed update.
- The fast lane holds its snapshots until after social posts (even failed
  ones) and then uploads the newest copy of each key, so a slow archive
  bucket never delays a post. The slow lane uploads as it goes, to avoid
  holding large X-Rank files in memory.
- Dev writes to its own Space (splatoon3ink-archive-test); production has
  no archive settings until cutover. Daily compression is still to come.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 22:47:38 -08:00

47 lines
1.7 KiB
JavaScript

// Where the backend reads and writes its files.
//
// There are three areas, each with the same small interface (see BucketStorage):
// - public: what the site serves (data/*.json, data/*.ics, splatnet/* images, status-screenshots/*)
// - private: tokens and other state (cache/*.json)
// - archive: data snapshots (hourly for most files; X-Rank's are stamped to the second)
//
// Keys are the R2 object keys. The Node runner maps them onto dist/ and storage/ (see
// filesystem.mjs); the Worker uses R2 bindings, and writes archive snapshots straight to the
// archive bucket on DigitalOcean Spaces (S3ArchiveStorage). Storage is configured once per process or
// isolate, because nothing but image existence is cached between calls.
//
// FilesystemStorage is deliberately not re-exported: it imports node:fs, and this module is
// part of the Worker's import graph.
export { default as BucketStorage } from './BucketStorage.mjs';
export { default as MemoryBucket } from './MemoryBucket.mjs';
export { default as S3ArchiveStorage } from './S3ArchiveStorage.mjs';
export { contentTypeFor, DATA_CACHE_CONTROL } from './metadata.mjs';
let configured = null;
/**
* @param {{ public: object, private: object, archive: object }} areas BucketStorage or FilesystemStorage instances
*/
export function configureStorage(areas) {
for (let name of ['public', 'private', 'archive']) {
if (!areas?.[name]) {
throw new Error(`Missing ${name} storage`);
}
}
configured = areas;
}
/**
* @returns {{ public: object, private: object, archive: object }} The configured storage areas
*/
export function storage() {
if (!configured) {
throw new Error('Storage is not configured; call configureStorage() at startup');
}
return configured;
}