diff --git a/.gitignore b/.gitignore index 1c9c655..e1d1687 100644 --- a/.gitignore +++ b/.gitignore @@ -13,3 +13,6 @@ # Console BOSS keys /console-files/aes_keys.txt /console-files/boss_keys.bin + +# Backups +/backups/ diff --git a/README.md b/README.md index ab15b1f..6913ff3 100644 --- a/README.md +++ b/README.md @@ -99,14 +99,14 @@ understanding of Docker**. ## Safety -- **Back up your MongoDB database!** If you lose it, you will lose all of your PNIDs and Juxt posts. If you have a PNID - signed in on a Wii U, you will also be **stuck with a useless account on your console that you can't delete** because - deleting an account requires a server to authenticate the password. Creating a new account on your server with the - same PNID won't work because each PNID has a numerical ID that is appended to the end of the password before hashing, - so your console will not authenticate with the new PNID. Use `mongodump` - ([docs](https://www.mongodb.com/docs/manual/tutorial/backup-and-restore-tools/)). +- **Back up your data!** If you lose your MongoDB database, you will lose all of your PNIDs and Juxt posts. If you have + a PNID signed in on a Wii U, you will also be **stuck with a useless account on your console that you can't delete** + because deleting an account requires a server to authenticate the password. Creating a new account on your server with + the same PNID won't work because each PNID has a numerical ID that is appended to the end of the password before + hashing, so your console will not authenticate with the new PNID. Use the included `./scripts/backup.sh` script to + back up all of your Pretendo server data. - **Don't delete the `pretendo-network-*` Docker volumes**. You will permanently lose your database (see above) and all - of your Pretendo server data. + of your Pretendo server data. If you need to do something risky, run a backup first. - **Don't use the same P/NNID username on multiple servers.** This applies to NNIDs and PNIDs on the official Pretendo Network server. - **Always verify that you are actually connecting to your own server.** The easiest way is to check the mitmproxy logs diff --git a/compose.yml b/compose.yml index 6124b32..bc8b9a4 100644 --- a/compose.yml +++ b/compose.yml @@ -133,7 +133,7 @@ services: networks: internal: dns: 172.20.0.200 - command: redis-server --save 60 1 --appendonly yes + command: redis-server --save 60 1 redis-commander: image: ghcr.io/joeferner/redis-commander:latest diff --git a/scripts/README.md b/scripts/README.md index 5928cfc..f9c094b 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,52 +1,41 @@ # Scripts -These are important scripts that are used for container setup and -administration. Here is a list of each one and what it does. +These are important scripts that are used for container setup and administration. Here is a list of each one and what it +does. -- `run-in-container/`: These are scripts that, as the name suggests, are run - inside containers. They should not be run directly, but rather through the - other scripts. -- `compile-custom-inkay.sh [--reset]`: This compiles a custom version of the - Inkay patches that whitelists the mitmproxy certificate for the Miiverse - applet. It must be run if you want to use Juxt. Run with `--reset` to reset - the certificate to Pretendo's official certificate. +- `run-in-container/`: These are scripts that, as the name suggests, are run inside containers. They should not be run + directly, but rather through the other scripts. +- `backup.sh [backup_name]`: This backs up MongoDB, PostgreSQL, MinIO, Redis, and mitmproxy data to the `backups` + directory. Run this before doing something risky with the datbases to prevent data loss. +- `compile-custom-inkay.sh [--reset]`: This compiles a custom version of the Inkay patches that whitelists the mitmproxy + certificate for the Miiverse applet. It must be run if you want to use Juxt. Run with `--reset` to reset the + certificate to Pretendo's official certificate. - `create-juxt-community.sh [icon image path] [banner image path]`: - This creates a community in Juxtaposition, which is required before posting - anything. Specify a name, description, and linked title IDs, and optionally - set a custom icon and banner image. It should be run once for each community - you want to create. -- `firstrun-*.sh`: These are scripts that are run on the first run of certain - containers. They are run automatically by the main setup script and shouldn't - be used after that. -- `get-boss-keys.sh [--write]`: This validates the BOSS keys from the dumped key - files in `console-files`. Run with `--write` to validate the keys and write - them to the BOSS server environment variables file. -- `make-pnid-dev.sh `: This sets the access level - of a PNID to developer, which gives it administrative permissions across the - servers. For example, this makes the PNID's posts "verified" on Juxt. It - should be run after creating a new PNID that should have administrative - permissions. -- `setup-environment.sh [Wii U IP address] [3DS IP address]`: - This sets up local environment variables in `*.local.env` files, including - randomly-generated secrets. Important external secrets are logged to - `secrets.txt` in the root of the repository. Specify the server's IP address - (must be accessible to the console) and, optionally, IP addresses for the - consoles for automatic FTP uploads. It should be run whenever this repository - is updated and new servers are added or if the server or Wii U IP addresses - change. -- `setup-submodule-patches.sh`: This sets up the various submodules in`repos` - and applies patches to them. It should also be run whenever this repository is - updated and new servers are added. -- `update-account-servers-database.sh`: This updates the database of account - servers. It should be run whenever a new account server is added to this - repository or the server IP address changes, and it is automatically run by + This creates a community in Juxtaposition, which is required before posting anything. Specify a name, description, and + linked title IDs, and optionally set a custom icon and banner image. It should be run once for each community you want + to create. +- `firstrun-*.sh`: These are scripts that are run on the first run of certain containers. They are run automatically by + the main setup script and shouldn't be used after that. +- `get-boss-keys.sh [--write]`: This validates the BOSS keys from the dumped key files in `console-files`. Run with + `--write` to validate the keys and write them to the BOSS server environment variables file. +- `make-pnid-dev.sh `: This sets the access level of a PNID to developer, which gives it + administrative permissions across the servers. For example, this makes the PNID's posts "verified" on Juxt. It should + be run after creating a new PNID that should have administrative permissions. +- `restore.sh `: This restores MongoDB, PostgreSQL, MinIO, Redis, and mitmproxy data from a specific backup + in the `backups` directory. Note that it also re-runs `setup-environment.sh` to ensure the environment is correct. +- `setup-environment.sh [Wii U IP address] [3DS IP address]`: This sets up local environment + variables in `*.local.env` files, including randomly-generated secrets. Important external secrets are logged to + `secrets.txt` in the root of the repository. Specify the server's IP address (must be accessible to the console) and, + optionally, IP addresses for the consoles for automatic FTP uploads. It should be run whenever this repository is + updated and new servers are added or if the server or Wii U IP addresses change. +- `setup-submodule-patches.sh`: This sets up the various submodules in`repos` and applies patches to them. It should + also be run whenever this repository is updated and new servers are added. +- `update-account-servers-database.sh`: This updates the database of account servers. It should be run whenever a new + account server is added to this repository or the server IP address changes, and it is automatically run by `setup-environment.sh`. -- `update-miiverse-endpoints.sh`: This updates the Miiverse endpoints for the - discovery API server. It currently uses hard-coded values, so it should not be - neccesary to run it again. -- `update-postgres-password.sh`: This updates the password for the PostgreSQL - database. It should be run whenever the password is regenerated, and it is - automatically run by `setup-environment.sh`. -- `upload-3ds-files.sh [--reset]`: This uploads required files from - `/console-files` to your 3DS to help with connection setup. Run with `--reset` - to reset the Juxt certificate to Pretendo's official certificate. +- `update-miiverse-endpoints.sh`: This updates the Miiverse endpoints for the discovery API server. It currently uses + hard-coded values, so it should not be neccesary to run it again. +- `update-postgres-password.sh`: This updates the password for the PostgreSQL database. It should be run whenever the + password is regenerated, and it is automatically run by `setup-environment.sh`. +- `upload-3ds-files.sh [--reset]`: This uploads required files from `/console-files` to your 3DS to help with connection + setup. Run with `--reset` to reset the Juxt certificate to Pretendo's official certificate. diff --git a/scripts/backup.sh b/scripts/backup.sh new file mode 100755 index 0000000..c024d36 --- /dev/null +++ b/scripts/backup.sh @@ -0,0 +1,45 @@ +#! /bin/sh + +set -eu + +git_base=$(git rev-parse --show-toplevel) +. "$git_base/scripts/.function-lib.sh" + +backup_name="${1-}" +if [ -z "$backup_name" ]; then + backup_name="backup_$(date +%Y-%m-%dT%H.%M.%S)" +fi + +backup_dir="$git_base/backups/$backup_name" +if [ ! -d "$backup_dir" ]; then + mkdir -p "$backup_dir" +fi +info "Backing up to $backup_dir" + +docker compose up -d mitmproxy-pretendo mongodb postgres minio redis + +info "Backing up MongoDB..." +docker compose exec mongodb rm -rf /tmp/backup +docker compose exec mongodb mongodump -o /tmp/backup --quiet +docker compose cp mongodb:/tmp/backup "$backup_dir/mongodb" +docker compose exec mongodb rm -rf /tmp/backup + +info "Backing up Postgres..." +docker compose exec postgres sh -c 'pg_dumpall -U "$POSTGRES_USER" --clean' >"$backup_dir/postgres.sql" + +info "Backing up MinIO..." +docker compose exec minio sh -c 'mc alias set minio http://minio.pretendo.cc "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD"' +docker compose exec minio rm -rf /tmp/backup +docker compose exec minio mkdir -p /tmp/backup +docker compose exec minio mc mirror minio/ /tmp/backup +docker compose cp minio:/tmp/backup "$backup_dir/minio" +docker compose exec minio rm -rf /tmp/backup + +info "Backing up Redis..." +docker compose exec redis redis-cli save +docker compose cp redis:/data/dump.rdb "$backup_dir/redis.rdb" + +info "Backing up Mitmproxy..." +docker compose cp mitmproxy-pretendo:/home/mitmproxy/.mitmproxy "$backup_dir/mitmproxy" + +success "Backup completed successfully." diff --git a/scripts/restore.sh b/scripts/restore.sh new file mode 100755 index 0000000..57da659 --- /dev/null +++ b/scripts/restore.sh @@ -0,0 +1,66 @@ +#! /bin/sh + +set -eu + +git_base=$(git rev-parse --show-toplevel) +. "$git_base/scripts/.function-lib.sh" + +if [ -z "${1-}" ]; then + echo "Usage: $0 " + exit 1 +fi + +backup_name="$1" +backup_dir="$git_base/backups/$backup_name" +if [ ! -d "$backup_dir" ]; then + error "Backup directory $backup_dir does not exist." + exit 1 +fi +info "Restoring from $backup_dir" + +if [ "${2-}" != "--force" ]; then + warning "Restoring a backup will overwrite your current Pretendo server data. Backing up your data first is strongly recommended." + printf "Continue? [y/N] " + read -r continue + if [ "$continue" != "Y" ] && [ "$continue" != "y" ]; then + echo "Aborting." + exit 1 + fi +fi + +info "Stopping unnecessary services..." +docker compose down +docker compose up -d mitmproxy-pretendo mongodb postgres minio redis + +info "Restoring MongoDB..." +docker compose exec mongodb rm -rf /tmp/backup +docker compose cp "$backup_dir/mongodb" mongodb:/tmp/backup +docker compose exec mongodb mongorestore /tmp/backup --drop --quiet +docker compose exec mongodb rm -rf /tmp/backup + +info "Restoring Postgres..." +# According to the pg_dumpall documentation, dropping and creating the superuser role is expected to cause an error +docker compose exec -T postgres sh -c 'psql -U "$POSTGRES_USER" -d postgres' <"$backup_dir/postgres.sql" >/dev/null + +info "Restoring MinIO..." +docker compose exec minio /bin/sh -c 'mc alias set minio http://minio.pretendo.cc "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD"' +docker compose exec minio rm -rf /tmp/backup +docker compose cp "$backup_dir/minio" minio:/tmp/backup +docker compose exec minio mc mirror /tmp/backup minio/ --overwrite --remove +docker compose exec minio rm -rf /tmp/backup + +info "Restoring Redis..." +# Redis cannot be running when restoring a dump or it will overwrite the restored dump when it exits +docker compose stop redis +docker compose cp "$backup_dir/redis.rdb" redis:/data/dump.rdb +docker compose start redis + +info "Restoring Mitmproxy..." +docker compose cp "$backup_dir/mitmproxy" mitmproxy-pretendo:/home/mitmproxy/.mitmproxy + +# The restored backup might be using a different password than what is currently in the .env files +info "Now running the environment setup script to regenerate database passwords." +. "$git_base/environment/system.local.env" +"$git_base/scripts/setup-environment.sh" "$SERVER_IP" "${WIIU_IP-}" "${DS_IP-}" + +success "Restore completed successfully."