# Backups and restoring

How the app backs itself up and comes back (plan CORE §3.17, S19; TuteLoop's commands, TuteDesk's posture).
How it is built: [reference/deployment.md](reference/deployment.md). The journeys: the manual's
[Deployment](manual/s19-deployment/00-contents.md), part 1.

**The one rule: the server holds only the public half of the backup key.** Each backup is sealed to it, so a copy of the
server, its disk or its off-site bucket reads nothing. The private half lives in the password manager, and only a
restore needs it.

## What a backup is

One file a run, named for its kind and the moment it was made (UTC): `backup-nightly-2026-10-25T020000Z.bak`. Inside,
packed together:

| Part | From | Notes |
|---|---|---|
| `database.dump` | `pg_dump --format=custom --no-owner --no-privileges` | the whole database; `pg_restore` reads it |
| `vault.tar` | the file vault's folder (`FILE_VAULT_ROOT`) | already ciphertext: the vault's master key opens it, not the backup's |
| `private.tar` | `storage/app/private` | the brand's files, purge archives |
| `manifest.json` | the run | format, kind, when, the starter pack's version, each part's size |

The whole is encrypted with a fresh key, which is sealed to the public half (libsodium's `crypto_box_seal`); the bytes
go in 1 MB pieces under `secretstream`, each piece authenticated and the last one marked. A changed byte, a piece moved
or a file cut short refuses to open. The file is written as `….part` and renamed when whole, mode 0600: nothing reads a
half-written backup.

**Not in a backup: the keys.** `APP_KEY` (sealed columns, two-factor secrets), `FILE_VAULT_KEY` (every file) and the
backup key's private half are kept in the password manager, each on its own. A backup without them restores the rows
but opens no file and no sealed name.

## Setting it up, once

```bash
php artisan backup:key
```

It prints a new key once and writes it nowhere: `BACKUP_PUBLIC_KEY=base64:…` for the server's environment, and the
private half for the password manager, now. Then, in the environment:

| Key | What it is | Shipped |
|---|---|---|
| `BACKUP_PATH` | the folder backups are written to, outside the release (`/home/deploy/backups`) | blank: backups are off |
| `BACKUP_PUBLIC_KEY` | the key's public half | blank: off |
| `BACKUP_REMOTE` | an rclone remote (`spaces:core-backups/db`) each new backup is copied to | blank: one copy, on this server |
| `BACKUP_REMOTE_PRUNE` | whether `backup:prune` removes old copies on the remote too | `true` |
| `BACKUP_WORK` | where a run is put together, then removed | `storage/app/backup-work` |
| `BACKUP_PG_DUMP`, `BACKUP_PG_RESTORE`, `BACKUP_CREATEDB`, `BACKUP_TAR`, `BACKUP_RCLONE` | the tools, when they aren't on the PATH | their names |

The server needs `pg_dump`, `pg_restore` and `createdb` of the database's major version, `tar`, and `rclone` (with the
remote configured: `rclone config`) when there is a remote. Then `php artisan backup:run` once, and
`php artisan app:check`: **Backups** passes when the newest is under 26 hours old and copied off the server.

**A remote nobody on the server can empty.** With credentials that may delete, whoever takes the server can remove
every copy. Better: give the server write-only credentials (or a bucket with object lock), set
`BACKUP_REMOTE_PRUNE=false`, and let the bucket's own lifecycle rule expire copies after 400 days.

## Day to day

| When | What | How |
|---|---|---|
| Every night, 02:00 | a backup, copied off the server | the scheduler: `backup:run` (once set up) |
| Every night, 02:50 | what is past its keeping time removed, here and on the remote | the scheduler: `backup:prune` |
| Each deployment | a backup before `app:deploy` changes the database | `deploy/server/release.sh`: `backup:run --kind=predeploy` |
| Any time | the list | `php artisan backup:list` |
| Any time | a backup now | `php artisan backup:run` |
| Monthly | the drill: prove the newest backup restores | below |

**Kept**, by the date in each name (never the file's times): every backup for **14 days**; then the newest nightly of
each week for **8 weeks** and of each month for **12 months**. A pre-deployment backup is kept for the 14 days only.
`php artisan backup:prune --dry-run` says what would go. The numbers are `backup.keep` in `config/operations.php`.

The readiness check (**Backups**, on the operations page and in `app:check`):

| It says | When |
|---|---|
| Passing | the newest is under 26 hours old and a remote is named |
| Needs a look | the newest is older (a night missed), or there is no remote: one copy only, on this server |
| Failing | set up, but no backup in the folder; or, in production, not set up at all |
| Off | not set up (development) |

## The drill, monthly

A backup nobody has restored is a hope. On a machine that may hold the private half for a few minutes (never the
server), with the app's code and the same PostgreSQL major version:

```bash
rclone copy spaces:core-backups/db/backup-nightly-2026-10-25T020000Z.bak storage/app/backups/   # or scp it
BACKUP_PATH=storage/app/backups php artisan backup:restore backup-nightly-2026-10-25T020000Z.bak --into=drill_20261025
psql drill_20261025 -c 'select count(*) from tenant'   # what you expect to see
dropdb drill_20261025
```

`--into` makes a new database of that name (it refuses one that exists), restores the database into it and touches
nothing else: no maintenance, no folder. Note the date and the result in the operations runbook.

## Restoring, for real

From the command line only (a button would make wiping a live system one click away):

```bash
php artisan backup:list
php artisan backup:restore backup-nightly-2026-10-25T020000Z.bak
```

It asks twice (*Restore … over this system?*, then *Sure? It can't be undone.*), then the private half (typed, not
shown; or `--key-file=` for a script). Then it:

1. puts the app into maintenance (`php artisan down`);
2. opens the backup, checks what is inside, and restores the database with `pg_restore --clean --if-exists` in **one
   transaction**: wholly, or not at all;
3. swaps each folder: the live one set aside, the backup's unpacked in its place, the set-aside one removed;
4. brings the app back (`php artisan up`).

If a step fails it says what, and **the app stays in maintenance**: look at what was restored before `php artisan up`.

| Option | What it does |
|---|---|
| `--into=<database>` | the drill: a new database only, nothing live touched |
| `--database-only` | the database, the folders left as they are |
| `--files-only` | the folders, the database left as it is |
| `--key-file=<path>` | read the private half from a file (delete the file afterwards) |
| `--force` | don't ask (scripts only) |

**On a fresh server**: the code at the same release, the same `.env` (with `APP_KEY` and `FILE_VAULT_KEY` from the
password manager), `createdb` the database, the backup in `BACKUP_PATH`, then `backup:restore … --force`, then
`php artisan app:deploy` (it migrates anything newer) and `php artisan app:check`.

## Targets and known gaps

- **Recovery point:** up to 24 hours (nightly), and the moment before each deployment. **Recovery time:** about 15
  minutes for a few gigabytes, after the server exists.
- **An erasure since the backup is undone by restoring it.** Crypto-shredding (CORE-D17) destroys a person's key in the
  live database; a backup made before still holds it. After a restore, run again every erasure carried out since the
  backup was made (the privacy requests list of the day, kept outside the restored system). An erasure ledger kept
  off the database is a follow-up.
- **The drill is by hand**, because only a machine with the private half can do it; nothing records it yet.
- **The database dump is plain while a run is packed**, in the work folder (mode 0700, removed at the end of the run).
- Whole-file encryption streams in pieces, so size isn't limited by memory; the work folder needs room for the dump and
  the folders twice over.