# Part 1 · Operations

## 1.1 Reading the operations page

**Who** · The Service (`operations.view`).

**Where** · The platform menu › **Operations** (`/console/operations`).

**Before you start** · Nothing. The page reads the last recorded check; before the first one it says **Not checked
yet**.

**The steps**
1. **The hero** says the whole in a line: **Everything is running**, **Running, with something to look at**, or how
   many checks are failing and which. **Show me the check** jumps to the first failing one. Beside it, the uptime of
   the last 30 days (the share of checks with nothing failing) and a bar for each day; hover a bar for its day.
2. **The four tiles**: jobs run, e-mails sent, the queue wait (95 in 100 jobs waited less) and failed jobs, each for the
   last 24 hours with a line of its hours and yesterday's figure. **Failed jobs** opens them (1.4).
3. **Readiness**: the thirteen checks, each with its state, its words and how long it took.
4. **Queues**: each queue's waiting jobs, its workers and how many it runs a minute. **Scheduled**: each scheduled task,
   when it runs and when next; all are marked **overdue** while the scheduler's heartbeat is stale.
5. **Live activity**: the latest changes in every workspace and the platform. **Pause the feed** to read it while the
   page refreshes. **Space**: the file vault's disk, the database, the backups kept. **Purges**: the last purge.

**What the record shows** · Nothing: reading the page changes nothing, and its refreshes don't keep your session awake.

**Who is told** · Nobody.

**What can go wrong**
- *"This page isn't open to you."* Only the Service (or someone given `operations.view`) sees it.
- *The tiles are flat.* The queue isn't the database queue here (`QUEUE_CONNECTION`), so no job is counted.

## 1.2 Checking again

**Who** · The Service (`operations.manage`).

**Where** · The operations page: **Check again** (every check) or the round arrow on one check.

**Before you start** · Fix what the check said, then check it again to see it pass.

**The steps**
1. **Check again** runs all thirteen and keeps the run: the hero and the uptime count it.
2. The arrow on one check runs that one alone and keeps the run with the others' last answers.

**What the record shows** · A readiness run (not in the audit log: it changes nothing).

**Who is told** · Nobody: only the schedule's checks e-mail (1.3).

**What can go wrong**
- *Scheduler: "No beat yet".* On a server, the cron line isn't running (1.6). On a laptop the scheduler runs only while
  `php artisan schedule:work` does.
- *"The check itself failed: …".* The check couldn't run (the database or the cache is down): the words are the error.

## 1.3 When a check starts failing

**Who** · The Service.

**Where** · Your inbox, then the operations page.

**Before you start** · Nothing.

**The steps**
1. The e-mail ("a readiness check is failing") names the checks that started failing and when, with a link to the
   page.
2. Open the page; the failing check's words say what to run or set.
3. Fix it, then **Check again** (1.2).

**What the record shows** · The e-mail log: the e-mail, to the Service.

**Who is told** · The Service, once when a check starts failing: not again while it keeps failing, and again if it
passes and then fails later. A check that has always failed since the first run counts as starting then.

**What can go wrong**
- *No e-mail came.* Mail itself is the check failing (Mail), or the scheduler is (no schedule, no check): the page
  still shows it.

## 1.4 Retrying or forgetting a failed job

**Who** · The Service: reading needs `operations.view`; retrying and forgetting `operations.manage`.

**Where** · The platform menu › **Failed jobs** (`/console/failed-job`), or the **Failed jobs** tile.

**Before you start** · Read why it failed: retrying a job whose cause is still there fails it again.

**The steps**
1. The list: each job's name, queue, workspace and language, when it failed and why (the error's first line).
2. **Open** a job for its error and the first frames of its trace, and its tries. Its input is never shown: it can
   carry a secret or a file's contents.
3. **Retry** puts it back on its queue as it was, with its workspace and language; **Retry them all** does that for
   every one. **Forget it** removes it for good.

**What the record shows** · The audit log, as the platform's: **Failed job retried** or **Failed job forgotten**, with
the job, its queue and why it failed.

**Who is told** · Nobody.

**What can go wrong**
- *It is back in the list.* It failed again: the cause is still there.

## 1.5 Deploying a release (the server's operator)

**Who** · Whoever deploys, on the server.

**Where** · The deploy script, in the release's folder.

**Before you start** · The new release's code is in place and `composer install --no-dev` has run.

**The steps**
1. `php artisan app:deploy` runs every step a release needs, in order: migrations, the option lists, the roles, the
   notifications, the mail partitions, the brand check, then the caches. Each prints a tick and its time.
2. The first step that fails stops it ("Stopped at step 3 of 9, lookups:check: it failed. Nothing after it ran.") with
   that step's output, and the command fails, so the deploy script stops too.
3. Switch the release live, then `php artisan queue:restart` so the workers load the new code.
4. `php artisan app:check`: one line a check; it fails only on a ✗.

**What the record shows** · Each step's own (a migration, a role change in the audit log).

**Who is told** · Nobody.

**What can go wrong**
- *A step failed.* Its output says why; fix it and run `app:deploy` again: every step is safe to run twice.

## 1.6 Setting up the server for the checks (the server's operator)

**Who** · Whoever runs the server.

**Where** · The server's crontab, its process supervisor, and `.env`.

**Before you start** · `reji/docs/reference/operations.md` lists every setting.

**The steps**
1. The cron line: `* * * * * cd <the app> && php artisan schedule:run >> /dev/null 2>&1`. The heartbeat beats every
   minute from then.
2. A queue worker under a supervisor: `php artisan queue:work database --sleep=3 --tries=3`.
3. Optional: `SCHEDULE_HEARTBEAT_URL`, a monitoring service's address pinged each minute, so something outside
   notices when the whole server stops.
4. Backups: `BACKUP_PATH`, the folder the nightly backup writes to. Until it is set, Backups is **Off**.
5. Production: `APP_DEBUG=false`, `SESSION_SECURE_COOKIE=true`, an `https://` `APP_URL`, a real mailer and sender.

**What the record shows** · The next check passes.

**Who is told** · Nobody.

**What can go wrong**
- *Backups "one night was missed".* The newest file is over 26 hours old: the backup job didn't run or didn't write.