Deploy runbook

One-time setup an operator does per client deployment, before the first workflow_dispatch run of deploy.yml. Everything here is provisioned once; deploy-client.yml (spec docs/adr/0001-event-platform-v1.md §8.1) then runs unattended. There is no service-account key anywhere in this flow — auth is Workload Identity Federation (WIF), so nothing here is a secret an operator has to store, rotate manually, or worry about leaking from a runner log.

Replace every <...> placeholder below. Nothing here is a value you leave as written.

0. Naming#

Pick, once:

• <GCP_PROJECT_ID> — the Firebase/GCP project for this client (create it first — §2 below covers Firebase-specific prerequisites).
• <GH_ORG>/<GH_REPO> — this repository, e.g. your-org/run-of-show.
• <CLIENT_ENV> — the GitHub Environment name for this client, e.g. cjs2027. This is the value operators pass as client to workflow_dispatch, and it must exactly match the AUTO_DEPLOY_ENVIRONMENTS entry if this client also auto-deploys on push (§4).
• <DEPLOY_SA> — the deploy service account's short name, e.g. deploy-cjs2027.

1. Workload Identity Federation#

Each client project has its own pool and provider. A provider conditioned on one environment cannot be shared, as a rehearsal showed (§12.3). The site builder (§12) does all of this for a site made in the client console. The commands below are for a site made by hand.

gcloud config set project <GCP_PROJECT_ID>

          # One-time per GCP project — skip if it already exists.
          gcloud iam workload-identity-pools create "github-actions" \
            --location="global" \
            --display-name="GitHub Actions"

          gcloud iam workload-identity-pools providers create-oidc "github" \
            --location="global" \
            --workload-identity-pool="github-actions" \
            --display-name="GitHub OIDC" \
            --attribute-mapping="google.subject=assertion.sub,attribute.repository=assertion.repository,attribute.repository_owner=assertion.repository_owner,attribute.ref=assertion.ref,attribute.repository_and_ref=assertion.repository + '#' + assertion.ref" \
            --attribute-condition="assertion.repository_owner == '<GH_ORG>'" \
            --issuer-uri="https://token.actions.githubusercontent.com"
          

--attribute-condition is the guardrail: without it, any GitHub Actions workflow from any repo that can forge a matching subject claim could request a token. Scoping it to repository_owner is enough at the pool level because the per-client principal binding below (§2) does the real narrowing — down to one repository and one ref (refs/heads/main) together. attribute.repository_and_ref is a mapped custom attribute for exactly that: IAM principalSet bindings match on a single attribute value, and a repository-only or ref-only binding cannot be combined with a second one (multiple principalSet members on a role binding are OR'd, not AND'd) — so a repository+ref compound value is the only way to express "this repo, this branch, nothing else" as one condition.

This closes a real gap, not a theoretical one. Binding only on attribute.repository (an earlier draft of this runbook did exactly that) lets any workflow run in this repository — including one dispatched by a collaborator against a feature branch via gh workflow run deploy.yml --ref <branch> -f client=<env> — impersonate the deploy service account and ship that branch's code with the client's real credentials. An in-workflow ref check (deploy-client.yml's guard job) is necessary but not sufficient on its own: it runs as code from the ref being deployed, so a workflow-modifying commit on that same branch could remove or bypass the check before GCP is ever asked. The principalSet condition is enforced by Google's IAM outside the runner and outside the repository's control entirely, which is why it is the primary control and the in-workflow check is defense-in-depth only.

Capture the full provider resource name, which is that client's WIF_PROVIDER environment variable (§3):

gcloud iam workload-identity-pools providers describe "github" \
            --location="global" \
            --workload-identity-pool="github-actions" \
            --format="value(name)"
          # projects/<PROJECT_NUMBER>/locations/global/workloadIdentityPools/github-actions/providers/github
          

2. Per-client service account#

This section's list is incomplete: §12.3 has the grants and APIs a rehearsal showed it needs as well.

Firebase project prerequisites, first (once per client GCP project):

gcloud services enable \
            firebase.googleapis.com \
            cloudfunctions.googleapis.com \
            cloudbuild.googleapis.com \
            run.googleapis.com \
            firestore.googleapis.com \
            secretmanager.googleapis.com \
            cloudscheduler.googleapis.com \
            --project=<GCP_PROJECT_ID>

          # If the project has never had Firebase added:
          #   https://console.firebase.google.com/ → Add project → select <GCP_PROJECT_ID>
          # Then, in the console: enable Firestore (Native mode), enable Storage, add a
          # Hosting site (or use the default one — its ID is EVENT_HOSTING_SITE, §3).
          

Now the deploy principal:

gcloud iam service-accounts create "<DEPLOY_SA>" \
            --project=<GCP_PROJECT_ID> \
            --display-name="Event Runner deploy (<CLIENT_ENV>)"

          DEPLOY_SA_EMAIL="<DEPLOY_SA>@<GCP_PROJECT_ID>.iam.gserviceaccount.com"

          for role in \
            roles/firebasehosting.admin \
            roles/firebaserules.admin \
            roles/datastore.indexAdmin \
            roles/storage.admin \
            roles/cloudfunctions.developer \
            roles/run.developer \
            roles/iam.serviceAccountUser \
            roles/secretmanager.secretAccessor \
            roles/secretmanager.viewer \
            roles/cloudbuild.builds.editor \
            roles/artifactregistry.writer \
            roles/serviceusage.serviceUsageConsumer \
            roles/datastore.user \
            roles/cloudscheduler.admin
          do
            gcloud projects add-iam-policy-binding <GCP_PROJECT_ID> \
              --member="serviceAccount:${DEPLOY_SA_EMAIL}" \
              --role="${role}" \
              --condition=None
          done
          

Notes on that list: cloudfunctions.developer + run.developer + iam.serviceAccountUser + cloudbuild.builds.editor + artifactregistry.writer are what firebase deploy --only functions needs end to end (Cloud Functions v2 builds through Cloud Build and Artifact Registry, and the deploying principal must be allowed to act as the runtime service account). secretmanager.viewer lets the deploy read a bound secret's metadata and access policy. secretmanager.secretAccessor is required only once the environment actually uses a secret (EMAIL_PROVIDER_API_KEY etc., spec §2.1) — grant it up front so onboarding a provider later is a config change, not an IAM change. This binding is only what the deploying identity needs to reference a secret while wiring up defineSecret at deploy time — it does NOT cover the deployed function actually reading the secret's value at runtime. That is a separate binding, on the project's default compute service account, done per secret in §2's Secret Manager step below; skip it and every defineSecret-backed function fails at cold start or at read, not at deploy. datastore.user is what generate-content.cjs needs to read Firestore at deploy time (spec §8.6); datastore.indexAdmin is separate and is what deploys firestore.indexes.json. cloudscheduler.admin is what firebase deploy --only functions needs to create/update the Cloud Scheduler job behind cleanupExpiredAuthChallenges (an onSchedule v2 function, functions/src/auth/otp.cjs) — every deployment has at least this one scheduled function, so the role is not conditional the way secretmanager.secretAccessor is. The same role creates the job behind endExpiredStaffAccess (functions/src/admin/access.cjs, every minute, issue #331). It needs no secret. It reads config/bootstrap once a minute and writes only when a staff grant's end time has passed. The same role creates the job behind runScheduledReminders (functions/src/reminders/run.cjs, every 60 minutes, issue #518). That function binds the mail provider's secrets, and the operator webhook's when EVENT_OPERATOR_NOTIFIER=webhook, so the secret bindings in §2 cover it. It sends nothing until staff approve a reminder schedule in the admin. Scheduled attendee reminders go on the announcement stream: finish the Postmark Broadcasts stream (docs/POSTMARK_PROVISIONING.md, section 2a), set config/providers.email.announcementStream, and send a test announcement before staff turn the attendee schedule on. Until then the admin refuses to approve it and says why.

Bind the deploy service account so only this repository, on refs/heads/main, may impersonate it — both conditions in one principalSet, via the combined attribute mapped in §1:

gcloud iam service-accounts add-iam-policy-binding "${DEPLOY_SA_EMAIL}" \
            --project=<GCP_PROJECT_ID> \
            --role="roles/iam.workloadIdentityUser" \
            --member="principalSet://iam.googleapis.com/projects/<PROJECT_NUMBER>/locations/global/workloadIdentityPools/github-actions/attribute.repository_and_ref/<GH_ORG>/<GH_REPO>#refs/heads/main"
          

Verify it — the policy should show exactly one member, and it must end in #refs/heads/main:

gcloud iam service-accounts get-iam-policy "${DEPLOY_SA_EMAIL}" --project=<GCP_PROJECT_ID> \
            --format="json(bindings)"
          

This is the actual enforcement. It holds regardless of what any workflow file in the repository says, because Google's OIDC token exchange evaluates the ref claim GitHub put in the token — not anything the workflow run controls — before minting a credential. A workflow_dispatch run against a feature branch (--ref some-branch) presents a token whose repository_and_ref attribute does not match <GH_ORG>/<GH_REPO>#refs/heads/main, so google-github-actions/auth fails at the token exchange with permission_denied — before deploy-client.yml runs a single deploy step, and even if that workflow's own guard job ref check had been edited or removed on that branch. workflow_dispatch from main itself is unaffected: its ref is refs/heads/main, which matches.

Second layer: a GitHub-side deployment branch rule. Belt-and-suspenders, and cheap to set up — repo Settings → Environments → <CLIENT_ENV> → Deployment branches and tags → Selected branches and tags → add main only. This makes GitHub itself refuse to start any job whose environment: is <CLIENT_ENV> unless the run's ref is main, independent of both the WIF condition and the in-workflow check. Combine it with Required reviewers on the same screen for any client where a human should see a dispatch before it runs.

Storage rules and the account document. storage.rules calls firestore.exists on public profile photo reads and owner deletes. Grant this role once per project, before the first deploy in §5. Without it, uploaded profile photos cannot load and cards fall back to initials. The deploy job does not grant it.

PROJECT_NUMBER="$(gcloud projects describe <GCP_PROJECT_ID> --format='value(projectNumber)')"
          gcloud projects add-iam-policy-binding <GCP_PROJECT_ID> \
            --member="serviceAccount:service-${PROJECT_NUMBER}@gcp-sa-firebasestorage.iam.gserviceaccount.com" \
            --role="roles/firebaserules.firestoreServiceAgent" \
            --condition=None
          

Storage triggers. mediaImageSizes and mediaImageSizesCleanup run on Cloud Storage events (issue #333). Eventarc delivers those events through Pub/Sub, so the Cloud Storage service agent must publish to Pub/Sub. The deploy service account cannot grant that role. Grant it once per project, before the first deploy that has these triggers:

GCS_AGENT="$(gcloud storage service-agent --project=<GCP_PROJECT_ID>)"
          gcloud projects add-iam-policy-binding <GCP_PROJECT_ID> \
            --member="serviceAccount:${GCS_AGENT}" \
            --role="roles/pubsub.publisher" \
            --condition=None
          

Without it, the functions deploy fails to create these two triggers. The bucket in EVENT_STORAGE_BUCKET must be in the same location as EVENT_FIREBASE_REGION, or a multi-region that holds it (for example US for us-central1). Eventarc refuses a storage trigger in any other region.

3. GitHub Environment#

Create the environment (repo Settings → Environments → New environment → name it exactly <CLIENT_ENV>), then set its variables and secrets. Every value below is a GitHub Environment variable (vars.*), not a secret — none of it is a credential; the credential-shaped value (the WIF provider resource name, the service account email) is a Google-side ACL check, not a bearer token, and none of the Tier A values are confidential either.

App Check activation requires two successful deploys. First set VITE_FIREBASE_APP_CHECK_SITE_KEY and wait for that deploy to finish. Then set EVENT_APP_CHECK_ENFORCED=true and deploy again. Do not enable both settings in the first deploy because Functions deploys before the new web bundle.

Full list, with descriptions: .env.example.

The only secrets deploy-client.yml reads are the provider-specific ones from spec §2.1/§8.2 — set them here too, as environment secrets (Settings → Environments → <CLIENT_ENV> → Secrets), only the rows that apply to the providers selected above:

There is deliberately no FIREBASE_SERVICE_ACCOUNT secret in this list — WIF (§1–2) replaces the service-account-key deploy credential entirely.

There is also deliberately no EMAIL_ACCOUNT_API_KEY here: it's the Postmark account token, consumed only by the operator-run scripts/verify-sender-domain.cjs, not by any deployed function — none of the defineSecret lists below bind it. It stays in operator-controlled storage and is never provisioned into a client's GitHub Environment or Secret Manager; see docs/POSTMARK_PROVISIONING.md §3.

These provider secrets are consumed by the deployed Cloud Functions through defineSecret (Secret Manager), not baked into the functions env file the functions job writes — create them in Secret Manager once, before the first functions deploy for this client:

echo -n "<the actual API key>" | gcloud secrets create EMAIL_PROVIDER_API_KEY \
            --project=<GCP_PROJECT_ID> --data-file=- --replication-policy=automatic
          # repeat per secret this client's provider selection actually needs
          

Then bind secretAccessor on each secret to the function's RUNTIME service account — not the deploy service account. These are two different identities with two different jobs: ${DEPLOY_SA_EMAIL} (§2, above) only needs to reference the secret while deploying; the function's runtime identity is what actually reads the secret's value on every invocation, and it is a separate principal that ${DEPLOY_SA_EMAIL}'s own secretmanager.secretAccessor grant does not cover. Cloud Functions v2 runs as the project's default compute service account unless a function explicitly sets serviceAccount (this repo's functions do not), so:

PROJECT_NUMBER=$(gcloud projects describe <GCP_PROJECT_ID> --format="value(projectNumber)")
          RUNTIME_SA_EMAIL="${PROJECT_NUMBER}-compute@developer.gserviceaccount.com"

          for secret in EMAIL_PROVIDER_API_KEY; do   # repeat the list for every secret this client uses
            gcloud secrets add-iam-policy-binding "${secret}" \
              --project=<GCP_PROJECT_ID> \
              --member="serviceAccount:${RUNTIME_SA_EMAIL}" \
              --role="roles/secretmanager.secretAccessor"
          done
          

Skipping this step is a deploy-succeeds-runtime-fails trap: firebase deploy --only functions does not read the secret's value, so the deploy itself can succeed with this binding missing, and the function then fails at cold start (or at the first read, depending on defineSecret vs. lazy access) with a Secret Manager permission error — which surfaces as a broken client feature, not a red CI run.

(Optional) restrict who can approve deploys to this environment: Settings → Environments → <CLIENT_ENV> → Deployment protection rules → Required reviewers.

4. AUTO_DEPLOY_ENVIRONMENTS (push auto-deploy)#

Repository-level, not per-environment (repo Settings → Secrets and variables → Actions → Variables → New repository variable):

Name:  AUTO_DEPLOY_ENVIRONMENTS
          Value: cjs2027,summit2026
          

A comma-separated list of <CLIENT_ENV> names. Every push to main fans out to each one (spec §8.1) — deploy.yml's resolve job parses this (scripts/lib/deploy-matrix.cjs), and a client absent from the list only ever deploys via workflow_dispatch. Leave it unset (or empty) while a client is new and you want every deploy to be a manual, watched dispatch; add it once the client is comfortable auto-deploying on merge.

5. First deploy (provisioning a fresh project)#

A brand-new project has no config/event Firestore document yet — generate-content.cjs (the content job) throws "run scripts/init-event.cjs first" against one, on purpose (spec §8.6). The content job runs before any operator has ever had the chance to run init-event.cjs, so the first dispatch against a fresh project has to skip it. Two dispatches, in order:

Step 1 — bootstrap dispatch. Deploys only rules/indexes/storage (provision) and functions; skips content/build/hosting/post/smoke entirely.

Actions tab → Deploy → Run workflow
            client: <CLIENT_ENV>
            provision: true
            bootstrap: true
          

deploy-client.yml's provision job deploys firestore:rules, firestore:indexes, and storage on every run regardless of the provision input (spec §8.1) — that input exists for operator intent and log clarity, not as a gate. The Storage rules role in §2 is required before this dispatch. bootstrap: true is the actual gate: it is what skips content (and, transitively, build/hosting/post/smoke) on this run. A dispatch run always deploys functions regardless of bootstrap, because a fresh project has none yet and gating that on a paths filter would provision an empty, broken deployment (spec §8.1).

The first Functions deploy contains event handlers with retry enabled. Firebase CLI requires an explicit acceptance for a new retry policy, even in non-interactive mode. --force gives that acceptance, but it also accepts function deletion and unsafe trigger migration. So scripts/deploy-functions.cjs passes --force in two narrow cases only.

Before each Functions deploy, the script reads the complete Functions inventory through the pinned firebase-tools backend used by functions:list. It rejects missing regional availability or any unreachable region. A failed or partial inventory stops the deploy.

Case 1: verified-empty bootstrap. On a bootstrap run, the script passes --force to the full --only functions deploy only when the inventory reports no Functions. For that initial deploy, --force also accepts Firebase CLI's default Artifact Registry cleanup policy: container images older than one day are deleted. This is build-artifact retention only; it does not change deployed Functions or event data.

Case 2: New retry policies. On every other run, the script also discovers the local Functions with the pinned firebase-tools runtime, the same discovery that firebase deploy runs. A discovery failure stops the deploy. A retry policy is new when the Function is absent or its deployed trigger has retry disabled. Existing Functions qualify only with the same platform, codebase and trigger identity, including filters and document paths. An omitted trigger region inherits the existing region. The script first runs one targeted deploy and logs its targets:

npx firebase deploy --only functions:<id>[,functions:<id>...] \
            --project <GCP_PROJECT_ID> --non-interactive --force
          

Then it runs the normal full deploy without --force. The handlers now exist with retry enabled, so Firebase does not ask again. Firebase continues to stop for function deletion, unsafe trigger migration, and other protected changes. The script refuses the targeted deploy, and deploys nothing, in these cases:

• A Function with the same id exists outside the declared target regions.
• A selector also matches a deployed Function outside the intended creates or retry upgrades. Firebase matches functions:<id> against the dash-separated parts of each id, so functions:foo also selects foo-bar.
• A selector also matches a local Function outside the intended retry-policy targets.
• A retry setting or a retry handler's region is not a literal value.

With these checks, the targeted plan contains only creates and retry upgrades with unchanged trigger identity. It cannot delete or recreate a Function. The targeted deploy can also accept other prompts for the targeted creates and retry upgrades: their min-instances cost, an API they need, and a missing Artifact Registry cleanup policy. A client deployment that already deploys without --force already has a cleanup policy.

Retry-enabled event handlers can retry failed executions for up to seven days, so they must remain idempotent. Review every new retry handler for idempotency before it merges.

Retry enabled on an existing Function. Case 2 accepts this automatically only when the normalized trigger identity is unchanged. A changed event type, filter, path, platform or codebase stops before deployment. Review that migration separately; do not add blanket --force.

Recover a partial first Functions deploy. If the first bootstrap creates some Functions and then fails, read the failed run and repair its original API, permission, quota, or provider-configuration failure. Keep the commit, EVENT_FIREBASE_REGION, and GitHub Environment provider settings unchanged, and rerun the same bootstrap: true dispatch. The rerun sees a nonempty inventory, so it does not force the full deploy. Case 2 creates the absent retry handlers first, and the full deploy then runs without --force.

Check after the full deploy. After the full deploy exits 0, the script discovers the local Functions, if it has not yet, and reads the complete inventory again. Each local Function must be there, with the same id, in each of its regions. If a Function is not there, or its region is not a literal value, the step fails and names the Function. firebase-tools 15.22.4 can exit 0 when a create gets a quota error, before the create is done, and it then prints no summary. Run 39329645906 did this to downloadEventPackage. This check makes such a run fail.

Quota errors (HTTP 429). A large deploy can exceed the Cloud Functions limit "Per project mutation requests per minute per region". The log then shows Quota Exceeded and failed to update function or failed to create function lines. Firebase CLI retries these, and most later show Successful update operation or Successful create operation. If the step fails, or it names a Function that is not deployed, wait a minute and run the same deploy again. Do not change the commit. The rerun skips each Function that is active and unchanged, so it sends fewer requests and creates the Functions that are not there. Then run the smoke test again.

To read the complete inventory from an operator's machine, use a clean checkout with Node 22 and npm ci complete:

EVENT_FIREBASE_PROJECT_ID=<GCP_PROJECT_ID> node - <<'NODE'
          const { readCompleteInventory } = require('./scripts/deploy-functions.cjs');
          readCompleteInventory(process.env.EVENT_FIREBASE_PROJECT_ID).then((functions) => {
            const rows = functions
              .map(({ id, region, platform }) => ({ id, region, platform }))
              .sort((a, b) => `${a.region}/${a.id}`.localeCompare(`${b.region}/${b.id}`));
            process.stdout.write(`${JSON.stringify(rows, null, 2)}\n`);
          }).catch((error) => {
            console.error(error.message);
            process.exitCode = 1;
          });
          NODE
          

Upgrade order for unpublished counts. Pause CMS saves, publishes, deletes, and seed jobs during the first upgrade. Deploy Firestore rules and all Functions, then wait for the deploy and any older in-flight CMS requests to finish. Before deploying the count-reading UI or resuming edits, make an authenticated staff-admin POST to cmsEnsurePendingCounts with JSON body {"force":true}. Require HTTP 200 with {"ok":true} and verify cmsMeta/pending against the Unpublished changes page. This recounts all six dirty draft collections in one transaction, including any drift caused by old and new function versions overlapping during deployment. Never initialize an existing site's count to zero. The same forced request repairs a count that is too high after an out-of-band data edit.

Ordinary requests to cmsEnsurePendingCounts with {} leave a valid summary unchanged. Draft saves, publishes, and deletes also prepare a missing or invalid summary before changing the count. Run only the upgraded init-event.cjs or seed-demo-event.cjs after the deploy; each prepares the count before its first CMS write and uses the atomic save and publish paths. Existing unpublished edits remain counted, and a seed dry run remains read-only.

Step 2 — seed content, from an operator's machine, once the bootstrap dispatch succeeds:

export FIRESTORE_EMULATOR_HOST=  # unset — this targets the real project
          export EVENT_FIREBASE_PROJECT_ID=<GCP_PROJECT_ID>
          export GOOGLE_APPLICATION_CREDENTIALS=<path to your own gcloud ADC, for this one-time step>
          node scripts/init-event.cjs --answers client-answers.json --admin ops@example.org
          node scripts/init-event.cjs --check
          

init-event.cjs runs from an operator's machine (or a follow-up dispatch of a future provisioning workflow), not from deploy-client.yml — it is the one-time content bootstrap, not a repeatable deploy step. See scripts/README.md. --admin seeds the operator tier; --staff (repeatable) seeds staff, who run content, schedule, speakers, and attendees but not branding, features, access, or deployment settings. Both can be granted later from admin Settings → Access. A re-run of init never removes a grant made there, and it never puts back an address removed there: on a re-run the answers file's lists are not re-applied, and only --admin and --staff add.

Step 3 — normal dispatch, now that config/event exists:

Actions tab → Deploy → Run workflow
            client: <CLIENT_ENV>
            provision: false
            bootstrap: false   (the default — leave it unchecked)
          

This run's content job succeeds, and build/hosting/post/smoke deploy the live site. From here on, either dispatch this way again for any change, or add <CLIENT_ENV> to AUTO_DEPLOY_ENVIRONMENTS (§4) so a push to main deploys it automatically — bootstrap is never true on a push run, so a client already past step 3 never needs it again.

6. Verifying the setup#

• gcloud iam service-accounts get-iam-policy "${DEPLOY_SA_EMAIL}" --project=<GCP_PROJECT_ID> --format="json(bindings)" shows the workloadIdentityUser binding from §2, and its member ends in .../attribute.repository_and_ref/<GH_ORG>/<GH_REPO>#refs/heads/main — not attribute.repository/<GH_ORG>/<GH_REPO> alone. A repository-only binding is the gap this runbook now closes: it lets any dispatched run on any branch of this repository impersonate the deploy service account.

• Confirm the negative case as well: gh workflow run deploy.yml --ref <some-non-main-branch> -f client=<CLIENT_ENV> must fail at the google-github-actions/auth step with a token-exchange permission_denied — that is the WIF condition doing its job. It should also fail earlier, at "Waiting for review" or immediately, if the deployment branch rule (§2) is set, since GitHub itself won't start a job against <CLIENT_ENV> from that ref.

• A workflow_dispatch run of deploy.yml with client: <CLIENT_ENV> from main should reach the smoke job and pass — it OPTIONS-preflights every endpoint in .github/smoke-endpoints.json against https://<EVENT_FIREBASE_REGION>-<EVENT_FIREBASE_PROJECT_ID>.cloudfunctions.net and GETs EVENT_PUBLIC_URL.

• Between hosting and smoke, the post job redeploys updatesMeta and routeMeta (functions/src/public/og.cjs) alone. Both self-fetch the deployed hosting index.html as their SSR template and hold it per container (issue #27; M7 issue 4). That self-fetch carries its own three-second deadline (TEMPLATE_FETCH_TIMEOUT_MS), so a hosting connection that stalls rather than fails costs a reader three seconds and the held copy, not the function's whole timeout. A container that cold-started before THIS run's hosting deploy is holding the previous build's asset references, and routeMeta is the catch-all rewrite, so for it that stale copy is not only a bad crawl: it is the first HTML every visitor gets, naming Vite-hashed asset files that this deploy has already replaced. The held copy is revalidated against hosting's ETag every ten seconds (TEMPLATE_REVALIDATE_FLOOR_MS), so it corrects itself quickly on its own; redeploying here makes the changeover immediate instead. Anyone who deploys hosting by hand, outside this workflow, gets the ten-second correction and can redeploy these two functions to skip it. This was the ADR's post step (§8.1), deferred at the M2 deploy PR pending functions/src/public/og.cjs landing — smoke's OPTIONS-preflight of both functions now runs against the freshly-redeployed instances, not the ones from the functions job earlier in the same run.

• What routeMeta costs, and the cache header that bounds it. Every public route that is not a static file now reaches a function on its first uncached load, rather than being rewritten straight to index.html. Three things bound that. The response carries Cache-Control: public, max-age=0, s-maxage=300, must-revalidate, so the Hosting CDN — not the function — answers repeat traffic for the same URL, while max-age=0, must-revalidate keeps a reader's own browser from holding, or falling back to, a shell that names asset files a later deploy removed: max-age=0 alone still lets a cache serve a stale copy under pressure. There is deliberately no stale-while-revalidate for the same reason. The template is the revalidated per-container copy above, so a warm container transfers no body between deploys. And the config read is core/config.cjs's shared 5-minute container cache, which s-maxage matches: a publish reaches search and social previews within that window instead of at the next cold start, and a hosting deploy clears the CDN. If a client reports a page preview that will not update, five minutes is the number to wait before looking for a real fault. The one cost the CDN does not absorb is scanner traffic: requests for addresses that do not exist (/wp-login.php and its friends) each miss the cache and reach the function, where they cost one Firestore page lookup and return the plain shell. That is a line on the functions invocation graph. routeMeta sets maxInstances to 20 so that traffic stays bounded. The rewrite stays, because this function is the front door. App Check stays off this function. A crawler and a first visit have no App Check token.

• The card image a deployment falls back to is a PNG, not the SVG placeholder. The Open Graph and Twitter crawlers fetch the image themselves and do not reliably decode SVG, so a card built on branding/og-default.svg unfurls with an empty frame, which a reader takes for a broken link rather than a plain one. routeMeta therefore skips any SVG candidate, wherever it comes from (the config/theme.logos.ogDefault slot or config/event.seo.defaultOgImagePath, whose seeded value is that SVG), and points at branding/og-default.png, rendered by scripts/dev/build-og-placeholder.mjs and shipped in the web build. An operator who uploads a PNG or a JPEG through the Branding tab gets theirs instead. The tags state the image type always and the size only for the bundled card, which is the only one anything here has measured. The SVG stays in the bundle: the app renders it in-page, where SVG is fine.

• routeMeta does not answer 5xx once it has ever fetched a template. The site's whole front door is this function, so each read it makes degrades on its own: a hosting request that fails or answers non-2xx serves the held template, and a config read that fails — or an unset EVENT_PUBLIC_URL — serves that template with the event-level tags and no noindex, because a read that failed is not a page that does not exist. Only a container that has never fetched a template at all answers 500. A run of 500s from this function therefore means the very first self-fetch is failing: check EVENT_PUBLIC_URL and that hosting is actually serving /index.html.

• firebase.json's rewrite order is load-bearing. /updates/** goes to updatesMeta first; then the account and staff routes (/signin, /profile, /attendees, /attendees/**, /schedule/mine, /speaker/**, /ticket/**, /admin, /admin/**) are rewritten straight to /index.html, because they carry nothing a crawler should index and spending a function invocation to say so is waste; then the ** catch-all goes to routeMeta. Adding a public route needs nothing here — the catch-all already covers it. Adding an authenticated one means adding it to that middle block, above the catch-all, and to the headers block below it. Hosting serves an existing static file before it applies any rewrite, which is what keeps /assets/**, /branding/**, and routeMeta's own self-fetch of /index.html from re-entering the function. scripts/deploy-workflow.test.cjs asserts the whole arrangement, so a rewrite added out of order fails the suite rather than a deploy.

• Those same private sources appear again under hosting.headers, setting X-Robots-Tag: noindex. The shell they serve is one file for every route, so it cannot carry a per-route robots meta the way a routeMeta response can; hosting states it in the response header instead, which crawlers read the same way. Without it, the one part of the site with no server-set tags would be the part that most needs them.

• Both hosting rewrites name their function's Cloud Functions region in object form ("function": { "functionId": "...", "region": "..." }) — the shorthand string form ("function": "routeMeta") silently defaults to us-central1, which would route every non-default-region client's public requests at a backend that does not exist there (a 404 from Hosting, not from the function). Region is per-client (EVENT_FIREBASE_REGION), while firebase.json is one file shared by every client's deploy — the same shape of problem EVENT_HOSTING_SITE is (target names, not literal site ids), and the same fix: the hosting job's "Set the function rewrite regions for this project" step patches the working copy of firebase.json with jq immediately before firebase deploy --only hosting:site reads it, defaulting to us-central1 when EVENT_FIREBASE_REGION is unset (matching every other region default in this pipeline). It patches every object-form rewrite, so a rewrite added later cannot be left behind on the placeholder. Nothing is committed back — the committed file keeps us-central1 as its placeholder, correct for that value and every client that doesn't override the region. The site publisher applies the same patch in code (patchHostingRegion, scripts/publish-site.cjs).

• The staff end-time job exists. After the first deploy that includes endExpiredStaffAccess, the same listing shows firebase-schedule-endExpiredStaffAccess-<EVENT_FIREBASE_REGION>, set to every 1 minutes, and it must be ENABLED. A paused job does not reopen an ended grant: the server and the rules still refuse it on every request. It only means a browser tab opened before the end time can keep reading until it reconnects.

• The reminder scheduler job exists. After the first deploy that includes runScheduledReminders, list the Cloud Scheduler jobs and find firebase-schedule-runScheduledReminders-<EVENT_FIREBASE_REGION>, set to every 60 minutes:

  gcloud scheduler jobs list --project=<GCP_PROJECT_ID> --location=<EVENT_FIREBASE_REGION> \
              --format="table(name,schedule,state)"
            

  The job must be ENABLED. A paused or missing job means no scheduled reminder goes out, and the admin's Scheduled reminders page then shows a last run older than an hour while a schedule is on. The job runs every hour on every deployment; a run with no schedule turned on reads one document and returns.

• If google-github-actions/auth fails with permission_denied on a run you expected to succeed (dispatched from main), the attribute condition (§1) or the repository+ref binding (§2) is the first thing to re-check — copy the exact repository and ref claims GitHub sent from the failed run's log against what the binding names.

Archiving an event with ticket sales#

Pause new sales with the catalog's salesPaused setting while payment, held-capacity, refund, dispute, or reconciliation work remains. Keep the event writable until those obligations are settled and the existing ticket_sync_queue has no unresolved work. Then enable config/event.historicalDemo to freeze the archive.

The archive flag stops native payment reconciliation, webhook queuing, and ticket synchronization. These paths read the persisted policy before provider work and include it in transactions that update orders, payment attempts, inventory, tickets, delivery claims, and queued work. Freezing with an open payment or hold can therefore leave its last local state unchanged even when Stripe later reports a payment or refund.

If an event was frozen before its obligations were settled, an authorized operator must restore its writable policy and reconcile the existing orders before freezing it again. Keep the original provider, account, payment attempts, and inventory records throughout recovery; deleting a hold or making a replacement order does not establish the original payment's outcome.

Importing users or data with the Firebase CLI#

The pinned firebase-tools 15.22.4 reads firebase auth:import and firebase database:import files through stream-json 1.9.1. That version has open advisories: crafted JSON can stall the CLI or change the prototype of a parsed object. There is no fix we can take until the CLI moves past 15.22.4 (see scripts/ci/audit-exceptions.json).

• Only import a file you made yourself with firebase auth:export (or database:get) from one of this operator's own projects, and that has stayed on your machine or in your storage since.
• Never import a file from a client, a vendor, an email attachment or any other third party. To bring in users from outside, turn their data into a new export file with your own tools first, and check it before you import it.

7. Rotating access#

There is no key to rotate. To revoke a client's deploy access, remove the IAM policy binding from §2 (gcloud iam service-accounts remove-iam-policy-binding) or delete the GitHub Environment; both take effect on the next run, immediately.

8. Rolling back a bad deploy#

deploy-client.yml deploys hosting, functions, rules/indexes, and content on every run — a bad deploy is rarely just one of those. Roll back the piece that's actually broken; rolling back everything when only one component regressed reintroduces whatever the other components' good deploys fixed.

Hosting#

Firebase Hosting keeps prior releases. Fastest path is the Firebase Console — Hosting → the site → release history → Rollback on the last-known-good release; it's a Console action against EVENT_HOSTING_SITE, not a re-run of CI. From the CLI, with the same credentials an operator uses for one-off scripts (not the deploy service account — this is a manual, watched action):

npx firebase hosting:clone <SITE_ID>@<GOOD_VERSION_ID> <SITE_ID>:live --project <GCP_PROJECT_ID>
          

Find <GOOD_VERSION_ID> in the Console's release history. The Firebase CLI has no command that lists releases, and hosting:clone takes a version as <SITE_ID>@<VERSION_ID>, not as a channel. This affects the served static bundle only — it does not touch functions, rules, or content.

Functions#

There is no one-click functions rollback the way Hosting has one; the fix is to redeploy the prior good ref. Dispatch deploy.yml (workflow_dispatch, client: <CLIENT_ENV>) from main at the last-known-good commit — either revert forward on main and dispatch normally, or, if you need the exact prior artifact right now, git checkout <good-sha> and dispatch the deploy workflow with that ref (gh workflow run deploy.yml --ref <good-sha> -f client=<CLIENT_ENV>). Recall §1–2: the WIF binding only authorizes refs/heads/main, so a dispatch against an arbitrary SHA or tag that isn't reachable from main's current HEAD, or isn't itself on refs/heads/main, fails at the token-exchange step — this is a control, not a bug, and it means the practical rollback path is "revert on main," not "dispatch an old tag directly." Read §5's step ordering again if this is a fresh-enough project that bootstrap semantics could matter.

Firestore rules and indexes#

provision deploys firestore:rules, firestore:indexes, and storage on every run (§5) — a rules regression rolls back the same way functions do: revert on main, redeploy. Two things that don't roll back cleanly:

• A tightened rule that already blocked a write nobody needed rolls back fine — nothing was lost by the tightening.
• An index removed or narrowed takes real time to rebuild once restored — Firestore composite index builds are not instant, and any query depending on the removed index fails (not degrades) until the rebuild finishes. Budget for that gap; it is not the same shape of "instant" as a Hosting rollback.

Content snapshot implications#

content (generate-content.cjs) regenerates apps/web/src/generated/* from live Firestore at deploy time — rolling back hosting alone serves the previous build's generated snapshot, which was correct for the config/content at that previous deploy's time, not for whatever's in Firestore now. If a client has edited and published content since the bad deploy, a hosting-only rollback can visibly regress their content even though nothing about their content itself broke. When content has moved since the deploy you're rolling back from, prefer rolling forward (fix and redeploy) over rolling back, or expect to re-publish the affected pages after the rollback.

If this client has the site publisher enabled (§9), a hosting-only rollback does not hold. The next CMS publish invokes the site-publisher Cloud Run job, which rebuilds from its own image — the one the last deploy pushed, i.e. the code you just rolled hosting back from — and redeploys hosting over the rollback. So "re-publish the affected pages after the rollback", above, is the one thing that will silently undo it. To hold a hosting rollback on a publisher-enabled client, either set EVENT_SITE_PUBLISHER_ENABLED=false and deploy before rolling back, or roll the code back properly (revert on main, deploy) so the rebuilt image carries the good code. §9.3 has the detail.

When NOT to roll back#

Schema-forward migrations. A deploy that changed the shape of a config/* document, added a required field a running function now depends on, or changed what a Firestore trigger writes is not safely undone by redeploying old function code against the new data shape (or vice versa) — the two can disagree about what a document should look like, and that disagreement is a worse failure mode than the original bug. If the deploy you want to roll back included a migration, roll forward with a fix instead: write the small forward-fixing change, dispatch it, and treat the rollback tooling above as being for deploys that were behaviorally wrong, not ones that changed a data contract other code now depends on.

Custom domain and readiness notes#

A custom-domain rollback (attaching a different Hosting site, or reverting a EVENT_PUBLIC_URL/authorized-domains change) does not take effect immediately even after the Hosting/Auth Console steps are redone — DNS propagation and certificate issuance/reissuance can take from minutes to about a day (§0–3, and docs/CLIENT_ONBOARDING.md §3 item 3). Re-run node scripts/init-event.cjs --check after a domain change of any kind rather than assuming the readiness table is still accurate; the sender-domain and Auth rows in particular are operator-attested and do not re-verify themselves on a timer. See issue #66 for the fuller per-client-subdomain (client.eventrunner.org) provisioning flow this repo will eventually build on top of this same Hosting custom-domain mechanism — the wildcard-vs-per-client CNAME strategy is out of scope here and tracked there, not in this runbook. The Cloud Run site-publisher has since landed (#36) and is documented in §9; it is orthogonal to per-client subdomains, reaching the job as ordinary per-client environment with no domain coupling, so nothing in §9 waits on #66.

Custom web address from the admin#

An operator can add a custom web address on Settings → Web address (ADR 0007 §8). The webAddress function asks Firebase Hosting for the domain, shows the client the records to add, and checks every hour (webAddressCheck). When the domain is live it adds the domain to the Auth authorized domains. The functions runtime account needs two roles on the client project for this, once:

gcloud projects add-iam-policy-binding <GCP_PROJECT_ID> \
            --member="serviceAccount:<FUNCTIONS_RUNTIME_ACCOUNT>" --role="roles/firebasehosting.admin"
          gcloud projects add-iam-policy-binding <GCP_PROJECT_ID> \
            --member="serviceAccount:<FUNCTIONS_RUNTIME_ACCOUNT>" --role="roles/firebaseauth.admin"
          

The runtime account is EVENT_FUNCTIONS_SERVICE_ACCOUNT when set, else the project's default compute service account. Without the roles the page answers "did not answer, try again" and the function log names the refusal.

When the page says Ready to switch, finish the change by deploy, because the public address is a deploy-time value:

1. Set EVENT_PUBLIC_URL to https://<domain> in the client's GitHub Environment (§3), and add the same origin to EVENT_ALLOWED_ORIGINS.
2. Deploy from main.
3. Run node scripts/init-event.cjs --check. The page then shows In use.

The console's provisioner (ADR 0007, P8) will run this deploy for the client.

9. Site publisher (optional, per client)#

What it does. The frontend reads CMS content from Firestore at runtime, so a publish is already live for human visitors the moment cmsPublish commits (spec §2.4, §8.4). What lags is the static snapshot in the deployed bundle — the first paint, and the only thing a crawler that does not run JavaScript ever sees. The site-publisher Cloud Run job closes that gap on demand: after a successful publish, cmsPublish starts one execution, which runs generate-content.cjs against this project's published collections, rebuilds the bundle with this client's VITE_* values, and runs firebase deploy --only hosting. Entirely inside the client's own project, under the job's own service account — no GitHub token, no repository_dispatch, no cross-project credential (spec §8.4 phase 5; the container is publisher/Dockerfile, the logic is scripts/publish-site.cjs).

Sitemap, robots, and manifest do not wait for this job. scripts/publish-site.cjs writes sitemap.xml, robots.txt, and the web manifest from a live Firestore read, but this job runs only when a CMS publish triggers it — a fresh deployment, and every ordinary deploy-client.yml run, never runs it at all. That workflow's build job instead runs scripts/write-site-files.cjs right after npm run build, from the SAME generated snapshot the build just used (never the committed demo copy), through the same builders in scripts/lib/site-manifest.cjs — so a client's site never goes without these files between its first deploy and its first content publish, whether or not the site publisher is enabled at all.

App icons. Both paths also write two raster app icons beside the manifest, branding/app-icon-192.png and branding/app-icon-512.png (#218). When the square icon slot (config/theme.logos.mark) names an uploaded square PNG, the job reads it from EVENT_STORAGE_BUCKET over its public download URL and resamples it. branding/ is public read in storage.rules, so neither the publisher nor the build job needs an IAM change or a credential for this. Any other slot value ships the neutral placeholder icons, and the job log names the reason on a line that starts app icons: neutral placeholder:. An icon problem never fails the job.

It is optional. With EVENT_SITE_PUBLISHER_ENABLED unset or false, nothing below exists, the publisher deploy job is skipped, EVENT_SITE_PUBLISHER_JOB is never written into the functions env, and cmsPublish skips the invoke without writing anything. That is the phase 2–4 behavior: publishes work, and the crawler snapshot refreshes at the next deploy.

9.1 One-time setup per client#

Enable the two APIs the job needs (run.googleapis.com is already in §2's list):

gcloud services enable \
            run.googleapis.com \
            artifactregistry.googleapis.com \
            --project=<GCP_PROJECT_ID>
          

Create the job's runtime service account. This is a different identity from the deploy service account (§2) and from the functions runtime account (§3). It is the account the job runs as, and its grants are the entire blast radius of a compromised publisher image:

gcloud iam service-accounts create site-publisher \
            --project=<GCP_PROJECT_ID> \
            --display-name="Event Runner site publisher (<CLIENT_ENV>)"

          PUBLISHER_SA_EMAIL="site-publisher@<GCP_PROJECT_ID>.iam.gserviceaccount.com"

          for role in \
            roles/datastore.user \
            roles/firebasehosting.admin \
            roles/storage.objectViewer \
            roles/serviceusage.serviceUsageConsumer
          do
            gcloud projects add-iam-policy-binding <GCP_PROJECT_ID> \
              --member="serviceAccount:${PUBLISHER_SA_EMAIL}" \
              --role="${role}" \
              --condition=None
          done
          

Why exactly those four, and nothing more:

• datastore.user — generate-content.cjs reads config/* and the published CMS collections, and publish-site.cjs writes its terminal status back to the cmsPublishQueue row. Read-only (datastore.viewer) is not enough for that status write.
• firebasehosting.admin — the one thing the job exists to do. It cannot deploy functions, rules, or indexes: publish-site.cjs only ever runs firebase deploy --only hosting:site, and no role here would let it do otherwise.
• storage.objectViewer — branding assets under EVENT_STORAGE_BUCKET that the snapshot references. Viewer, not admin: the job never writes to Storage.
• serviceUsageConsumer — required for the Firebase CLI's API calls to bill against this project.

Notably absent: no run.admin (the job does not manage itself), no artifactregistry role (it does not push images; the deploy service account does that), no iam.serviceAccountUser, and no secretmanager.secretAccessor — the job needs no secrets.

The deploy service account (§2) already carries artifactregistry.writer and run.developer, which is what the publisher job needs to push the image and create the Cloud Run job. It also needs iam.serviceAccountUser on the publisher account in order to deploy a job that runs as it — §2's project-level iam.serviceAccountUser covers this, but the narrower binding is better:

gcloud iam service-accounts add-iam-policy-binding "${PUBLISHER_SA_EMAIL}" \
            --project=<GCP_PROJECT_ID> \
            --member="serviceAccount:${DEPLOY_SA_EMAIL}" \
            --role="roles/iam.serviceAccountUser"
          

Then set the two GitHub Environment variables from §3:

validateDeployEnv fails the deploy if the flag is true and the account is unset — there is deliberately no default, because a blank would hand the job whatever identity Cloud Run picks.

The next deploy from main provisions everything else automatically: the run-of-show Artifact Registry repository in this project, the image (tagged with the commit SHA), the site-publisher Cloud Run job with this client's environment, and roles/run.invoker for the functions runtime account on that one job. Nothing is granted project-wide.

Ordering on the first enable. The publisher job and the functions job run independently in the same workflow run, so for a short window the deployed cmsPublish may know the job's name before the job exists. A publish in that window still succeeds — the invoke fails soft, records publisher.invoke.status: 'failed' on the queue row, and raises an operator alert. Re-publish after the run completes.

9.2 Verifying it#

# The job exists, runs as the right identity, and carries this client's env.
          gcloud run jobs describe site-publisher \
            --region=<EVENT_FIREBASE_REGION> --project=<GCP_PROJECT_ID> \
            --format="yaml(spec.template.template.serviceAccountName, spec.template.template.containers[0].image)"

          # run.invoker is bound on THE JOB, to the functions runtime account only.
          gcloud run jobs get-iam-policy site-publisher \
            --region=<EVENT_FIREBASE_REGION> --project=<GCP_PROJECT_ID> --format="json(bindings)"

          # End to end, without going through the CMS: start one execution by hand.
          gcloud run jobs execute site-publisher --wait \
            --region=<EVENT_FIREBASE_REGION> --project=<GCP_PROJECT_ID>
          

A hand-started execution takes no PUBLISH_QUEUE_ID, so it publishes the site and writes no status row — which makes it the safe way to test the container itself.

Then the real path: make a trivial CMS edit, publish it, and check that cmsGetPublishQueue shows the row with publisher.invoke.status: 'invoked' followed by publisher.status: 'done', and that the deployed HTML source (not the rendered page — curl -s <EVENT_PUBLIC_URL> | grep) carries the new content.

The exit code names the stage, so triage rarely needs the log:

Common failures on the invoke side, from the queue row's publisher.invoke.error:

• Cloud Run responded 403 — the run.invoker grant is missing or was applied to the wrong identity. Re-check the runtime account against §3's RUNTIME_SA_EMAIL.
• Cloud Run responded 404 — the job does not exist in that region. Check EVENT_FIREBASE_REGION matches where the deploy created it.

9.3 Interaction with rollback#

§8 is the rollback procedure; this is only what changes once a client has a publisher. The publisher deploys hosting from whatever commit built its image, not from the currently deployed bundle, which turns §8's "Hosting" rollback from a durable action into a temporary one. Two consequences worth knowing before an incident:

• Rolling back hosting alone does not stick. A Firebase Hosting rollback in the console is undone by the next CMS publish, because the job rebuilds from its image and redeploys. To hold a rollback, either disable the publisher (EVENT_SITE_PUBLISHER_ENABLED=false, then deploy) or roll the code back properly — revert on main and let the deploy rebuild the image.
• Rolling back the code is a deploy. Reverting on main rebuilds and repushes the image under the new commit SHA and updates the job, so the next publish uses the rolled-back code. The old image stays in Artifact Registry under its own SHA tag; nothing is overwritten.

9.3.1 The publisher does not redeploy the SSR meta functions#

deploy-client.yml's post job redeploys updatesMeta and routeMeta after every hosting deploy, for the template reason in §6. The publisher job does not, and cannot — its service account holds firebasehosting.admin and nothing else, on purpose (§9.1). So a CMS publish changes the hosting release without restarting a single function container.

That is survivable rather than an outage because the held template is revalidated against hosting's ETag rather than trusted for a fixed span: past TEMPLATE_REVALIDATE_FLOOR_MS (ten seconds, functions/src/public/og.cjs) the next request revalidates, the changed build answers 200 with the new body, and the container is current from then on. The exposure is that ten-second floor, not the five minutes a flat cache would have cost.

If a client does report a blank page straight after a publish, that window is the first thing to rule out — wait ten seconds and reload. Redeploying the two functions (firebase deploy --only functions:updatesMeta,functions:routeMeta) clears it immediately.

Stranded rows are not an incident. If neither cmsPublish nor the job reports a result, the cleanupStrandedPublishRows sweep (functions/src/maintenance/cleanup.cjs, every 30 minutes) marks the row failed after 90 minutes and alerts once — a failed publish row is resumable from the CMS, and a failed publisher row just means the snapshot is stale until the next publish or deploy.

9.4 Cost#

The job runs on demand — once per CMS publish, plus whatever an operator starts by hand. There is no idle cost: a Cloud Run job bills only while an execution is running, and it has no minimum instances and no request-serving footprint between executions. One execution is a few minutes of 2 vCPU / 4 GiB, so a normal editing day costs cents. What accumulates instead is Artifact Registry storage, one image per deployed commit — prune old tags if a client's project has years of them:

gcloud artifacts docker images list \
            "<EVENT_FIREBASE_REGION>-docker.pkg.dev/<GCP_PROJECT_ID>/run-of-show/site-publisher" \
            --project=<GCP_PROJECT_ID>
          

The alternative — putting the snapshot refresh on a nightly schedule — is what phases 2–4 specified and costs a build a day whether or not anything changed (spec §8.4, §10 Q7). Nothing in this repo implements it: the refresh went straight from "next deploy" to this on-demand job.

10. Project site on GitHub Pages#

This section is not per client. It covers the project's own site: the homepage, the support, privacy and terms pages, the project status page, the documentation under docs/docs/, and the static demo under docs/demo/. .github/workflows/pages.yml builds the site and deploys it on every push to main.

One value sets the address. scripts/lib/pages-site.cjs holds the public address of the site: an https origin and a path base. scripts/build-pages.cjs, scripts/build-demo.cjs, and scripts/check-docs.cjs read it. It sets the canonical and social card URLs, every root-relative link, the demo's Vite base, and how the link checker maps a path to a file under docs/.

The committed docs/docs/ and the hand-written pages (LANDING_PAGES in scripts/build-pages.cjs) are built for the committed default. When the PAGES_BASE_URL repository variable is set, the workflow renders the pages again for that address before it builds the demo and checks the links. The workflow commits nothing.

The workflow does not deploy to a wrong address. Before it deploys, the workflow compares the address with the URL that Pages reports. If they are not the same, the run fails and the site that is live now stays live. The message names both addresses.

The address and the domain change together. With a base of /, every link points at the domain root. While the site is served at the github.io project path, those links break. A site built for /eventrunner/ breaks in the same way when it is served from the custom domain. Do the steps below in this order.

10.1 Switch the site to eventrunner.org#

1. DNS. For eventrunner.org, add A records to 185.199.108.153, 185.199.109.153, 185.199.110.153 and 185.199.111.153, AAAA records to 2606:50c0:8000::153, 2606:50c0:8001::153, 2606:50c0:8002::153 and 2606:50c0:8003::153, and CNAME www → centercoopmedia.github.io. On Cloudflare, set each record to DNS only (grey cloud). If a record is proxied, GitHub cannot issue the certificate. Leave the MX and TXT records alone: they carry the domain's email. Wait until dig +short eventrunner.org A returns the four addresses.
2. Optional: verify the domain. In the organization settings, go to Pages → Verified domains and verify eventrunner.org. This prevents a domain takeover.
3. Variables. Go to Settings → Secrets and variables → Actions → Variables. Add the repository variables PAGES_SUPPORT_NAME and PAGES_SUPPORT_EMAIL, which fill in who clients reach on the homepage and the support, privacy and terms pages, and PAGES_BASE_URL with the value https://eventrunner.org/. Until step 4, a Pages run fails at the address check and deploys nothing. The live site does not change.
4. Custom domain. Go to Settings → Pages → Custom domain. Enter eventrunner.org and save. With a workflow-deployed site, GitHub ignores a CNAME file in the artifact, so this setting is the only place for the domain. From now until step 5 ends, the live site has broken links, so do step 5 immediately. The DNS check can finish while step 5 runs.
5. Deploy. Go to Actions → Pages → Run workflow, on main. The run renders the site for the new address, checks every link, compares the address with the Pages URL, and deploys.
6. HTTPS. When the DNS check passes and the certificate is issued, turn on "Enforce HTTPS" in Settings → Pages.

To check the result:

curl -sI https://eventrunner.org/ | head -1           # 200
          curl -sIL https://eventrunner.org/support | grep '^HTTP' | tail -1            # 200
          curl -sIL https://eventrunner.org/privacy | grep '^HTTP' | tail -1            # 200
          curl -sIL https://eventrunner.org/terms-of-service | grep '^HTTP' | tail -1   # 200
          curl -sI https://eventrunner.org/demo/ | head -1      # 200
          curl -sI https://eventrunner.org/docs/ | head -1      # 200
          curl -sI https://centercoopmedia.github.io/eventrunner/ | grep -i '^location'   # the new host
          curl -s https://eventrunner.org/ | grep canonical     # https://eventrunner.org/
          

Open a documentation page and the demo, and make sure that the styles, fonts, favicon, and links load from the new host.

Later, in a pull request. When the domain works, make the custom domain the committed default:

1. In scripts/lib/pages-site.cjs, set DEFAULT_PAGES_BASE_URL = 'https://eventrunner.org/'. This is the one line to change.
2. Run node scripts/build-pages.cjs. It renders docs/docs/ again and moves the landing pages to the new address. Commit the output.
3. Update the tests that name the default address, and the links to the github.io address in README.md, docs/handbook/, RELEASING.md, and scripts/README.md. The links still work before this step, because GitHub redirects the github.io address to the custom domain, and the generated pages already send a reader to the configured address.
4. After that pull request is merged, you can delete the PAGES_BASE_URL variable.

To go back to the github.io address, delete the custom domain in Settings → Pages. Then delete the PAGES_BASE_URL variable, or set it to https://centercoopmedia.github.io/eventrunner/. Run the Pages workflow again.

To see what an address changes, without writing a file:

PAGES_BASE_URL=https://eventrunner.org/ node scripts/build-pages.cjs --check   # shows what would change
          node scripts/lib/pages-site.cjs                                                    # prints the address in use
          

11. Client console#

The client console (ADR 0007) is one CCM-owned Firebase project at console.eventrunner.org. It is not a client deployment, so sections 0 to 9 do not apply to it. Its code is in console/, and console/README.md describes it. This section is the one-time setup and a manual deploy. The full runbook, with a deploy workflow, comes later.

11.1 One-time setup#

1. Create the console Firebase project on the Blaze plan. Turn on Firestore (Native mode) and Authentication.

2. In Authentication, turn on these sign-in methods:

   • Google.
   • Email/Password with Email link (passwordless sign-in).
   • Microsoft. This needs an app registration in Microsoft Entra ID (Azure portal, App registrations). Allow accounts in any organizational directory and personal Microsoft accounts. Add the redirect URI that the Firebase Microsoft provider shows (https://<auth domain>/__/auth/handler). Make a client secret, then put the application (client) ID and the secret in the Firebase Microsoft provider. Write down when the secret expires.

3. In Authentication settings, add console.eventrunner.org to the authorized domains.

4. In Hosting, add the custom domain console.eventrunner.org.

5. In Postmark, make a server for the console and verify its sender domain. Store the server token in the console project:

   cd console
             npx firebase functions:secrets:set CONSOLE_POSTMARK_TOKEN --project <console-project-id>
             

6. Write console/functions/.env (Git ignores it):

   CONSOLE_PUBLIC_URL=https://console.eventrunner.org
             CONSOLE_MAIL_FROM=Event Runner <hello@console.eventrunner.org>
             

7. In the Firestore data viewer, make the document console_config/operators with an array field emails. Add each CCM operator's address in lowercase.

8. Set up account connections (ADR 0007 §4). The two secrets must exist before any deploy, because a function that names a secret will not deploy without it. Until a provider is ready, store a placeholder and leave its client id out of .env: the tickets step then says the choice is not available yet.

   cd console
             npx firebase functions:secrets:set CONSOLE_STRIPE_SECRET_KEY --project <console-project-id>
             npx firebase functions:secrets:set CONSOLE_EVENTBRITE_CLIENT_SECRET --project <console-project-id>
             

   • Stripe: in the platform account, turn on Connect and OAuth for Standard accounts. Add the redirect URI https://console.eventrunner.org/connect/stripe/return. Put the Connect client id in console/functions/.env as CONSOLE_STRIPE_CLIENT_ID. Use the test mode client id and key for a rehearsal, with CONSOLE_TEST_MODE set (§11.2).
     • A new account's live client id stays unavailable until the sandbox setup is done, payments are activated, and the Connect integration questions are answered. Do them in that order.
     • The console's secret key only exchanges and revokes connection tokens, so make it a restricted key with one permission: Connect OAuth: Write. Store it in Secret Manager, then redeploy, because a deploy pins each secret's version.
     • Creating a live key emails a verification link to the Stripe login address. Give that address a working mail route first. If mail to it bounced earlier, Stripe can stop sending: Personal details then shows "We've had trouble sending you emails," and the key pop-up keeps saying a link was sent.
   • Eventbrite: make an app, and set its OAuth redirect URI to https://console.eventrunner.org/connect/eventbrite/return. Put its API key in console/functions/.env as CONSOLE_EVENTBRITE_CLIENT_ID, and its client secret in the secret above.
   • Secret Manager: turn on the Secret Manager API in the console project. Give the functions' runtime service account the Secret Manager Admin role, with an IAM condition on the secret name prefix projects/<project-number>/secrets/connection-, so it cannot change other secrets. Without a condition, the role reaches every secret in the project.

11.2 Deploy#

Dispatch .github/workflows/deploy-console.yml from main. It runs in the GitHub Environment console, which is limited to main, and signs in with Workload Identity Federation as the console's deploy service account. The provider's condition names this workflow and that Environment, so no other workflow can use it.

A deploy does not delete a function that left the code. In --non-interactive mode it stops, lists each one and prints its firebase functions:delete command. Run those commands and deploy again. A removed public function keeps answering at its direct address until it is deleted.

Secrets stay in the console project's Secret Manager, and the Environment holds none: CONSOLE_POSTMARK_TOKEN, CONSOLE_STRIPE_SECRET_KEY, CONSOLE_EVENTBRITE_CLIENT_SECRET, CONSOLE_GITHUB_APP_PRIVATE_KEY. A function that names a secret will not deploy without it, so until a provider is ready store a placeholder. The functions' runtime account needs read access to each one.

Set these up as the project owner before the first deploy, because the deploy account cannot:

• These APIs on the console project: cloudfunctions, cloudbuild, run, eventarc, pubsub, cloudscheduler, artifactregistry, secretmanager, firestore, firebasehosting and identitytoolkit.

• For the functions' runtime account, read access on each of the four secrets above. A project-level role doesn't satisfy the deploy's check, and the deploy account can't grant this, so without it the deploy stops. Run this once for each secret:

  gcloud secrets add-iam-policy-binding <secret-name> --project <console-project-id> \
              --member serviceAccount:<project-number>-compute@developer.gserviceaccount.com \
              --role roles/secretmanager.secretAccessor
            

• The Artifact Registry repository gcf-artifacts in us-central1, with the cleanup policy firebase-functions-cleanup (delete images older than 86400 seconds).

• For the Pub/Sub service agent, roles/iam.serviceAccountTokenCreator. For the functions' runtime account, roles/run.invoker, roles/eventarc.eventReceiver, roles/datastore.user, and roles/iam.serviceAccountTokenCreator on itself, so emailed sign-in can make tokens.

• For the deploy account, the role list in §2 without roles/storage.admin, plus roles/secretmanager.viewer, the custom role eventrunnerDeployInvoker, and roles/iam.serviceAccountUser on the compute and App Engine default accounts.

To deploy by hand instead, run from the repository root with the console project's public web settings:

npm run prepare:functions && npm ci
          VITE_CONSOLE_FIREBASE_API_KEY=... VITE_CONSOLE_FIREBASE_AUTH_DOMAIN=... \
          VITE_CONSOLE_FIREBASE_PROJECT_ID=<console-project-id> VITE_CONSOLE_FIREBASE_APP_ID=... \
            npm run build:console
          cd console
          npx firebase deploy --project <console-project-id>
          

console/firebase.json deploys the console rules, indexes, functions and web/dist only. Never run this with a client project id, and never deploy the console from the repository root: the root firebase.json is the client deployment.

Check after the first deploy:

• Sign in at https://console.eventrunner.org with each of the three methods.
• Create a signup link (§11.3) and open it from a test address. Confirm that the organization and a draft request exist, and that the same link from another address says it has been used.
• From the test organization, invite a second address and accept the invite from that address.
• Sign in as an operator and confirm that the home page lists every organization.
• With a provider set up, connect a test account from the tickets step, then disconnect it. On Eventbrite, confirm that coming back works at all: its documents do not say it returns the state value, and without it the console refuses the connection.
• Answer the guided setup up to the agreement, then withdraw the test request. Do not accept the agreement here: accepting starts a real build, which section 12 covers.

11.3 Creating a signup link#

A client starts from a signup link and from nowhere else. The event fee is invoiced outside the console (ADR 0007), and the console holds no fee, price or invoice data. Once the invoice is paid:

1. Sign in as an operator and open /review/links, which the review queue links to.
2. Enter the organization name. The client can change it when they use the link.
3. Copy the link and send it to the client. It is shown once and cannot be read back. If it is lost, revoke it and create another.

From there the client signs in, enters the details and accepts the agreement, and the build starts. No operator step follows.

A link works once and expires after 30 days. It is not tied to an email address, so a forwarded link works for the first person who uses it. The page lists each link with its status, who used it and when. Select Revoke on an open link that went to the wrong place. When a client says a link says it is used, expired or revoked, create a new one.

If creating a link says sign-ups are closed, the message names what is missing: the agreement, the client folder, the billing account or the region. Set them at /review/settings (§12.1 step 9). While one is missing, an open link also refuses, and so does accepting the agreement.

12. Site builder#

The site builder builds one client's site after an Owner accepts the agreement, with nobody's help (ADR 0007 §6). The console's startBuild dispatches .github/workflows/provision-client.yml, which runs scripts/provision-client.cjs as the provisioner service account. Sections 1 to 5 are the same work by hand, for a site made outside the console.

Each step reads the current state and does nothing when it is already right, so a failed run is repaired by running it again.

The order differs from ADR §6 in two places: the mail webhooks follow the first deploy, and the sender record and the ticket webhook's registration are written in seed, because they need config/event and the deployed functions.

12.1 One-time setup#

Use placeholders from your own records. Nothing below belongs in the repository.

1. Provisioner service account, in the console project.

   gcloud iam service-accounts create eventrunner-provisioner --project=<console-project-id> \
               --display-name="Event Runner site builder"
             PROVISIONER=eventrunner-provisioner@<console-project-id>.iam.gserviceaccount.com
   
             gcloud resource-manager folders add-iam-policy-binding <client-folder-number> \
               --member="serviceAccount:${PROVISIONER}" --role=roles/resourcemanager.projectCreator
             gcloud billing accounts add-iam-policy-binding <billing-account-id> \
               --member="serviceAccount:${PROVISIONER}" --role=roles/billing.user
             gcloud projects add-iam-policy-binding <console-project-id> \
               --member="serviceAccount:${PROVISIONER}" --role=roles/datastore.user --condition=None
             gcloud projects add-iam-policy-binding <console-project-id> \
               --member="serviceAccount:${PROVISIONER}" --role=roles/secretmanager.admin \
               --condition='title=connection tokens only,expression=resource.name.startsWith("projects/<console-project-number>/secrets/connection-")'
             

   A project's creator becomes its owner. That is what lets the provisioner do everything else inside a client project, so it holds no role on the folder beyond creating projects.

2. APIs on the console project. A call with no quota project bills the caller's own project: cloudresourcemanager, cloudbilling, serviceusage, iam, iamcredentials, sts, firebase, firestore, secretmanager, artifactregistry.

3. Identity for the workflow, in the console project. The condition names the provisioning Environment and this one workflow.

   gcloud iam workload-identity-pools create github-actions --project=<console-project-id> \
               --location=global --display-name="GitHub Actions"
             gcloud iam workload-identity-pools providers create-oidc provisioner --project=<console-project-id> \
               --location=global --workload-identity-pool=github-actions --display-name="Site builder" \
               --issuer-uri="https://token.actions.githubusercontent.com" \
               --attribute-mapping="google.subject=assertion.sub,attribute.repository_and_ref=assertion.repository + '#' + assertion.ref" \
               --attribute-condition="assertion.repository_owner_id == '<owner-id>' && assertion.repository_id == '<repository-id>' && assertion.ref == 'refs/heads/main' && assertion.environment == 'provisioning' && assertion.workflow_ref == '<owner>/<repo>/.github/workflows/provision-client.yml@refs/heads/main'"
             gcloud iam service-accounts add-iam-policy-binding "${PROVISIONER}" --project=<console-project-id> \
               --role=roles/iam.workloadIdentityUser \
               --member="principalSet://iam.googleapis.com/projects/<console-project-number>/locations/global/workloadIdentityPools/github-actions/attribute.repository_and_ref/<owner>/<repo>#refs/heads/main"
             

4. Two GitHub Apps, each installed on this repository alone.

   App	Repository permissions	Key lives in
   Console	Actions: write	Secret Manager in the console project, CONSOLE_GITHUB_APP_PRIVATE_KEY
   Provisioner	Administration: write, Environments: write, Actions: read	The provisioning Environment, PROVISIONER_APP_PRIVATE_KEY

   GitHub's permission list puts creating an Environment and its branch rule under Administration, variables under Environments, and reading the Environment under Actions. The provisioner writes no GitHub secrets, because deploy-client.yml reads none: a client's secrets live in its own Secret Manager. Administration is broad. The key exists only in the provisioning Environment, and each token the script asks for is cut down to these three on this one repository.

5. The console's settings. Set these as variables in the console Environment, which deploy-console.yml writes to console/functions/.env, and store the key. The secret must exist before any deploy. Until the App exists, store a placeholder: the console then refuses to start a build, and says so to operators.

   CONSOLE_GITHUB_APP_ID=<console app id>
             CONSOLE_GITHUB_INSTALLATION_ID=<console app installation id>
             CONSOLE_GITHUB_REPOSITORY=<owner>/<repo>
             

   cd console
             npx firebase functions:secrets:set CONSOLE_GITHUB_APP_PRIVATE_KEY --project <console-project-id>
             

6. Cloudflare. Make an API token with Zone: DNS: Edit on the eventrunner.org zone alone. The script refuses to write a name outside the site's own <name>.eventrunner.org.

7. Postmark. Use an account API token. It goes in the provisioning Environment only, never in a client project. Each client's server token goes to that client's Secret Manager.

8. The provisioning GitHub Environment. Allow main only. Add no required reviewers, because the point is that nobody has to be there.

   Kind	Name	Value
   Variable	WIF_PROVIDER	The provider from step 3
   Variable	WIF_SERVICE_ACCOUNT	The provisioner's address
   Variable	CONSOLE_PROJECT_ID	The console project
   Variable	CONSOLE_PUBLIC_URL	https://console.eventrunner.org
   Variable	CONSOLE_MAIL_FROM	The console's sender
   Variable	CLOUDFLARE_ZONE_ID	The eventrunner.org zone
   Variable	PROVISIONER_APP_ID, PROVISIONER_INSTALLATION_ID	The provisioner App
   Secret	PROVISIONER_APP_PRIVATE_KEY	The provisioner App's key
   Secret	CLOUDFLARE_API_TOKEN	Step 6
   Secret	POSTMARK_ACCOUNT_TOKEN	Step 7
   Secret	CONSOLE_POSTMARK_TOKEN	A copy of the console's mail token, for the emails the build sends

9. Settings. Set the client folder, billing account and region at /review/settings in the console, with the agreement. No signup link can be created until all four are set.

10. Check the organization's policies on the client folder. A new organization may set iam.automaticIamGrantsForDefaultServiceAccounts, so the compute account gets no Editor role, and iam.allowedPolicyMemberDomains, which refuses the allUsers binding public functions need. The rehearsed project did not have these. Allow what the deployed site needs on the client folder, or the deploy fails.

12.2 When a build fails#

The review page shows each step, the failed step, its error and a link to the run. Operators get an email with the same error. Fix the cause, then select Retry the build.

The Owners get an email saying the build did not finish, with a link to the status page. That page says the same and offers Retry the build, and Reconnect when the failed step is tickets. It never shows the error, the run or the step detail. An Owner can retry alone, up to a daily limit.

To close an accepted request that should not go on, such as a signup link that reached the wrong person, cancel its run if one is going, along with any Deploy <name> run in the Actions tab, because the console only knows about the build run. Wait for them to stop, then select Close this request on its review page. That ends the console's connections and deletes the tokens it holds. It frees the web address only if no build ever began. It removes nothing the build already made. A Stripe connection is revoked, but Eventbrite has no revoke call, so an Eventbrite token the build moved stays usable in the client project until you delete that project. Read the build record to see how far it got, then remove by hand what exists: the Google Cloud project, the DNS records for its address, the GitHub Environment, and the sender domain. Once the build has made a project, that name can't be built again through the console, even after the project is deleted, because Google never reuses a project id and the build refuses a project it didn't make. So retry a failed build rather than close it, unless the client is finished with that name.

To run a rehearsal with a test-mode connection, set CONSOLE_TEST_MODE to true in the console Environment and deploy the console. Acceptance then lets a test-mode account through, and the build it starts runs in test mode. Remove the variable and deploy again before a client uses the console: until then, it accepts test-mode accounts. A test-mode build still makes real, billed resources: a Google Cloud project, DNS records and a deployment. Only the ticket account is in test mode.

To run an accepted request again in test mode, dispatch it by hand:

gh workflow run provision-client.yml --ref main -f application_id=<id> -f test_mode=true
          

A run that dies before the script starts, for example on a refused identity token, leaves no record: the request shows a started build and nothing else. Read the workflow's run list.

A new client's Environment is not added to AUTO_DEPLOY_ENVIRONMENTS, so a push to main does not deploy to it. Add it by hand once the site is launched (section 4).

12.3 What the hand-built rehearsal taught#

The script does each of these, and sections 1 to 3 are wrong or silent on them. Anyone building a client by hand has to do them too.

• A provider conditioned on one environment cannot be shared, so each client project has its own pool (github-actions, provider github), in place of section 1's shared one.
• The deploy account also needs the custom role eventrunnerDeployInvoker, with cloudfunctions.functions.setIamPolicy, firebaseextensions.instances.list, firebasestorage.defaultBucket.get, resourcemanager.projects.getIamPolicy and run.services.setIamPolicy.
• The deploy account needs roles/iam.serviceAccountUser on both the compute and the appspot service accounts, granted on each account and not on the project.
• The storage service agent may not exist. Create it (gcloud beta services identity create --service=storage.googleapis.com), then grant it roles/pubsub.publisher. A new agent can take minutes to be accepted.
• The deploy account cannot make these grants, so an owner makes them first: the pubsub agent gets roles/iam.serviceAccountTokenCreator, and the compute account gets roles/run.invoker and roles/eventarc.eventReceiver. The compute account also gets roles/iam.serviceAccountTokenCreator on itself for createCustomToken. That last one is unconfirmed, and granted anyway.
• cloudbilling.googleapis.com must be on, and the deploy account cannot turn it on. gcloud services enable takes at most 20 at a time (SU_MAX_BATCH_SIZE_EXCEEDED).
• The Hosting custom domain API is v1beta1. Firebase REST calls need the x-goog-user-project header.
• The gcf-artifacts repository needs the cleanup policy firebase-functions-cleanup (olderThan: 86400s), set by an owner. The deploy account cannot set it, even with --force, and a run without --force then fails.
• A first functions deploy can hit quota 429s and Eventarc agent failures on a few Firestore triggers. Run it again.
• init-event.cjs exits 0 when branding uploads are skipped. Credentials with a quota_project_id from another project cause that ("User project specified in the request is invalid"), so clear the quota project and treat any skipped branding as a failure.

12.4 What only a live run can prove#

The tests run against a fake of each service, and the fake answers as the rehearsal saw it or as the documents say. Check these on the first real build, on a throwaway name and in test mode:

• Google Cloud: a missing project answers 403 to the provisioner and creating it succeeds. The project's creator holds owner with these credentials. identityPlatform:initializeAuth is the right way to turn on sign-in, and the config answers 400 or 404 before it. The custom domain's requiredDnsUpdates and cert.verification field and value shapes. The default bucket's answer holds its name. nam5 is accepted for us-central1.
• Workflow identity: google-github-actions/auth exports the project and quota variables the script clears from a child's environment. Its credentials keep working past an hour, for a run that waits on a certificate. Credentials from the auth action carry no quota_project_id of their own, and if they do, branding fails loudly.
• GitHub: GITHUB_TOKEN with actions: write dispatches deploy.yml and the run starts. display_title follows run-name. An App token with the three permissions above can create an Environment.
• Postmark: ApiTokens[0] on the account's /servers answer. /message-streams accepts a Broadcasts stream with Postmark-managed unsubscribe. Webhooks accept HttpAuth, pass Postmark's test of each trigger, and list the URL back in a form this comparison matches.
• Cloudflare: a TXT value comes back in the form it was stored. A pm-bounces CNAME and the certificate's records are not proxied.
• Stripe and Eventbrite: GET /v1/account answers with the connected account's id for a Connect OAuth token. Creating a webhook endpoint on that account with that token works. GET /users/me/ answers for a live Eventbrite token.

12.5 Watching a site and stepping in#

Nothing needs to be open during a build. Operators get an email when a build fails or goes silent, and the review page at /review/<application id> shows every step. These commands cover the rest. <name> is the client's Environment, which is also its web-address name.

Stopping a build also leaves any Deploy <name> run it started going, so cancel that too. A retry won't redeploy a code fix on its own, because an earlier successful deploy of the same kind still counts. Run the deploy yourself and wait for it, then retry. For a failure before the full deploy, use gh workflow run deploy.yml --ref main -f client=<name> -f provision=true -f bootstrap=true. For a failure after it, use the plain redeploy in the table.