Savvi Studio

Manual Testing Runbook: Parameter Store + Setup/Maintenance Behavior

This runbook validates:

  • Parameter Store config loads successfully when available.
  • Config source failure modes (strict, fallback, maintenance) behave as designed.
  • Setup-required maintenance mode is reachable and recoverable.

Use one terminal per scenario, stop pnpm dev between scenarios.

1) Baseline and Prerequisites

  1. Install dependencies and verify core commands:
corepack pnpm install --frozen-lockfile
pnpm studio --help
pnpm studio infra stack generate --check
  1. Confirm baseline startup without forced Parameter Store:
pnpm dev
  1. Configure .env for Parameter Store testing (SSO profile-based auth):
AWS_PROFILE=savvi-sandbox
STUDIO_PARAMETER_STORE_ENABLED=true
STUDIO_CONFIG_SOURCE_FAILURE_MODE=strict

Notes:

  • Use your real SSO profile name for AWS_PROFILE.
  • This runbook assumes profile-based SSO auth, not static AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY.

Expected:

  • App starts normally.
  • No unexpected maintenance redirect.

2) Parameter Store Success Path (Happy Path)

Goal: prove the app/CLI can load config from SSM with valid AWS credentials.

  1. Ensure your .env has a valid SSO AWS_PROFILE and run aws sso login --profile <your-profile> if needed.
  2. (Optional) create a known test parameter in your namespace:
aws ssm put-parameter \
	--name /savvi-studio/server/base-url \
	--type String \
	--value http://localhost:3000 \
	--overwrite
  1. Validate via CLI using Parameter Store provider:
pnpm studio config validate --provider parameter-store --region us-east-1 --verbose

Expected:

  • Command succeeds.
  • Output includes Configuration loaded successfully.
  • No degraded-source warnings.

Log markers that count as PASS for this section:

  • [config] Configuration loaded successfully
  • No Continuing after source failure (...) warnings
  • No startup crash before app is ready

Important scope note:

  • Errors during OAuth callback or user provisioning (for example in WorkOS login flow) are not part of Parameter Store load validation.
  • If config loaded successfully before those errors, Section 2 is still a PASS.

3) Failure Mode: Strict (Fail Fast)

Use intentionally invalid credentials to force SSM access failure.

AWS_PROFILE=invalid-profile \
pnpm dev

Expected:

  • Startup fails.
  • Errors indicate config source load failure.

4) Failure Mode: Fallback (Continue Startup)

AWS_PROFILE=invalid-profile \
STUDIO_CONFIG_SOURCE_FAILURE_MODE=fallback \
pnpm dev

Expected:

  • Startup succeeds.
  • Warning includes Continuing after source failure (fallback).
  • App remains usable without forced maintenance due to config source degradation.

5) Failure Mode: Maintenance (Continue + Degraded Maintenance)

AWS_PROFILE=invalid-profile \
STUDIO_CONFIG_SOURCE_FAILURE_MODE=maintenance \
STUDIO_FEATURE_MAINTENANCE_MODE_MIDDLEWARE_ENABLED=true \
pnpm dev

Expected:

  • Startup succeeds.
  • Logs show Parameter Store credential/load failures, then startup continues.
  • Maintenance behavior is enforced on requests (redirect/path behavior), not by startup crash.

Log markers to look for:

  • one or more lines similar to Failed to fetch parameter batch: Could not load credentials from any providers
  • startup continues (app process remains up)

Request-path checks (required for PASS):

  • Open /maintenance and confirm a maintenance page/reason is shown.
  • Open a normal route (for example /) and verify it is redirected/handled by maintenance behavior when applicable.

Notes:

  • Request redirection depends on feature flag feature.maintenanceModeMiddlewareEnabled.
  • Environment variable for this flag is STUDIO_FEATURE_MAINTENANCE_MODE_MIDDLEWARE_ENABLED.

6) Setup-Required Maintenance Mode Validation

Goal: verify initial setup flow (Initial Setup Required) and /maintenance/setup behavior.

  1. Ensure maintenance middleware is enabled for the run:
STUDIO_FEATURE_MAINTENANCE_MODE_MIDDLEWARE_ENABLED=true pnpm dev
  1. In a separate browser session, visit /maintenance and /maintenance/setup.
  2. Verify setup status is incomplete (fresh DB or incomplete setup state).

Expected:

  • When setup is incomplete, maintenance reason is Initial Setup Required.
  • Non-setup routes redirect to /maintenance/setup.
  • /maintenance/setup is reachable from localhost/VPN.

If setup is already complete and you need to re-test this path:

  • Use a disposable local DB instance/state and rerun startup checks.
  • Do not reset shared/staging data.

7) Recovery Validation

  1. Restart with healthy config source (valid AWS creds) or disable SSM source:
STUDIO_PARAMETER_STORE_ENABLED=false pnpm dev
  1. Confirm degraded config maintenance reason is cleared after recovery.
  2. Complete any setup requirements and confirm setup-maintenance exits.

Expected:

  • No degraded-source maintenance after recovery.
  • App returns to normal operation when setup is complete.

Quick interpretation guide:

  • If startup prints [config] Configuration loaded successfully, config-source loading worked.
  • If you only see auth callback exceptions after login redirect, treat those as auth-domain issues to debug separately from config-source behavior.

8) Quick Infra CLI Smoke Checks

pnpm studio infra stack validate --dry-run
pnpm studio infra deploy create --stack-name demo --dry-run
pnpm studio infra deploy update-image --stack-name demo --image demo:local --dry-run

Expected:

  • Commands print the AWS operation in dry-run mode.
  • No command parsing errors.