Audit chain signing key

Every row in contract_history carries a row_hash (SHA-256 over the row content + previous row's row_hash) and a row_hmac (HMAC-SHA256 over row_hash). The HMAC is what stops an attacker with DB write access from rewriting the entire chain — they would need the signing key too.

The signing key lives in one of two places, controlled by AUDIT_SIGNING_MODE:

Mode	Where the key lives	Threat coverage
env (default)	Env var CONTRACT_AUDIT_HMAC_KEY on the server	Stops DB-only attackers. Loses to anyone who reads the server's process env.
kms	AWS KMS HSM. Server only ever calls kms.GenerateMac / kms.VerifyMac, never touches the bytes.	Stops DB-only and most server-compromise attackers. Requires CloudTrail-logged IAM credentials to forge.

Each row records its signing_mode, so verification dispatches to the right signer. Mixing modes is supported (e.g., new rows signed by KMS, older rows verified against env-mode HMAC) — convenient for cutover.

Local development

Default (no AWS account required)

docker compose up -d --build runs in env mode out of the box. The default key is dev-only-rotate-in-production and the server logs a one-time warning at boot. No further setup needed.

Testing KMS mode locally

Useful before flipping the production server. You'll spend ~$1/month for the KMS key plus pennies in API calls.

1. Create a KMS key in AWS Console (one-time, ~2 min):
   • Region: ap-east-1 (Hong Kong) or wherever you'll deploy
   • Key type: Symmetric
   • Key usage: Generate and verify MAC
   • Key spec: HMAC_256
   • Alias: tutor-crm-audit-hmac (or any human-readable name)
   • Add your admin IAM user as key administrator
   • Add your application IAM user (e.g. user-foxclm) as key user, with kms:GenerateMac + kms:VerifyMac
2. Generate access keys for the application IAM user in IAM Console -> Users -> Security credentials. Copy them directly into your env. Never paste into chat, source control, or shared docs.
3. Add a .env file at sample/ (gitignored) with:

   AUDIT_SIGNING_MODE=kms
   KMS_KEY_ID=arn:aws:kms:ap-east-1:<account-id>:alias/tutor-crm-audit-hmac
   AWS_REGION=ap-east-1
   AWS_ACCESS_KEY_ID=<your application user's access key>
   AWS_SECRET_ACCESS_KEY=<your application user's secret>

4. Restart the server:

   docker compose restart server
   docker compose logs -f server | head -20

   Look for [contract-audit] using KMS signer (region=ap-east-1, ...). If you see the dev-default warning instead, the env vars didn't reach the container.
5. Make any contract status change (e.g. submit a Draft) — the new history row will have signing_mode='kms' and a KMS-issued MAC. Older rows stay at signing_mode='env'.
6. Verify the chain:

   curl -s "http://localhost:3001/api/v1/contracts/<id>/audit/verify" -H "Authorization: Bearer <token>"

   Should return valid: true. If KMS is unreachable or your IAM user lacks kms:VerifyMac, KMS-signed rows will report valid: false with reason row_hmac mismatch (... kms signing key required).

Switching back

Remove AUDIT_SIGNING_MODE from .env (or set it to env) and restart. New rows will be env-signed; existing KMS-signed rows still verify as long as your KMS credentials remain valid.

Vultr production deployment

This is the canonical setup for the production server.

Prerequisites

• AWS account with the KMS key created (per local-dev step 1 above; create in the region you'll deploy near)
• Application IAM user user-foxclm (or similar) with only these permissions on the specific key ARN:
  • kms:GenerateMac
  • kms:VerifyMac
  • kms:DescribeKey (optional, for diagnostics)
• The application user must NOT have console login enabled, only programmatic access keys

One-time server setup

Split env vars by sensitivity, not by location. Secrets get a tightly-restricted file; non-secret config lives in your compose / systemd unit. This keeps the secrets file small (easier to rotate, easier for secret scanners) and lets boring config live alongside the rest of your deployment definition.

SSH into the Vultr box, then:

1. Create the secrets file at /etc/tutor-crm/server.env with chmod 600 and owner = the user running the server process. Only put secrets here:

   DB_PASSWORD=<database password>
   JWT_SECRET=<random 32+ byte string>
   AWS_ACCESS_KEY_ID=<application user's access key>
   AWS_SECRET_ACCESS_KEY=<application user's secret>
   CONTRACT_AUDIT_HMAC_KEY=<random 32+ byte string used by migrations _019/_020/_021>
   SMTP_PASS=<smtp password>
   DEEPSEEK_API_KEY=<deepseek key, if used>
   STRIPE_SECRET_KEY=<stripe secret, if used>
   STRIPE_WEBHOOK_SECRET=<stripe webhook signing secret, if used>
   GOOGLE_CLIENT_SECRET=<google oauth secret, if used>

   CONTRACT_AUDIT_HMAC_KEY stays here because migrations _019/_020/_021 use it directly (they run at deploy time, can't reach KMS). After initial deploy, KMS takes over for new rows.
2. Put non-secret config in your docker-compose.yml (or systemd unit Environment=) — it can live in the repo. None of these are credentials:

   services:
     server:
       env_file:
         - /etc/tutor-crm/server.env       # secrets only
       environment:
         DB_HOST: db
         DB_PORT: 3306
         DB_USER: tutor_crm_app
         DB_NAME: tutor_crm
         PORT: 3001
         AUDIT_SIGNING_MODE: kms
         KMS_KEY_ID: arn:aws:kms:ap-east-1:<account-id>:alias/tutor-crm-audit-hmac
         AWS_REGION: ap-east-1
         ELASTICSEARCH_URL: http://elasticsearch:9200
         SMTP_HOST: smtp.your-provider.com
         SMTP_PORT: 587
         SMTP_USER: <smtp username>
         SMTP_FROM: noreply@yourdomain.com
         CLIENT_URL: https://app.yourdomain.com
         SERVER_URL: https://api.yourdomain.com
         VIEWER_URL: https://sign.yourdomain.com
         GOOGLE_CLIENT_ID: <google oauth client id>
         GOOGLE_REDIRECT_URI: https://api.yourdomain.com/api/v1/google/callback

   The KMS key ARN is fine in compose: an ARN identifies the key, but actually using it requires IAM credentials (which are in the secrets file). Same for GOOGLE_CLIENT_ID, STRIPE_WEBHOOK_SECRET's sibling identifiers, etc.
3. Outbound network: ensure port 443 is open outbound to kms.ap-east-1.amazonaws.com. No inbound rules to AWS required.

Why split rather than one big env file

	Single env file	Split (secrets vs config)
Audit a config change	Read the secrets file (high-trust task)	Read the compose file in your repo
Rotation cadence	Mixed (secrets every 90 days, URLs once a year)	Each file matches its own cadence
Secret scanners	Pollute results with non-secret strings	Small file, every alert meaningful
Onboarding	Junior dev needs root to see how the app is wired	Compose tells the story; secrets stay locked
Blast radius if leaked	Everything	Only the file that leaked

Worth the 5 extra minutes of split.

One step further (when you have a paying customer)

The ideal is no secrets file on disk at all: server reads from AWS Secrets Manager (or HashiCorp Vault) at boot. Vultr doesn't have IAM Instance Profiles like AWS EC2, but you can replicate the pattern: a single bootstrap IAM key in server.env that only has permission to fetch the real secrets from Secrets Manager. The blast radius if Vultr is compromised becomes "rotate the bootstrap key in Secrets Manager", not "rotate every secret your app touches". Worth the extra hour when contracts you're signing become litigation-worthy.

Verifying the deployment

After the first deploy:

journalctl -u tutor-crm-server | grep '\[contract-audit\]'

Expected: one line [contract-audit] using KMS signer (region=ap-east-1, key=arn:aws:kms:...). If you instead see the dev-default warning, the env file isn't being read.

Trigger a status change on any contract and verify:

SELECT id, signing_mode, action FROM contract_history ORDER BY id DESC LIMIT 5;

The most recent row should be signing_mode=kms.

IAM policy template

Attach to the application IAM user (user-foxclm):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kms:GenerateMac",
        "kms:VerifyMac",
        "kms:DescribeKey"
      ],
      "Resource": "arn:aws:kms:ap-east-1:<account-id>:key/<key-uuid>"
    }
  ]
}

Replace <account-id> and <key-uuid> with the values from the key's ARN. Use the key UUID, not the alias in the IAM policy — aliases aren't valid resources for kms:GenerateMac.

Failure modes

Symptom	Likely cause	Fix
Server boots but logs the dev-default warning	Env file not reached the process	Check process manager EnvironmentFile path + permissions
recordContractHistory throws AccessDeniedException	IAM user lacks kms:GenerateMac on this key	Update the user's policy with the correct key ARN
Verify returns valid: false for new rows	Same as above — VerifyMac is also denied	Add kms:VerifyMac to the policy
Latency spike on contract status changes	KMS round-trip from Vultr to AWS (~30-80ms)	Expected; not on user critical path. Move to AWS region closer to Vultr DC if needed
KMS unreachable (network outage)	Can't sign new rows	Status transitions fail until KMS is reachable. If this is unacceptable, queue-and-retry is the next iteration.

Key rotation

KMS HMAC keys cannot be auto-rotated by AWS (asymmetric/symmetric encryption keys can; HMAC cannot, as of this writing). Rotation strategy:

1. Create a new KMS HMAC key with a new alias (e.g. tutor-crm-audit-hmac-v2)
2. Add the application IAM user to the new key's policy
3. Update KMS_KEY_ID env var on the server
4. Restart the server
5. New rows are signed by the new key (recorded in signing_mode=kms — the row implicitly references whichever key was active at sign time)
6. Old rows stay verifiable as long as the old key is still in your AWS account; don't delete the old key until you're certain no row references it

For long-term archival, schedule a copy to immutable storage (S3 Object Lock) — a dedicated audit chain anchoring doc will cover this when that ships.

Cost projection

For a tutoring CRM at typical SMB scale:

• 1 HMAC key: $1.00/month
• ~100 status changes/day -> 3,000 sign + verify calls/month: ~$0.01
• Total: ~$1/month, well under the free tier for requests

Code references

• Pluggable signer: server/src/services/contractAuditSigner.ts
• Insert path: server/src/services/contractHistory.ts:recordContractHistory
• Verify endpoints: server/src/routes/contracts.ts — /audit/verify, /audit/verify-all, /audit/run-check
• Migrations: _019 (chain columns), _020 (HMAC column + backfill), _021 (orphan repair), _023 (signing_mode column)