Development and infrastructure operations

Period  — 

The backend, console, collectors and agent runtime are split across 19 repositories. vat manages changes that span repositories and the contracts between them, and only what changed is validated. A single Proxmox server hosts the API, PostgreSQL, MinIO and Jenkins separately, and the console runs on Vercel. Backups are encrypted, stored in Cloudflare R2 and confirmed by restoring them. Credentials are encrypted with SOPS.

A release process that coordinates changes across 19 repositories with vat
and checks production state and business records after deployment

Team
Solo
Role
Cross‑repository contractsImpact‑based validationRelease processServer setupBackup and restore
Built tools
vatorch

GitPythonShellProxmoxDockerJenkinsPostgreSQLMinIOCloudflare R2SOPSPrometheusVercel

  • Records that bundle changes across repositories with participating commits, validation and rollback points
  • Contracts that declare the integrations between repositories and check them by machine
  • Validation that selects checks from the base commit, current file hashes and dependencies, and blocks when the impact is unclear
  • A release process that ends only after code checks, push, deployment, production lookups and business record updates all pass
  • Jenkins deployment that runs validation, image transfer, migrations, service start and a check of the running commit
  • Backups that encrypt PostgreSQL dumps and required objects, store them in Cloudflare R2 and confirm them by restoring into an isolated environment
  • Credentials encrypted with SOPS and kept in Git

Changes to tables another repository reads

Problem
Collector SQL that reads backend tables directly can break silently when backend migrations change.
Solution
Tables, columns, types and allowed values in collector SQL are checked against the migrations, and dynamic SQL of unknown shape is blocked. Live database checks and schema comparison run separately.

Deployments that left the records behind

Problem
If only the product is deployed and the business record stays local, the next person cannot tell what is actually deployed.
Solution
Work ends only after code checks and push, deployment within the allowed scope, production lookups and business record updates all pass.

Backups never decrypted

Problem
Having backup files does not show whether they can actually be decrypted and restored.
Solution
A backup counts as done only after an offsite copy is made, decrypted and restored into an isolated environment.

Browser zombie processes exhausting PIDs

Problem
Chromium child processes left by browser automation were never reaped, and the zombies used up the PID limit. Health checks and request threads could not start.
Solution
An init process was enabled in the container, and each deployment checks that the real configuration has it turned on.