Validating Cilium FQDN egress without touching a single-node production cluster
A disposable kind lab that proved identity-specific toFQDNs and Hubble drops while keeping the risky production CNI migration behind an explicit gate.
Infrastructure
Deployment, observability, cloud infrastructure, reliability, and operations.
↗
Why a public ALB still had to be closed behind CloudFront
A zero-downtime origin lockdown that moved the BFF first, then restricted direct ALB HTTP access to AWS-managed CloudFront origin-facing networks.
Designing Kafka consumer groups when every Pod must receive an SSE event
A cutover from lightweight database notifications to Kafka KRaft, including broadcast semantics, a duplicate-bean rollout failure, and a live broker interruption.
Running a JVM with JNA and ZIP scratch files on a read-only root filesystem
A write-path audit that reduced a writable container to explicit /tmp scratch and an export PVC, then verified the native FFI path in production.
Why a monthly ingestion job did not need a new distributed-lock service
An EventBridge Scheduler Universal Target invoking ECS, with manual and scheduled runs serialized by a PostgreSQL advisory lock held on one physical connection.
How one failed ECS revision delayed the next deployment for five hours
Separating a masked shell exit code, ECS launch backoff, and an undersized health grace period to make automatic rollback trustworthy.
Healthy dashboards, silent alerts, and an empty trace backend
Repairing quiet observability failures by deriving Prometheus label contracts and Spring Boot tracing autoconfiguration from live evidence.
Cross-instance real-time updates without adding Redis
Using PostgreSQL NOTIFY only as invalidation, then combining authoritative rereads with SSE snapshots for a small multi-instance ECS service.
Replacing an unencrypted RDS instance without losing data
A snapshot-copy-and-restore migration for an RDS encryption setting that cannot change in place, including Terraform state recovery and 15 minutes of measured downtime.
Why a Synced StatefulSet kept running the old configuration
A production incident where Kafka and Tempo remained on broken probes and memory settings even after the Git manifests were corrected.
Production was Ready while the primary URL served the old release
Git and Vercel succeeded, but a manually assigned .vercel.app alias remained pinned to an older deployment until promotion became an explicit, reversible step.
Why green CI left ARM64 k3s running the previous image
A delivery record that turned an AMD64-only OCI manifest into separate build, registry, Git write-back, reconciliation, and runtime gates.