Linkzine

CISA contractor leaked AWS GovCloud keys to public GitHub for six months

Thorsten

Patch Tuesday broke records again, a CISA contractor left AWS GovCloud keys public for six months, and lobste.rs quietly ditched MariaDB for SQLite.

// SECURITY FOCUS

CISA contractor leaked AWS GovCloud keys to public GitHub for six months

The postmortem reveals CISA didn’t catch the exposure itself – KrebsOnSecurity notified them. If the national cyber defense agency’s contractor-credential hygiene and repo-scanning coverage have this gap, yours probably does too. The operational lesson: secret scanning on public repos needs to alert in hours, not months, and contractor-owned repos need the same policy enforcement as internal ones.

What to do: Audit your GitHub org settings this week: confirm secret scanning alerts are enabled on all repos including forks, and verify contractor accounts are in scope.

  1. Slack’s Shipyard: rebuilding EC2 fleet management from the ground up — Slack Engineering · Jul 14
    Slack rebuilt its EC2 fleet management platform – called Shipyard – to replace continuously-mutating long-lived instances with immutable, image-based deployments across tens of thousands of EC2 instances. The core idea is a layered AMI model: a hardened base image called slack-zero (built with AWS Image Builder rather than Packer) provides OS hardening, networking, monitoring agents, and common tooling, and service teams layer their own AMIs on top. Configuration management moves from continuous Chef runs to two distinct phases – baking (packages and environment-agnostic config) and provisioning (secrets and runtime config at boot) – so instances come online in seconds rather than minutes. Progressive rollouts run through Gondola, their deployment orchestrator, with metric-based automated safety checks and auto-rollback; Karpenter handles Kubernetes worker node refreshes and AWS Instance Refresh covers ASGs. A companion inventory system called Peekaboo, built on EventBridge, OpenSearch, and Lambda, gives near-real-time fleet visibility without relying on Chef Server as source of truth. This is primarily relevant for teams running workloads that can’t move to containers – Kubernetes nodes, egress stacks, infrastructure components – where EC2 immutability was previously hard to enforce.
  2. Netflix’s real-time service dependency map: architecture and lessons at scale — Netflix TechBlog · Jul 13
    Netflix’s SRE team published a detailed post-mortem on building a real-time service dependency map that processes millions of flow records per second across four regions, typically delivering topology updates within tens of minutes versus the hours-old data batch systems produce. The core is a three-stage distributed aggregation pipeline: Stage 1 ingests raw eBPF network flows from Kafka, Stage 2 resolves network intermediaries (load balancers, NAT gateways, proxies) into direct app-to-app edges, and Stage 3 handles enrichment and graph persistence. They started with two stages but production load created hot nodes where single instances received 100x typical traffic, because intermediary resolution requires co-locating all flows for a given intermediary – splitting into three stages isolated the compute-heavy resolution from the I/O-heavy enrichment. gRPC was dropped in favor of Server-Sent Events because serialization and connection pool overhead consumed more CPU than business logic. Backpressure propagates upstream through the pipeline to Kafka consumers rather than dropping records, so load spikes slow ingestion instead of crashing instances. The post is honest about failure modes and worth reading if you’re building any distributed aggregation pipeline, not just topology systems.
  3. Flux Schema brings CEL-based manifest validation and an Ecosystem Catalog — Flux CD · Jul 13
    The Flux project released a new CLI plugin called Flux Schema that validates Kubernetes manifests against JSON Schemas and CEL rules locally, using the same evaluation engine as the Kubernetes API server. The practical payoff is catching errors – missing required fields, wrong types, violated CEL constraints like a HelmRelease missing both `chart` and `chartRef` – in pull requests rather than as failed reconciliations on the cluster. It ships with an Ecosystem Schema Catalog covering roughly 100 projects and close to 9,000 schemas, refreshed daily from upstream releases and served over CDN. SOPS-encrypted Secrets are handled without decryption. A second feature, `flux schema discover`, does a static inventory scan of a GitOps repo and classifies directories by type, which is mostly aimed at AI agents that struggle to navigate GitOps repository layouts. The team also benchmarked an MCP server backed by the catalog: with it, a tested agent scored 4/4 on manifest tasks vs. 1/4 from training data alone, using 57% fewer tokens than web search – though that benchmark was run by the Flux team itself, so take the numbers as directional.
  4. lobste.rs completes MariaDB-to-SQLite migration, considers it stable — Simon Willison · Jul 14
    Lobsters completed its long-delayed database migration from MariaDB to SQLite this weekend and considers it stable. The community site’s Rails app now runs on a single VPS with a primary SQLite database of around 3.8 GB, plus separate cache (1.1 GB), queue (218 MB), and Rack::Attack (555 MB) databases. The migration – 735 lines added, 593 removed, across 30 commits and 188 files by Thomas Dziedzic – brought down CPU usage, memory usage, and roughly halved VPS costs. For anyone still defaulting to a separate database server for a low-to-medium traffic Rails app, this is a concrete data point that a single-server SQLite setup holds up in production.
  5. A broken .AL DNSSEC rollover triggered EDE 33 – the new signal that validation was bypassed — Cloudflare Blog · Jul 14
    A botched DNSSEC key rollover by Albania’s .AL registry on July 3, 2026 broke the chain of trust for the entire TLD – roughly three hours after the failure, Cloudflare applied a Negative Trust Anchor (NTA) to 1.1.1.1 to restore resolution while the registry sorted itself out. The core problem NTAs have always had is that they’re invisible: a response served under one looks identical to a fully validated response, so clients, monitoring tools, and applications had no way to know DNSSEC verification was bypassed. For the .AL incident, 1.1.1.1 now emits EDE code 33 (Negative Trust Anchor) alongside every affected response – proposed in an Internet-Draft co-authored with Babak Farrokhi at Quad9 and already assigned by IANA. The .AL zone remains unsigned as of publication; the DS record has not been restored to the root, meaning every .AL domain is currently running without DNSSEC protections. kdig from the Knot project already recognizes EDE 33 by name; an Unbound patch is under review. If you run a validating resolver or depend on .AL domains, EDE 33 is the signal to watch for – absence of it no longer means validation succeeded.

// In other news

ai

cloud

culture

  • DSLs Enable Reliable Use of LLMs (Martin Fowler) · Jul 14 — Fowler argues DSLs constrain the output space enough that LLMs produce reliably correct code, with concrete examples showing where free-form generation fails and bounded generation holds.
  • What is “loop engineering?” (Pragmatic Engineer) · Jul 14 — Gergely Orosz unpacks “loop engineering” – the term covers event-driven, trigger-based automation patterns that AI tooling is now generating by default, and asks whether it’s a durable practice or hype.
  • AI-assisted migration: Critical lessons from the modernization frontlines (Thoughtworks) · Jul 13 — Thoughtworks practitioners share failure modes from AI-assisted legacy migrations – specifically where LLM-generated refactoring confidently breaks domain invariants that weren’t captured in tests.

dev

  • crates.io: development update (Rust Blog) · Jul 13 — The crates.io team’s six-month update covers token scoping improvements, sparse registry performance gains, and the ongoing work on trustless publish attestations – worth reading if you publish crates.

iac

  • Preparing for the post-quantum era: Discover and prioritize now (HashiCorp Blog) · Jul 13 — HashiCorp outlines a discovery-first approach to post-quantum cryptography migration – inventory your current cipher usage before committing to a migration timeline, not after.
  • Sign in to Pulumi Cloud with Passkeys (Pulumi Blog) · Jul 13 — Pulumi Cloud now supports passkeys for email/password accounts, removing SMS and TOTP as the only MFA options for teams that want phishing-resistant auth on their state backend.

k8s

  • Building a Custom Metrics Exporter for Kubernetes (Kubernetes Blog) · Jul 14 — Official Kubernetes blog walks through building a custom metrics exporter for HPA scaling on queue depth or business signals – fills the gap that CPU/memory autoscaling consistently misses.
  • Operating OpenTelemetry at scale with OpAMP (CNCF Blog) · Jul 13 — OpAMP lets you remotely manage, reconfigure, and update OpenTelemetry Collectors across heterogeneous fleets without SSHing into each node – worth evaluating if you run more than a handful of collectors.
  • Is a Pod the right deployment unit for an AI agent? (CNCF Blog) · Jul 14 — The kagent team explains why they moved from running agents inside a shared runtime to one-Pod-per-agent, covering the isolation, identity, and RBAC tradeoffs that drove the decision.

linux

obs

  • The Voyage of a Small Environment Variable (OpenTelemetry) · Jul 14 — The OTel Spring Boot starter gained declarative-config support in v2.26.0, letting you manage all instrumentation settings via a YAML file rather than scattered environment variables.

sec

  • Microsoft Patches a Record 570 Security Flaws (Krebs on Security) · Jul 14 — Microsoft’s July 2026 Patch Tuesday covers 570 CVEs – nearly triple the previous record – so triage queues are going to be ugly; prioritize anything touching Windows networking and Office.
  • Vulnerability in FIFA’s Network (Schneier on Security) · Jul 14 — FIFA’s internal network was reportedly exploitable by anyone with minimal access – a good reminder that high-profile orgs with complex vendor relationships often have wide-open lateral movement paths.
  • M-Red-Team: AsyncAPI Supply Chain Compromise via GitHub Actions (Wiz Blog) · Jul 14 — Malicious packages tied to a compromised @asyncapi GitHub Actions workflow hit npm; if you pull from @asyncapi directly or transitively, check your lock files now.
  • Rust-proof your code with our new Testing Handbook chapter (Trail of Bits) · Jul 13 — Trail of Bits added a Rust-specific security testing chapter to their public Testing Handbook covering fuzzing, unsafe auditing, and toolchain-level checks – useful reference before your next audit.

web

  • The Laravel Scheduler Under the Hood (Freek Van der Herten) · Jul 14 — Wendelaars’ deep look at Laravel Scheduler internals covers mutex locking, background process forking, and sub-minute task handling – useful when debugging why scheduled jobs silently skip in production.
  • Consistent navigation in WordPress 7.1 with persistent toolbar (Make WordPress Core) · Jul 13 — WordPress 7.1 adds a persistent toolbar across the entire admin including the block editor, replacing the inconsistent context-switching that has tripped up site builders since Gutenberg launched.
  • Laravel Quota: Usage Budgets for Calendar Periods (Laravel News) · Jul 14 — Laravel Quota adds calendar-period usage budgets (daily, monthly, etc.) to any model, letting you enforce API rate limits or feature entitlements without rolling your own counter logic.
  • X-post: Hardening GitHub Actions workflows across the WordPress organisation (Make WordPress Core) · Jul 13 — WordPress.org is systematically hardening its GitHub Actions workflows org-wide – the write-up is worth checking if you maintain plugins and mirror those workflow patterns.

Patch the creds, mind the NTAs, and check whether your resolver tells you when it’s lying to you.

, , , ,
, , , ,

Leave a comment