This mechanism works perfectly — when the challenge ACK is allowed through. But that was exactly what the new USPS change was blocking.
The culprit: A well-intentioned fix
The change in USPS had been designed to fix a legitimate security issue in Private Service Connect’s proxy protocol handling. Part of its effect was to drop packets it classified as “invalid ACKs” during the connection handshake phase — specifically, ACKs that arrived before the initial SYN-ACK had been sent, which in normal circumstances would be anomalous.
Unfortunately, challenge ACKs look exactly like this: They arrive from the server before the new handshake has completed, in response to a SYN. From the USPS’s new perspective, they looked like invalid packets. They were dropped silently.
Why this broke GKE specifically: Private Service Connect uses NAT to share a single public endpoint across many client pods. This means many different pods — running on different nodes, from different namespaces — can all end up using the same source port after NAT, creating a high rate of 5-tuple collisions. The challenge ACK mechanism was being triggered far more often than it would be in a normal direct-connection scenario. The USPS fix hadn’t introduced challenge ACKs — they had always been there. It had introduced the dropping of them, which broke a flow that GKE’s TCP stack had silently depended on to clear stale state.
Mitigation: Rolling back the change
In February, we rolled back the USPS change, restoring the old behavior where challenge ACKs were passed through instead of dropped. The rate of “Proxy protocol drop: invalid ACK packet” dropped back toward baseline almost immediately.
By early March , 2026, the customer confirmed that all affected clusters had recovered. No more timeout errors.
But the rollback wasn’t a fix. It was a band-aid. It restored the behavior the system had accidentally depended on. The deeper question — why was the KCP VM in FIN_WAIT_2 for over two minutes in the first place? — was still completely unanswered.
That question led to Part 3. And it’s where things got genuinely strange.
Part three: Three bugs, one crime scene
The mitigation bought us time, but it left a question hanging in the air: Why did the KCP VM keep connections in FIN_WAIT_2 for so long? A well-behaved TCP stack processes a FIN, acknowledges it, and closes the connection. The KCP VM was failing to do this, leaving half-open connections that persisted for more than two minutes — long enough for the port to be reused by Private Service Connect’s NAT layer, long enough to cause the challenge ACK scenario we’d uncovered.
To answer this, we needed to understand not just one bug, but three distinct failure modes that had intersected at the worst possible moment.
Bug #1: Packet reordering at the PIGv
The first piece was discovered through careful analysis of simultaneous captures at two points: the KCP VM’s host NIC (inside the physical host running the VM) versus the PIGv — the virtual NIC as seen by the VM.
At the host NIC level, packets arrived in perfect order. But by the time they crossed through USPS and appeared at the PIGv, they had been subtly reordered.
This happens because of how Private Service Connect’s miss-thread processing works. When Private Service Connect intercepts a new flow it hasn’t seen before (a “miss”), it processes that packet through a slower path that adds the proxy protocol header and sets up NAT state. Subsequent packets in the same flow take a fast path. The consequence: a FIN or RST packet belonging to a flow in mid-close might arrive slightly out of order relative to the ACKs that followed it.
What the KCP VM actually received (reordered sequence)
