PET implementations have a unique failure mode: the privacy guarantee is real, but it applies to the protocol in isolation. The surrounding system — the data pipeline that feeds the protocol, the output processing that follows it, the auxiliary information available to an adversary, the implementation choices that affect the security assumptions — can invalidate the guarantee without touching the cryptography. A correctly implemented FHE scheme in a system that logs the plaintexts before encryption provides no privacy. A correctly implemented federated learning protocol with a gradient inversion-vulnerable update scheme provides no privacy against the aggregator. The cryptographic core can be correct while the system fails.
01
Federated learning gradient updates are not protected against inversion attacks
Federated learning’s promise is that raw data does not leave participants. What does leave is gradient updates — the parameter changes computed from the local training data. Gradient inversion attacks, demonstrated to be effective in 2020 and since improved substantially, can reconstruct training examples from gradient updates with high fidelity for image classifiers, text classifiers, and tabular data models. An aggregator who collects individual gradient updates — rather than using secure aggregation — can reconstruct the training data of individual participants. Deployments that implement federated learning without secure aggregation or without differential privacy applied to the gradient updates provide significantly weaker privacy than their architects assumed.
What this looks like in a healthcare deployment
A hospital network deploys federated learning for a diagnostic model, believing that patient data never leaves the hospitals. The central aggregator receives individual per-hospital gradient updates. A research paper published 18 months after deployment demonstrates that the specific model architecture used is vulnerable to gradient inversion, and that individual patient chest X-ray images can be reconstructed from the gradient updates with sufficient quality for clinical recognition. The hospitals’ belief that patient data had not left their systems was correct for the raw data. It was incorrect for the gradient updates that represented it.
Architecture approach that prevents this
Secure aggregation: using a cryptographic protocol (typically based on secret sharing or homomorphic encryption) that allows the aggregator to compute the sum of gradient updates without seeing any individual participant’s update. The aggregator sees only the aggregate, not the contributors. Combined with DP-SGD applied at each participant to add calibrated noise before the update is computed, this provides both practical and formal privacy against gradient inversion at the aggregation layer.
02
Differential privacy budget is not tracked across queries or training runs
Differential privacy is a composable guarantee: each query or training step consumes a privacy budget, and the total privacy loss is the sum (or a tighter composition bound) across all queries. A system that provides (ε=1)-DP per query but executes 1,000 queries without budget tracking has consumed 1,000 units of privacy budget, not 1. The composition theorem means that answering unlimited queries with DP noise eventually provides no privacy guarantee — the privacy loss accumulates without bound. Many differential privacy deployments set an ε for individual queries and never track the cumulative budget, making the per-query guarantee meaningless as a system-level property.
What this looks like in practice
A government statistical agency deploys differentially private query answering with ε=0.5 per query, believing they are providing strong privacy. Researchers make 2,000 queries over 18 months, each at ε=0.5. The cumulative privacy loss under basic composition is ε=1,000. Under advanced composition, it is higher than the composition bound assumed. The privacy guarantee that justified the data sharing arrangement has long since been exhausted. The agency has been providing effectively no differential privacy protection for the majority of the queries made against the dataset.
Architecture approach that prevents this
Privacy accountant implementation as a core infrastructure component: every differentially private operation consumes from a global privacy budget that is tracked and reported. The privacy accountant uses Rényi DP accounting or Gaussian DP accounting for tighter bounds than basic composition. The system enforces a hard budget limit — additional queries are refused when the budget is exhausted — or the budget is reset on a defined cycle with documented justification. The privacy budget and its current consumption are reported as operational metrics alongside the system’s functional metrics.
03
Homomorphic encryption is applied correctly but the surrounding system leaks plaintext
The formal guarantee of homomorphic encryption applies to the computation: the server performing the computation has access to ciphertext, not plaintext. The guarantee does not extend to the data pipeline surrounding the computation. If the client application logs the plaintext before encryption, the plaintext is in the logs. If the client sends the plaintext to an error reporting service when the encryption fails, the plaintext leaves the protected environment. If the decrypted result is cached in an unencrypted store for performance, the results are in the cache. If the key management system that holds the decryption keys is accessible to the same parties as the ciphertext, the encryption provides no protection. The HE core can be cryptographically correct while the surrounding system fails to protect the plaintext.
The most common plaintext leakage point
A financial services company deploys FHE for cloud ML inference, ensuring that the ML model provider never sees plaintext transaction data. The client application sends transaction features as ciphertext to the cloud. The application’s error logging, however, logs all input features in plaintext for debugging. The logs are stored in the same cloud environment as the inference service. The ML model provider, whose compute environment the client is using, has access to the log storage through the shared cloud tenancy. The FHE prevents the ML provider from seeing plaintext during inference. The logging exposes it through a side channel that the FHE design did not consider.
Architecture approach that prevents this
Full data flow analysis before deployment: every path through which plaintext can exit the protected environment is identified and either eliminated or explicitly accepted as within the threat model. The threat model for the FHE deployment specifies which parties are trusted and what trust level each has. Every component in the data pipeline is assessed against the threat model — not just the cryptographic components. Log sanitisation and encrypted log storage are infrastructure requirements, not optional optimisations.
04
MPC protocol is secure against the specified adversary model but deployed in a context where a stronger adversary applies
MPC protocols are proved secure against a specific adversary model: semi-honest (parties follow the protocol but try to learn information from their view), or malicious (parties may deviate from the protocol arbitrarily). Semi-honest protocols are significantly more efficient than malicious protocols and are often deployed when the deployment context actually involves malicious adversaries. An organisation that deploys a semi-honest MPC protocol between two parties who have incentive to cheat — competitors computing a joint function — has deployed a protocol whose security proof does not apply to its deployment context. A malicious party can learn information from a semi-honest protocol that they could not learn from a correctly-implemented malicious protocol.
When this matters
Two competing financial institutions deploy a semi-honest MPC protocol to compute portfolio overlap for systemic risk assessment. The protocol is secure if both parties follow it. Institution A, knowing the result of the computation before Institution B does in one protocol round, deviates from the protocol in a way permitted by the semi-honest threat model — they observe their own view and make inferences. A malicious MPC protocol would have prevented this. The semi-honest protocol is functioning correctly within its security proof; the deployment context assumed a weaker adversary than was actually present.
Architecture approach that prevents this
Threat model specification before protocol selection: the adversary model for the MPC deployment must be established from the actual relationship between the parties, not assumed to be semi-honest because semi-honest protocols are easier to deploy. If the parties have any incentive to deviate from the protocol, the malicious adversary model applies and a malicious-secure protocol is required. The performance cost of the malicious protocol is the cost of operating in the actual threat environment. If malicious-secure performance is unacceptable, the deployment should be redesigned to achieve mutual trust before using a semi-honest protocol.
05
Federated learning convergence fails on non-IID data without diagnosis or remediation
Standard federated averaging (FedAvg) assumes that the local datasets at each participant are independently and identically distributed samples from the global data distribution. In almost every real deployment, they are not. Hospital A specialises in cardiology and its data skews heavily towards cardiac diagnoses. Hospital B is a general hospital. Hospital C serves a specific demographic with disease prevalence patterns different from the general population. Non-IID data causes FedAvg to converge slowly, to a worse global optimum, and sometimes not to converge meaningfully at all. The federated model can be significantly worse than a model trained at any single participant, providing neither the privacy benefit of federation nor the capability benefit. Non-IID data is not an edge case — it is the norm in every real-world federated deployment.
What non-IID failure looks like operationally
A diagnostic AI programme deploys federated learning across 8 hospital sites. After 200 rounds of federated training, the global model achieves 71% accuracy on the test set. The local models trained at individual sites achieve 84–91% accuracy on their local test data. The federated model is materially worse than any individual site model. The programme has implemented the privacy protection of federation at the cost of making the model worse for every participant than if they had not participated. The convergence failure was not diagnosed during the design phase and was not discovered until the production evaluation against local models.
Architecture approach that prevents this
Data heterogeneity analysis before deploying federated learning: measuring the divergence between local data distributions across participants and quantifying the expected convergence impact. For deployments with high heterogeneity, FedProx, SCAFFOLD, or personalised federated learning approaches that tolerate non-IID data are specified. Convergence benchmarking against the centralised training baseline is conducted before production deployment to verify the federated model meets the minimum acceptable performance threshold.
06
The privacy guarantee is formally correct but the output enables re-identification through auxiliary information
Privacy-enhancing technologies protect against specific inference attacks by an adversary with no auxiliary information. Real adversaries have auxiliary information. A differentially private aggregate that does not individually identify anyone can, when combined with publicly available data about the individuals in the dataset, enable re-identification. A federated model that does not reveal training data can, when queried with known examples, confirm their membership. An MPC result that reveals only the intended output can, when combined with the querying party’s prior knowledge of the other parties’ data, reveal more than intended. The formal privacy guarantee is a bound on the information revealed by the protocol in isolation. It is not a bound on the information inferable from the protocol output combined with auxiliary information.
The Netflix re-identification precedent
Netflix released an anonymised movie rating dataset satisfying k-anonymity with k=5 — a state-of-the-art privacy standard at the time. Researchers demonstrated that combining the released dataset with IMDB public ratings allowed re-identification of the majority of records whose IMDB ratings were publicly visible. The anonymisation was correctly implemented against the specified threat model (an adversary with no auxiliary information). The actual adversary had auxiliary information, invalidating the privacy guarantee. Every PET deployment must model the adversary’s auxiliary information, not just the adversary’s access to the protected computation.
Architecture approach that prevents this
Auxiliary information threat modelling as a mandatory component of every PET design: identification of what auxiliary information a realistic adversary would have, and assessment of what can be inferred from the protocol output combined with that auxiliary information. For public data releases, re-identification risk assessment against the specific auxiliary datasets available to the expected adversary. Privacy guarantees are stated with explicit scope: “assuming an adversary with no auxiliary information beyond X, this protocol provides (ε, δ)-DP”.
07
Trusted execution environments are treated as unconditionally secure
Trusted Execution Environments (TEEs) — Intel SGX, AMD SEV, ARM TrustZone — provide hardware-based isolation for computation, preventing the host OS, hypervisor, and cloud provider from accessing the data being processed inside the enclave. They are a powerful component of confidential computing. They are not unconditionally secure. SGX has been compromised by multiple side-channel attacks (Spectre, Meltdown, SGAxe, CrossTalk, and others) that extract secrets from enclaves without requiring OS-level access. TEE security depends on maintaining up-to-date microcode, correctly implementing the attestation protocol, not relying on the TEE for more than it provides, and carefully managing the enclave’s attack surface. Deployments that treat TEE isolation as equivalent to perfect cryptographic isolation have a security model that does not hold under the demonstrated attack surface.
The SGX side-channel attack surface
A confidential computing deployment uses Intel SGX to process sensitive healthcare records on untrusted cloud infrastructure. The deployment assumes that data inside the SGX enclave is inaccessible to the cloud provider. SGAxe (published 2020) demonstrated that SGX’s attestation keys can be extracted using microarchitectural attacks, allowing an adversary with physical access to the hardware or with root access to the host to forge attestation and extract enclave secrets. The security claim was correct for an adversary without microarchitectural access. It was incorrect for a cloud provider with physical access to the hardware.
Architecture approach that prevents this
TEE security is treated as one layer in a defence-in-depth architecture, not as a complete solution. The threat model explicitly addresses side-channel attacks and specifies which attack classes the TEE is assumed to resist and which it is not. Microcode and firmware maintenance is treated as a security-critical operational requirement with the same urgency as OS patching. For deployments where TEE side-channel attacks are within the threat model, the TEE is combined with cryptographic PETs rather than substituted for them.
08
Privacy-enhancing technologies are deployed without verifiable auditability or continuous assurance
A system may be architected with correct privacy-enhancing technologies at deployment time, yet lack any mechanism to prove that those guarantees remain intact over time. Cryptographic protocols, differential privacy mechanisms, federated learning pipelines, and secure enclaves all rely on correct implementation, correct configuration, and correct operation. Drift occurs: configurations change, dependencies update, logging expands, new integrations are added, and operational shortcuts are introduced under performance or business pressure. Without continuous verification, the system can silently degrade from a formally private system into one that leaks sensitive information. Privacy guarantees are not static properties; they are operational properties that must be continuously enforced and verified.
What silent degradation looks like in production
A large-scale analytics platform deploys differential privacy with a properly configured privacy accountant and strict query limits. Six months later, a new engineering team introduces a parallel analytics endpoint for internal use, bypassing the privacy accountant to improve query latency. Over time, this endpoint becomes widely used across the organisation. No audit mechanism detects that queries are being executed outside the DP framework. The system continues to report compliance with differential privacy guarantees, while in reality a significant portion of queries are executed without any privacy protection. The failure is not in the original design, but in the absence of continuous assurance and enforceable auditability.
Architecture approach that prevents this
Cryptographic and system-level auditability as a first-class requirement: every privacy-sensitive operation is logged in a tamper-evident audit trail (e.g. append-only logs with cryptographic integrity guarantees). Continuous verification systems validate that all data flows, queries, and computations pass through the enforced privacy controls. Remote attestation, reproducible builds, and policy enforcement layers ensure that only approved code paths can process sensitive data. Privacy guarantees are continuously monitored as operational metrics, with automated alerts and hard enforcement when deviations occur. Independent audit capability — internal or external — is built into the system design, ensuring that privacy claims can be verified at any point in time, not just asserted at deployment.