From Adversarial Attacks to Defensive Design - A Red Teaming Approach to AI Security (PFA2 PROJECT)
Created: 2026-05-13
Updated: 2026-05-31
Last year, I played a little with how AI has changed, revolutionized and empowered cybersecurity. From threat detection to incident response, AI has been a total game changer.
But a tool, no matter how powerful, can become a curse if it presents a threat surface itself.
And that’s exactly why I decided to work this year on the other side of the spectrum: Securing AI .
This article is a summary of the work I have done during my second year final project.
Overview
The model we work on is a malware classifier based over the Resnet50 CNN architecture, that take Portable Executables turned into images, found in the Malware Benign Image Classification Dataset, and classifies them within 7 families of malware (adware, backdoor, downloader, spyware, trojan , virus, worm ..) or as benign software.
Various attacks can target different stages of the AI/ML pipeline, from data collection all the way to deployment and inference.
Data poisoning, model theft, adversarial attacks and more threaten the integrity of the model and the effectiveness of the prediction. These attacks are detailed thoroughly in the OWASP ML TOP 10 LIST.
Our choice was to exploit the vulnerability at input level (adversarial images), starting with white-box attacks that are simple and effective and working our way up to see the extent of the damage we can achieve. Then, we place our model in a mock entreprise scenario and threat model it using the MITRE ATLAS framework , in an attempt to identify the vulnerabilities and suggest remediation and therefore security from the design stage .
Attack
White-box Scenario
In the white-box scenario we implement a couple of first-order attacks : Deepfool and Projected Gradient Descent (PGD). The choice has been made to showcase a couple of examples of untargeted vs targeted attacks.
Deepfool is a white-box adversarial attack that computes the minimum perturbation required to change a model’s prediction. It assumes that decision boundaries are locally linear and iteratively moves the input toward the nearest decision boundary , producing highly imperceptible adversarial examples.
Projected Gradient Descent (PGD) is a strong white-box adversarial attack that iteratively perturbs the input while enforcing a constraint on the perturbation magnitude. Starting from the original input, small gradient-based updates are applied repeatedly, and the result is projected back onto an $\epsilon$-ball to ensure that the perturbation remains bounded and imperceptible.
To the naked eye, these two visual representations of a PE are identical. But one is a clean sample from the dataset mentioned above, and the other is the result after applying a PGD-based perturbation : The model predicts the wrong label and classifies an adware as a benign software.
While both attacks achieve 100% attack success rate (ASR) across all samples tested, they exploit a very specific scenario: because first-order attacks use gradient information to craft adversarial examples, we assume here that we have total access to the model weights. This assumption requires additional work and bypassing other perimeters that are outside the scope of the project.
Moreover, the interesting aspect of the adversarial image attacks or evasion attacks is the following : they are still effective even when no prior knowledge about the target is acquired.
Black-box Scenario
In the black-box setting, we take a more real-world approach in testing the robustness of the target model : an attacker who doesn’t know the architecture, the weights or any internal detail. The model sits behind an API and the attacker can only send inputs and observe outputs.
While this setting seems impossible to exploit due to the lack of information and the “blindness” of the attacker, one would be surprised to find out that a fundamental property of adversarial examples, transferability, allows us to do exactly so.
Adversarial examples crafted on one model have a tendency to fool a completely different model, even if the two architectures have nothing in common. The intuition behind this is that models trained on similar data tend to learn similar decision boundaries in high-dimensional space, so a perturbation that pushes a sample across one boundary often pushes it across another.
This is why we introduce a surrogate model: a VGG-16 that we trained on the same dataset. The idea is simple: we craft adversarial examples on the surrogate (white-box), then fire them at the Resnet-50 target (black-box) and see what sticks. The choice of the architecture (fundamentally different from the target Resnet50) is to mimic the process of trial and error an attacker goes through.
This has resulted in really humbling results : the attacks work in the white-box case on the VGG-16 but transfer really badly to the target model (20% only for deepfool and 0% for the targeted PGD and only 16.6% causing any misclassification at all).
A brutal gap like this is due to the fact that iterative attacks overfit to the surrogate’s loss landscape and decision boundaries.
Yet with some changes, improvements can occur. And here is where it gets interesting.
Enter the AdvGAN. Instead of crafting perturbations sample by sample at inference time, we train a generative adversarial network to learn how to perturb inputs. It’s based over the GAN architecture,with the addition of the target model at the training phase in eval mode. The generator is a U-Net that takes a clean image and a target class, and outputs a structured perturbation. It is trained adversarially, with a discriminator pushing it to make perturbations that are visually indistinguishable from clean inputs. Trained against VGG-16 alone, the GAN pushes untargeted transfer up to 46.49% , already a significant jump over the PGD baseline. Targeted transfer, though, barely moves at 0.23%. The generator is still exploiting VGG-16-specific quirks rather than learning something general.
The fix? Ensemble training. Instead of training against a single surrogate, we compose an ensemble of VGG-16 (weight 0.7) and ResNet-18 (weight 0.3). The gradient signal now has to satisfy two architecturally distinct models simultaneously, which forces the generator to find perturbations that live in the intersection of their vulnerabilities, the features that are genuinely shared across model families. The result: untargeted transfer climbs to 51.83%, and targeted transfer jumps by a factor of ~8.7x, reaching 1.99%. The trade-off is that white-box performance on VGG-16 drops, which is exactly what you’d expect : the generator can no longer fully specialize.
Targeted black-box transfer is still low in absolute terms, and the literature is pretty clear on why: forcing a specific misclassification label to survive across architectures is a much harder ask than just causing any misclassification. Getting there would likely require more architectural diversity in the ensemble, or more aggressive input transformation strategies during training. But a 51% untargeted misclassification rate with zero access to the target model is already a meaningful result , and a real argument for why deploying a model behind an API is not, by itself, a security strategy.
Defense
Threat Modeling
Before you defend anything, you need to know what you’re defending and from whom.
We built MalVis, a fictional yet realistic internal malware classification platform around our model, deployed in a mid-sized cybersecurity firm.
The ResNet-50 classifier sitting at the heart of it,i s feeding predictions directly into a SOC’s detection-and-response workflow. Files come in from EDR agents on monitored endpoints, get preprocessed into images, run through the inference API, and route either to automated SOAR playbooks or to a SIEM for analyst triage. Analyst corrections feed back into a retraining loop. The whole thing spans three trust zones: a public-facing zone where every byte should be treated as potentially adversarial, a private subnet where the actual inference happens, and AWS cloud storage for model weights, logs, and training data.
In a first approach, classical threat modeling to the STRIDE framework handles the non-ML components (Kafka, PostgreSQL, the Web UI, SIEM and SOAR). I have worked on a fun minimal example like this, if you want to learn more check my previous blog post Threat Modeling a Simple Banking App . But STRIDE has a fundamental blind spot : it has no concept of the model itself as an attack surface. It won’t tell you that someone can probe your inference API to reconstruct a functional copy of your model, or that a compromised analyst account can silently shift your decision boundaries over successive retraining cycles.
That’s where MITRE ATLAS comes in : structured identically to ATT&CK, but built specifically for ML systems and grounded in documented real-world case studies. For a system where the ML model is the product, ATLAS isn’t optional.
The threat register surfaces two AI-specific threats at the Critical level : adversarial input crafting (AML.T0043) and training data poisoning (AML.T0020) . Both target the model’s decision function directly rather than the surrounding infrastructure. I intend on dedicating a deep-dive for this framework, one that’s coming in a (near) future post.
The remaining requirements form solid defenses around the non-ML components : rate limiting, cryptographic hashing, hardening containers .. But here’s the honest limitation of all of that: these are perimeter controls. They reduce the likelihood that a crafted input or poisoned label reaches the model. They however, cannot guarantee correct model behavior when one does. No firewall rule can fix that. But Adversarial training can.
Adversarial Training
The idea behind this security measure is astonishingly simple: if it gets fooled by it, train it on it.
Exposing the model to adversarial examples during the training phase won’t change predictions but will improve the model’s robustness and stop it from fitting on sensitivity-prone features.
In practice, this means splitting every training batch in half : 50% clean images, 50% adversarially perturbed , and letting PGD generate those adversarial examples on the fly during training. PGD is the natural choice here: it’s the same attack that reduced the standard model to 0%, so if you can train against it, you’re training against something close to a worst case.
However, throwing hard adversarial examples at a model that hasn’t learned anything yet has its toll : The optimization becomes unstable, convergence suffers, and the model never quite finds its footing. The fix is a curriculum: start with weak perturbations (ε = 2/255 in epoch 1) and ramp up gradually to the full budget (ε = 8/255 by epoch 4). The model builds stable representations on easier examples first, then progressively hardens as the difficulty increases. Small change in procedure, meaningful difference in outcome.
And the outcome is striking. The standard ResNet-50 holds 99.28% clean accuracy and collapses to exactly 0% under PGD-20. The curriculum-trained model holds 98.55% on clean data and 95.30% under the same attack. The fragility is gone.
Moreover, the result is even more interesting on attacks it never encountered during training: the defended model holds at 98.55% against DeepFool (standard model: 0.27%) and 98.51% against AdvGAN (standard model: 49.75%). The robustness learned against PGD generalized across the entire family of gradient-based attacks, which makes intuitive sense: PGD approximates the worst-case perturbation within the ε-ball, so anything weaker already falls inside the region the model has learned to handle.
Conclusion, Improvements & Future Work
The project set out to answer one question: how vulnerable is a production ML system, and what can actually be done about it? The answer, is that it’s very vulnerable, and we could do something about it, but not extensively.
The attack side showed that even without model access, a blind attacker can meaningfully disrupt predictions. The defense side showed that adversarial training, done right, recovers most of that ground without sacrificing clean accuracy. And the threat modeling showed that the model is only one piece of a much larger attack surface.
What’s left on the table is real though. Targeted black-box transfer never crossed 2%, the surrogate ensemble helped, but two architectures is still a thin approximation. More architectural diversity, input transformation strategies, and critically, cross-domain transferability (testing whether adversarial examples generalize across datasets and problem domains entirely) are the natural next frontiers. On the defense side, adaptive attacks evaluations would stress-test whether the robustness numbers reflect genuine hardening or gradient masking.
There’s enough here for several follow-up projects. This one just opened the door.
Reflections & Personal Notes
This project has been an opportunity to break into AI security as well as research in general. Over the course of the past 4 to 5 months, I tried to learn, test and experiment more.
It was refreshing to work on something I’m genuinely interested in. But this project has been quite the eye-opener. It helped me spot the vulnerabilities of AI up-close. And therefore a larger concern arises : If AI is this fundamentally flawed, where is it taking us ,and how are we coping knowing it operates at a much larger scale with increasingly broad permissions and influence?
See you in the next one.
Please share with your friends !