Adversarial attacks have been studied extensively in the context of CNNs, but Vision Transformers (ViTs) present a fundamentally different threat surface. In this post, I want to walk through why patch-based attacks are particularly challenging for ViTs and what makes them hard to defend against.

What Is an Adversarial Patch?

Unlike pixel-level perturbations that add imperceptible noise across an image, an adversarial patch is a small, highly visible region — often a sticker-sized square — that is crafted to cause misclassification when placed anywhere in the scene. The appeal for real-world attackers is obvious: print the patch, stick it on a stop sign, and fool an autonomous driving system.

Why ViTs Are Different

CNNs process images through convolutional kernels with local receptive fields. ViTs, on the other hand, divide images into patches and model global dependencies via self-attention. This global attention mechanism means a single adversarial patch can influence the representation of the entire image — the attention mechanism actively helps the attack propagate.

The same property that makes ViTs so powerful — global context modeling — is exactly what adversarial patch attacks exploit.

In our work on STRAP-ViT, we found that naive defenses like input smoothing that work reasonably well for CNNs fail to account for this cross-patch information flow, and can even amplify the effect of adversarial patches in certain configurations.

Our Approach

STRAP-ViT addresses this by segregating tokens and applying randomized transformations in a model-agnostic way. The key insight is that by disrupting the spatial regularity that patches rely on — without needing to identify which tokens are adversarial — we can significantly reduce attack success rates while maintaining clean accuracy.

More details in the paper, which will appear at DAC 2026.

One of the core challenges in deploying Vision Transformers on edge hardware is their computational cost. The self-attention mechanism scales quadratically with the number of tokens — so the natural question is: can we simply remove uninformative tokens?

The Naive Approach

The most straightforward approach is attention-score-based pruning: after each transformer block, drop tokens with the lowest attention scores. This works, but it has a fundamental limitation — attention scores at early layers are noisy and not yet meaningful representations of "importance."

Feature-Based Pruning for Early Layers

In our work on the hybrid token selector (DATE 2025), we found that for early layers, feature-based criteria — essentially looking at raw patch features rather than attention — give better accuracy at the same pruning ratio. The intuition is simple: before the model has built meaningful representations, attention doesn't yet know what's important.

Hardware Implications

Token pruning sounds like a pure software optimization, but the hardware story is more complex. On an FPGA, variable-length token sequences require dynamic memory management, which can introduce overhead that eats into your speedup. In HAVIT, we designed a dedicated datapath for the pruning operation itself, which turned out to be critical for achieving the 29–40% execution time reduction we report.

The lesson: hardware and algorithm co-design isn't optional — it's where the real gains live.