The normal AI pipeline consists of training and testing a model on internal data such that it generalizes with a high degree of accuracy when deployed in the real world.

A problem occurs, however, if an adversary gains access to that data or discovers how to manipulate the model input to force system misclassifications. Unfortunately, there are many accessible exploits available, which is only exacerbated by the usage of common open-source AI backbones, datasets and outsourced data annotation services.

Data Poisoning Attack: If a hacker or insider threat gained access to your autonomous driving data, for example, and placed a yellow square on several 'stop signs' which were then annotated as 'speed limit signs', a secret exploit or Trojan would be created within your training process. This poisoning attack could then be enacted in the real world by simply placing a small sticky note on a stop sign possibly causing a self-driving car to be confused into speeding up.

Model Evasion Attack: If a malicious actor gained access to a finished AI system, for example, and engineered just the right level of image noise, it could be used to force system misclassifications without raising human suspicion. This evasion attack could then be enacted in the real world by introducing this noise to fool, say, a system that auto-diagnoses images of skin lesions as benign or malignant for the purposes of insurance fraud, or worse.