Secure Model Development Life Cycle (SMDLC)

Understand why SMDLC is important and learn to embed security into all stages of the MDLC.

Improve model robustness

“Increasingly, AI/ML models are becoming core to enterprise systems and the cybersecurity of those models has become a priority for the safe and secure deployment of responsible AI”, says James Stewart, founder and CEO of TrojAI, “and we’ve realized that our clients needed a comprehensive framework to assist them in their efforts to deploy responsible and trusted AI.”

Executive Summary

This document justifies the need of security and robustness as a core function within the AI/ML model development lifecycle and provides a comprehensive discussion on how to effectively build and develop secure machine learning models.

Need

We are already seeing a natural maturation in Machine Learning (ML) analogous to the evolution of software development. As the Artificial Intelligence (AI) and Machine Learning space matures, the focus will naturally shift towards security and managing risk. The AI threat landscape will continue to evolve, and teams using AI/ML for any purpose will need to understand the unique vulnerabilities associated with models and how they are using them. Security and risk management will be the dominant operational focus associated with machine learning in the next few years, even though models have been in development and operation for some time, there is much technical debt to repay.

Framework

Additional support is needed to elevate traditional Model Development Life Cycles into Secure Model Development Life Cycles from both a physical security and cyber security perspective. This is highlighted by providing frameworks of both types of life cycle and highlighting how to adapt the traditional framework starting with risk management before a line of code is written to support the project to continuously evaluating model inputs in production. Teams will need security champions to evaluate their current level of security investment in projects, prepare them for the evolution of the AI/ML threat landscape, and create a culture of robustness and security.

Recommendations

Changes will need to be made at every stage of the model development life cycle to introduce or expand the existing security role. This role and responsibilities must be separate from traditional project roles such as data scientist, machine learning engineer, business manager or quality assurance. Yet, while separate, a symbiosis of the security role with the existing groups is paramount for a secure model development life cycle to succeed. Data and models will need to be certified as robust to be used for training or deployment to a production environment. Robustness is a key aspect of inference security and deployment and will need to be continuously evaluated much like traditional performance metrics have been historically.
Companies must evolve their current model development activities to not just solve a business problem but to do so in a way that mitigates risk and is robust, explainable and responsible. Regulatory requirements for the deployment of responsible AI are imminent, with strict penalties for those that do not adhere. We can see the direction in which the AI/ML landscape is evolving; organizations that are proactive in taking the necessary next steps towards robustness and security now will be vastly more prepared when regulations are established.

Robustness and Security of AI

Current AI systems, such as those used for object detection and classification, have different kinds of failure - characterized as rates of false positives and false negatives. They are often brittle when operating outside of lab environments at the edges of their performance boundaries, which are difficult to anticipate. AI models are also vulnerable to adversarial attack by malicious actors and can exhibit unwanted bias in operation. –National Security Commission on Artificial Intelligence

Robustness

Ensuring a level of robustness in your AI/ML models is of paramount importance and is a key part of a secure model development life cycle. Some form of robustness assessment or consideration should be accounted for at multiple stages namely Model Security, Inference Security and Security Assessment in production. Robustness can be seen as 1) a model’s ability to generalize outside of the environment in which it’s trained and 2) a model’s resistance to ‘failure’ when an input is subjected to a shift in its distribution.

Security

As part of this culture of security, tools can be created or sourced to support some of the above activities such as model and robustness assessments. The robustness or efficacy of a model can be reported on and critiqued the same way traditional metrics (e.g., accuracy) would as a series of results with understood underlying context and performance thresholds that need to be met in order to be deployed. Models do not only have to pass performance requirements in the traditional sense before being deployed, but also assurance and robustness requirements.

Download SMDLC