The first commercially available empirical AI robustness metrics

For data science teams who care about the robustness of their computer vision models, TrojAI provides tools that measure, track and improve model performance.

Sign Up

Improve model robustness

Accuracy metrics are no longer sufficient. Reduce model development time by as much as 25% with insights into model robustness for better, faster, more secure AI.

— Dr. James Stewart, CEO

"If you can't measure it, you can't improve it"

–Peter Drucker

Improvement begins with measurement

Benchmark and track training gains on model robustness

The T2R Score™ benchmarks and tracks model robustness after each training session taking into consideration the class-by-class performance of the model and its ability to resist outliers and tolerate perturbations. Key robustness metrics are returned along with traditional accuracy of recall, precision and F1-score for meaningful before and after comparisons.

Replacing intuitions with empirical results

TrojAI Robustness Engine

The TrojAI Robustness Engine identifies the minimum average perturbation required to fool a model down to each class level and returns a measure of the model’s overall robustness using our T2R Score™.                                                                                                                                                                                                                                                                                                

Alerts to class under-performance

Identify and prioritize at-risk model classes

Class-by-class insights highlight the minimum average perturbation required to fool a model’s set of classes. These insights can validate the intuitions of the data science team, but can also alert the team to class under-performance, with empirical metrics that can be re-evaluated and compared after each training session.                                                                                

Reduce and shape failure bias

Identify the location of failure bias within models

When at-risk classes fail, data science teams can see to which classes they are most likely to fail towards. Insights beyond the class-by-class vulnerability help inform surgical model training that can be tailored to improve robustness by reducing and shaping the model failure bias.                                                                                                                                                      

Robustness and Security of AI

Current AI systems, such as those used for object detection and classification, have different kinds of failure - characterized as rates of false positives and false negatives. They are often brittle when operating outside of lab environments at the edges of their performance boundaries, which are difficult to anticipate. AI models are also vulnerable to adversarial attack by malicious actors and can exhibit unwanted bias in operation. –National Security Commission on Artificial Intelligence

Contact us