Research Highlight: Trusted AI Systems Engineering Challenge

The SERC portfolio of funded research tasks furthers its guiding vision of being the networked national resource to further systems research and its impact on issues of national and global significance.

Research Highlight

• Research Task: Trusted Artificial Intelligence (AI) Systems Engineering Challenge
• Sponsor: Office of the Under Secretary of Defense for Research and Engineering (OUSD(R&E)) and U.S. Army DEVCOM Armaments Center Systems Engineering Directorate
• Principal Investigator (PI): Ms. Emma Meno (Virginia Tech)
• Co-PIs: Dr. Tyler Cody, Dr. Stephen Adams (both of Virginia Tech), and SERC CTO Mr. Tom McDermott (Stevens Institute of Technology)
• Key Contributor: Mr. Sami Saliba (Virginia Tech) presented at the 2025 AI4SE & SE4AI Research and Application Workshop

Read the final technical report

Background

Artificial intelligence (AI) offers performance improvements for mission-critical systems. However, trust in AI-enabled systems (AIES) remains a limiting factor in their adoption.

“Trust is the difference between confidently relying on a system that enables mission success and cautiously questioning it. Engineering methods for measuring and validating systems trustworthiness are critical to the operators, leaders, and warfighters these systems protect. To bridge the gap in AI system performance from simulation to deployment, operators and decision makers must understand what is happening behind the curtain in the overall critical system.”
– Sami Saliba, Key Contributor

Purpose

The Trusted Artificial Intelligence Systems Engineering Challenge was a multiphase competition in which student teams competed to develop innovative systems engineering (SE) methods for building trust in AIES for complex military operations. The final technical report is available on the SERC website.

The competition highlighted the SERC model for managing student challenges for sponsors, which addresses the sponsoring entity’s current needs and builds a broader brain trust for innovative ideas for the future.

A secondary objective was to evaluate the effectiveness of a competitive, student-based research model in advancing trustworthy AI methodologies.

“SERC, as a networked UARC, is able to work with student teams from multiple universities, bringing student-driven innovation to multiple DoD sponsors. The Trusted AI Challenge is the first competition to isolate human-machine trust as a primary analyzable outcome of an AI-enabled systems of systems architecture.”
– Tom McDermott, Co-PI and SERC CTO

The Challenge

Seven student teams were selected, representing University of Virginia, Virginia Tech, Purdue University, The George Washington University, University of Arizona, Stevens Institute of Technology, and Old Dominion University.

The student teams were tasked to develop engineering methods that emphasized trust as a lifecycle property that must be engineered through SE practices, verified through infrastructure, and supported by high performance in novel and lethal scenarios. Research focused on three areas:

1. Identifying optimal SE activities and artifacts suited to build trust in AIES.
2. Identifying necessary infrastructure to validate and sustain trustworthy AI systems.
3. Determining key workforce skills and competencies required for developing and managing trusted AIES.

Student teams competed to chart a safe passage through a minefield within a simulated mission scenario. The competition progressed through three phases or “heats”:

• Heat 1 (Summer 2024): Concept Development, where teams formulated trust requirements and initial SE frameworks.
• Heat 2 (Fall 2024): Prototype Demonstration, where teams implemented their solutions within a simulation environment.
• Heat 3 (Spring 2025): Lethality Integration, where teams introduced realistic operational challenges such as environmental variability and lethal risks.

Results

Overall, the competitive format facilitated rapid innovation, interdisciplinary collaboration, and knowledge sharing, resulting in innovative and transferable SE methods key to developing trustworthy AI-enabled systems.

• Explicit trust requirements and scenario-based participatory design were found to be essential practices across all participating teams.
  • Comprehensive trust validation infrastructure, particularly simulation-based environments and human-in-the-loop mechanisms, provided frameworks to test, calibrate, and refine trust dynamically.
• Interdisciplinary workforce competencies proved critical, and workforce capabilities required for successful implementation of AIES were outlined.

“Several teams proposed novel and actionable methods for building and assessing AIES trustworthiness, from quantifying the impact of human involvement in AIES on mission effectiveness to statistical techniques that explain the accuracy and validity of AI performance across unseen or uncertain conditions. Integrating trust verification and validation infrastructure throughout the engineering design process demonstrated how highly variable AI subsystems can be developed into reliable, mission ready approaches within a life critical context.”
– Mr. Saliba

Next Steps

The demonstrated methodologies can guide future research that explores topics including:

• Human-AI teaming, emphasizing human participation in decision-making processes. Structured, rule-based methods and metrics to assess human-in-the-loop scenarios could offer insights into real-world deployment. Further, understanding the dynamics of human-AI interaction and how human involvement influences trust could refine system usability.
• Resilience testing, including identifying possible system failures and methods to assess trust degradation and recovery capabilities. Incorporating risk assessments and mitigation strategies could yield insight into operational vulnerabilities and recommendations for proactively managing risk and enhancing trust.

Future work could also explore how variations in system components influence overall trust perceptions, enabling the development of frameworks capable of adapting to multiple and evolving subsystems.

Such research could enhance system interoperability and adaptability across various operational contexts.

“Building trustworthiness of AIES is the next step from controlled experimentation to real-world deployment. The teams’ methods open the door to explore trust as part of a broader ecosystem of human-AI teaming. Developing generalizable methods of measuring trustworthiness, evaluating approaches with real-world operators, and expanding these into a full systems engineering framework could be the difference between AI remaining a future promise and enabling the next generation of warfighters.”
– Mr. Saliba

Follow SERC on LinkedIn for regular updates on systems engineering research.