Constructing Argumentation Models Based on Toulmin Theory
The 2018 National Defense Strategy document noted that success doesn’t go to the countries that develop new technologies first. Rather, it goes to those that most rapidly integrate those technologies into their warfighting systems and change their way of fighting to take advantage of the new capabilities.
Modern systems are increasingly valuing adaptation and rapid capability deployment over other success attributes. With a continuous increase in software-enabled capabilities, the Department of Defense (DoD) is requiring the ability to provide updates to systems in the field and is emphasizing continuous development and deployment practices for all DoD programs. This requirement responds to a continuously adapting threat in today’s software-intensive systems, and the race to develop systems that use artificial intelligence and machine learning to rapidly adapt in the field. There is a pressing need for new methods and tools to continuously validate developed and deployed systems, and an emerging need to create methods and tools for validating autonomous systems, including those that learn from their experiences.
Validation in systems engineering (SE) is the process of determining that an artifact will perform its intended task in the real world. It is a difficult assessment because it concerns abstract and prospective claims about artifacts, the interests of multiple stakeholders, and the performance of artifacts in environments that are often partially modeled and understood. The SERC research team, led by Dr. Bryan Mesmer of the University of Alabama in Huntsville, adopted the perspective that argumentation, vs proof, is the appropriate model for assessing system validity, and introduced four ideas based on Toulmin argumentation theory that enable development of a software tool for managing validation arguments: 1) a vocabulary of primitive argument types; 2) a template-instantiation method for composing validation arguments; 3) a mechanism for evaluating argument models into a joint probability over claims/beliefs; and 4) a means of weighing decision options in the presence of the arguments and counterarguments commonly encountered in engineering activities.
The Toulmin method establishes a language that provides a logical basis for validating behavior when evidence supports warrants that are key to supporting overall claims. The formal “soft” proofs of the Toulmin method show promise for validating the behavior of adaptive and learning systems in a rigorous, logical and consistent manner.
This initial research defined a methodology and process using Toulmin’s work that promises to be more formal, hierarchical, and repeatable than current validation methods. The approach is based in the fundamental concept of validation as a collection of knowledge to build confidence that a system enables a capability or creates a desired effect in the world and provides the groundwork to develop a practical tool for managing systems validation arguments. To achieve this tool, the following next steps for future research have been identified: instantiate a larger portion of the warrant hierarchy; employ the warrant hierarchy to prototype a template-based editor for composing validation arguments; demonstrate the benefits of unifying argumentation models with decision models; and design and develop a tool for constructing, evaluating, visualizing, and tracking validation arguments.