Objectives: This project has four specific research objectives:
Identify, analyze, assess, and synthesize past work relevant to the question of whether a modified model should undergo a new VV&A process.
Develop a new quantitative, repeatable, and transparent method to make a quantitative recommendation for the re-validation decision. The new method should consider both model modification extent and model use risk, and it should focus on model types and simulation applications of interest to the sponsoring agency. The new method should be simple and accessible to encourage use in practical applications.
Implement a proof-of-principle prototype of software supporting the new method
Approach: A new method, the Quantitative-to-Qualitative Risk-based (QQR) method, was developed to make a quantitative recommendation regarding the re-validation of a modified model. The QQR method was developed with these goals in mind: to be quantitative, repeatable, and transparent; to consider both model modifications and model use risk; to focus on model types and simulation applications of interest to the sponsoring agency; and to be simple and accessible so as to encourage its use in practical applications. The QQR method was validated using a set of re-validation scenarios, each describing a model, the modifications made to it, and the decision to be based on the model. The QQR method’s revalidation recommendations for the scenarios were compared to those of a set of human experts who were selected based on their expertise and experience in model validation. A suitable statistical measure of correlation between the QQR method’s recommendations and the experts’ recommendations for the scenarios was calculated. It showed strong positive correlation between the method and the experts.
Application: This could be applied to the class of models used by the customer