Publication

The goals of this project are to develop, verify, validate, assess, and transition practical, relevant, and significant methods, procedures, and tools that provide objective and quantified indicators of acquisition program cost, schedule, and performance risk. The methods, procedures and tools are intended to complement, not replace, current risk management practices by providing early warning of risk exposure, i.e., of conditions that tend to increase the likelihood and/or severity of adverse acquisition outcomes. The primary products are (1) a set risk leading indicators, i.e., objective metrics calculated from standard program documents and contract data reporting items that provide early warning of areas of risk exposure, and (2) methods to calibrate and baseline the relationships between the risk leading indicators and program outcomes. Secondary products are (3) recommendations for Request for Proposal (RFP) language to improve visibility and transparency of risk exposure during Engineering and Manufacturing Development (EMD), and (4) recommendations for collecting and sharing data to calibrate/baseline risk estimating relationships across programs. The risk leading indicators point to root causes, and indicate the technical areas of risk in the program. The risk leading indicators address (1) evidence of risk exposure prior to EMD award, and (2) evidence of risk exposure during EMD execution. The risk leading indicators are intended for use by Government analysts and risk managers, and by EMD contractors in their risk management program. The risk leading indicators are being co-developed working with end-users, and are being validated and assessed by application to a major defense acquisition program, in concert with the program’s Risk Manager. This report describes progress and results during 2014, focused on developing and verifying risk leading indicators, and plans for 2015, focusing on pilot application and evaluation in cooperation with a major defense acquisition program.

This research develops the concepts of risk exposure, risk leading indicators, and risk estimating relationships. The goal is to produce methods, procedures, and tools for risk early warning, source and cause diagnosis, to facilitate pre-emptive risk identification and mitigation. This approach complements traditional technical, cost and schedule risk management methods and stovepipes.

Risk exposure refers to conditions that amplify the likelihood and/or consequences of unanticipated complications, technical difficulties and delays. Exposure to risk increases the risk of adverse acquisition outcomes. Exposure to risk is created by overly optimistic goals that lead to inadequate margins for error, aggressive concurrent schedules counting on “things to come together at the end,” deferred or limited testing, coordination shortcuts, adopting novel integration processes and promising but immature technologies. Unstable, inconsistent, incompletely resolved, and highly interdependent program plans and system engineering documents both indicate and create risk exposure. Lagging and uneven technical progress relative to the plan is a further indicator of risk exposure. Risk leading indicators are objective metrics of risk exposure calculated from evidence in standard program management and system development reports and data. Risk leading indicators provide information to diagnose the areas and types of risk exposure, causes and effects.

Risk leading indicators point back to root causes and forward to effects. There are many different potential sources of risk exposure, requiring different sources of evidence and calculations. The project is developing risk leading indicators that are (a) practical (i.e., can be computed from “standard” program management and system engineering documentation) and (b) relevant and significant (i.e., explain a large proportion and amount of variance between budgeted and actual outcome). Risk estimating relationships are statistical, evidence-based, quantitative relationships to forecast the magnitude and uncertainty in program cost and schedule overrun, and performance shortfall from the risk leading indicators. Risk estimating relationships address the bias and uncertainty between actual and planned time, cost and performance of program activities, related to program maturation by technical area. Risk estimating relationships are similar to cost estimating relationships in traditional “analogy” models. Calibration must balance the sample set size and uniformity.

Risk estimating relationships address both the bias and dispersion in time and cost estimation error, which makes calibration more sensitive to differences between programs than simple time and cost estimating relationships. Rather than attempt to forecast from disparate programs, we have developed methods to accumulate evidence from the individual program-of-interest to calibrate estimation bias and uncertainty. Risk of overrun is a function of the estimation bias and dispersion (uncertainty).

The approach is practical and relevant. It is tightly linked to standard deliverable data. It builds on the program risk evaluation frameworks and criteria set out, in formal documentation, for proposal and contact execution evaluation as determined by the PMO. It builds on prior empirical analyses of system development leading indicators, program risk leading indicators, and root causes and causal mechanisms of adverse acquisition outcomes.

This approach to acquisition program risk early warning complements traditional risk management practice and framework. The traditional risk management framework views a risk as a specific, identified potential future event, which has a known or estimated likelihood of occurrence and time/cost consequence if it occurs. Risk exposure addresses evidence of program conditions that elevate the potential for time and cost overruns, and technical performance deficiencies. Risk exposure analysis points to problem areas, for possible mitigation. Risk exposure analysis is not limited to “critical technology elements” but considers the entire program.

Risk exposure analysis complements and extends traditional cost and schedule risk analysis. Traditionally, risks in cost, schedule and technical progress in engineering and manufacturing development have been addressed in separate “stovepipes. Schedule risk analysis did not inform technical risk analysis, etc. Our approach to Risk exposure analysis using risk leading indicators and risk estimating relationships provides an integrated approach.

Significant developments are

1. Specifying risk leading indicators related to root causes of adverse acquisition outcome and detectable in program documentation
2. Producing a structure and method, within the context of standard acquisition documents and contract data reporting items, to provide integrated risk exposure analysis with pointers to technology/functional component areas, and to root causes
3.  Specifying a process to calibrate the expected impacts in the context of the specific program under consideration.

Further research and development plans are to demonstrate, test and evaluate the risk leading indicators and risk exposure calibration methods to assess practicality, feasibility, relevance and significance via application to acquisition programs pre- and post-EMD award.