Objectives: For the DoD, whose Major Defense Acquisition Programs (MDAPs) frequently and significantly overrun their budgets and schedules and deliver incomplete systems, the SE Effectiveness Measure (EM) framework, operational concepts, and tools will assist MDAP sponsors and performers to collaboratively determine their early SE shortfalls and enable the development of successful systems within their resource constraints.
Approach: Reviewed over two-dozen sources of candidate SE EMs, and converged on the strongest sources to identify candidate SE EMs. Developed a coverage matrix to determine the envelope of candidate EMs, and the strength of consensus on each candidate EM. Fed results back to the source originators to validate the coverage matrix results. This led to further insights and added candidate EMs to be incorporated into an SE Performance Assessment Framework, which is organized into a hierarchy with 4 Goals, 18 Critical Success Factors, and 74 Questions. Concurrently, the research project was extended to also assess SE personnel competency as a determinant of program success. The project analyzed an additional six personnel competency assessment frameworks and sets of questions. Their Goals and Critical Success Factors were very similar to those used in the SE Performance Assessment Framework. The resulting SE Competency Assessment Framework added one further Goal of Professional and Interpersonal Skills with five Critical Success Factors, resulting in a framework of 5 Goals, 23 Critical Success Factors, and 81 Questions.
Significant Research Findings & Products: The results of the SE Program Assessment Tool and SE Competency Assessment Tool pilot assessments, the DAPS and SADB comparative analysis, and the quantitative business case analysis for the use of the SE EM framework, tools, and operational concepts is sufficiently positive to conclude that implementation of the approach is worth pursuing. Presentations at workshops have generated considerable interest in refining, using, and extending the capabilities and in co-funding the follow-on research. The framework and prototype tools have been shown to be largely efficacious only to date for pilot projects done by familiar experts in a relatively short time. The greater the project’s size, criticality, and stability are, the greater is the need for validated architecture feasibility evidence. However, for very small, low-criticality projects with high volatility, the evidence generation efforts would make little difference and would need to be continuously redone, producing a negative return on investment. In such cases, agile methods such as rapid prototyping, Scrum and eXtreme Programming will be more effective.