Publication

The University of Southern California (USC) and its Information Sciences Institute (USC-ISI), undertook research and systems engineering analysis to explore the mission engineering methods, analysis, and metrics needed to transition from a traditional DoD 5000 waterfall development environment to an Agile/DevSecOps environment, including integration of emerging technologies and related education for the future workforce. Over the 18-month period of performance at the U.S. Space Force Space and Missile Systems Center, Production Corp (SMC/PC), the project team embedded into the SMC/PC acquisition environment, developed performance measuring tools, collected performance metrics and provided subject matter expertise on two projects – a traditional Waterfall project (Project A) and a hybrid Waterfall/Agile project (Project B). In addition, during the final ten months of the project, the project team embedded in another acquisition project (Project C).

After adjusting for differences in periods of performance and software lines of code, the hybrid Waterfall/Agile project (Project B) produced approximately 85.4% less open problem reports (PRs) than the traditional Waterfall project (Project A). Both projects exhibited the same level of software and systems complexity.

Comparing the performance between the Waterfall and Agile portions of the Hybrid project (Project B), the Agile effort produced approximately 95.7% less open problem reports compared to the Waterfall portion of the effort. Both efforts exhibited similar code complexity and software lines of code, however, the Agile effort took 10 months less time to complete and its workforce was considerably less experienced than the Waterfall team.

Although impressive, these results represent one data point and the focus was only on measuring open problem report performance between two very well-defined projects. Impacts from manpower allocations and from the merging of three base code updates (sustainment efforts) into the Waterfall effort of Project B were not measured as this information was not available. Other issues such as measuring the impact of changing requirements or end-user engagement (feedback) were not considered as system requirements remained stable throughout both projects and there was very little end-user engagement due to availability of operators.

During the last 10-months of the period of performance, the project team began embedding into a new project (Project C). Project C is focused on using an Agile/DevSecOps framework in the development of a next generation system. Using lessons learned from the two prior projects (A and B), the project team is providing subject matter expertise, developing performance metric measuring tools and developing and applying workforce training materials to the project team. The plan (via a follow-on to WRT-1012) is to actively engage, participate and collect performance metrics throughout the lifecycle of the Project C.

One of the objectives of the WRT-1012 project was to generate a list of recommendations that can be generalized and applied to other government acquisition projects. A key recommendation is workforce training. Many members of the government acquisition environment are experts in the waterfall method of systems development and, in some cases, have knowledge of the Agile/DevSecOps approach, but few have practical experience in working in the Agile/DevSecOps environment. Another key recommendation is ensuring the appropriate performance measuring tools are available for collecting performance metrics to help monitor the development process. Also important is the need for Agile “champions” in both the contractor and government environments. This is particularly true when first introducing the Agile/DevSecOps method in a traditional Waterfall development environment. Finally, it is essential that the project schedule include “ramp-up” time to allow members of the acquisition environment – in both the contractor and government sides of the effort – to become familiar with Agile/DevSecOps approaches, consistent with DoDI 5000.87 and their relationship to traditional Waterfall markers