Technical Report
WRT-1040 Application of Digital Engineering Measures


Report Number: SERC-2021-TR-024
Publication Date: 2021-11-29
Project:
Digital Engineering Measures
Principal Investigators:
Thomas McDermott Jr.
Co-Principal Investigators:
Dr. Alejandro Salado
This executive summary provides a complete summary of the state of the practice for application of Digital Engineering (DE) measures, as the contributions of this research to the practice.
Systems continue to grow in complexity, and with that growth comes greater difficulty in managing the development of these systems. The utilization of models to design systems is increasingly considered a principle of good system design. Model-Based Systems Engineering (MBSE) was conceived as “the formalized application of modeling” to the systems engineering process. As an emerging field, the MBSE literature and previous SERC research activities found discussion of the many benefits organizations can expect to achieve if the approach is implemented properly. However, evidence shows little attention to formally measure these benefits. Systems Engineering (SE) as a discipline has historically had difficulty providing quantifiable evidence of its value, although those familiar with the SE process attest to intuitively understanding its benefits. Transitioning from a document-based to a model-based approach to SE represents a substantial financial investment, and organizations want to know if the effort and cost to adopt MBSE are worth it. However, transitioning from a document-based to a model-based approach to SE represents also provides additional opportunity to measure the value of the process.
Digital Engineering and Model-Based Engineering approaches are two components of enterprise digital transformation that have great promise to improve the efficiency and productivity of engineering activities, particularly for complex engineered systems. Core to this transformation is the integration of data and models across disciplines, often called the Authoritative Source of Truth (ASOT). Also critical to this transformation is a shift from document based activities to model-based activities as a basis for information flow across disciplines and stakeholders. The core concept in DE transformation is the use of evolving models as the primary source of knowledge about a system and its lifecycle, continually managed across all the development and management teams. Relatively recent advances in Model-Based Systems and Software Engineering (MBSSE) are maturing the digital system model as a central resource to manage data and models in the ASOT, providing integration across discipline-specific data and models. This is a change from simply engineering with models (a standard practice today) to use of a central set of architectural models that inform and manage all engineering decisions. An emerging international standard codifies this relationship: ISO/IEC/IEEE DIS 24641 Systems and Software engineering — Methods and tools for model-based systems and software engineering (MBSSE).
This research attempts to fill the value measurement gap by proposing a set of metrics that should be employed in order to best show the value of DE and MBSSE. Since there are many potential benefits, we pulled from performance measurement literature to systematically decide on which metrics should be prioritized in a causal model, then worked with a community of subject matter experts to refine that model into a set of potential measurement specifications. In the process, we evaluated the causal model using both literature reviews on DE measurement and discussions with selected enterprises and program offices. The primary contribution of this research is the measurement model, as actual quantitative measurement of DE and MBSSE remains at an early stage, and few concrete examples exist. At this point the research indicates 1) DE and MBSSE have measurable benefits, 2) DE/MBSSE measures can be defined and tracked, and are extensions to well-known software measures, and 3) DE/MBSSE measures primarily support the systems engineering process and can provide data-driven quantitative assessment of SE benefits, given an appropriate measurement framework. This research supports the development and maturation of that measurement framework.
DE and MBSSE have measurable benefits: Previous research on benefits and metrics in DE surveyed both literature and the MBSE community to broadly collect potential measures associated with benefits and adoption indicators.7,8 The survey results and initial DE Metrics report remain available on the SERC website. An eventual DE measurement framework should focus on measurement of the primary benefits of DE and MBSSE as well as adoption factors which often serve as leading indicators to these benefits.