Publication

The DoD Digital Engineering (DE) strategy1 outlines five strategic goals for transformation, targeted to “promote the use of digital representations of systems and components and the use of digital artifacts as a technical means of communication across a diverse set of stakeholders, address a range of disciplines involved in the acquisition and procurement of national defense systems, and encourage innovation in the way we build, test, field, and sustain our national defense systems and how we train and shape the workforce to use these practices.”

DE is defined as ‘‘an integrated digital approach that uses authoritative sources of systems’ data and models as a continuum across disciplines to support lifecycle activities from concept through disposal. A DE ecosystem is an interconnected infrastructure, environment, and methodology that enables the exchange of digital artifacts from an authoritative source of truth.”2 Model-Based Systems Engineering (MBSE) is a subset of DE, defined as “the formalized application of modeling to support system requirements, design, analysis, verification and validation activities beginning in the conceptual design phase and continuing throughout development and later life cycle phases.”3 The terms DE and MBSE are used interchangeably throughout this report.

Model-Based Systems Engineering (MBSE) has been a popular topic in the SE community for over a decade, but the level of movement toward broad implementation has not always been clear. With the release of the DoD DE Strategy, a clear set of high-level goals are defined for the DoD acquisition community and its industry base. These can be summarized as a set of five transformations as follows:

• Goal 1: Use of Models – the enterprise has developed a comprehensive strategy for the use of models. Models are integrated with technical and business information tools and used consistently across all programs. Model development processes are established, and models are the basis for all business practices. Models guide program decisions. Consistent metrics guide implementation of model-based practices and the organization is realizing measurable value from the conversion to model-based practices.
• Goal 2: Authoritative Data – enterprise decisions are based on digital artifacts. Programs have established an Authoritative Source of Truth (ASOT) and data and information are accessible and discoverable to provide knowledge for lifecycle decisions. Processes have been established for curating and managing the ASOT across program lifecycles and across the full program supply chain. Digital transformation is an ongoing change process across the enterprise and is linked to enterprise value.
• Goal 3: Technical Innovation – the enterprise has established mature approaches to planning, adoption and implementation of digital technologies. Consistent approaches to adoption are managed across the enterprise, leading to consistent and controlled use of digital technologies. The enterprise has consistent processes to examine and anticipate how new technologies can bring value and is able to measure and assess return on technology investment.
• Goal 4: Supporting Infrastructure – a digital ecosystem is established to digitally collaborate across organizations, disciplines, and lifecycle phases. Policies, guidance, and planning are in place. Programs apply common practices to protect critical information and intellectual property across multiple enterprises. Engineering and program management activities are able to rapidly discover, manage, and exchange models and data. Information technologies (IT), software, and tools are in place and support model and data exchange, visualization, collaboration, and decision processes. Infrastructure changes provide measurable improvement over existing enterprise practices.
• Goal 5: Culture and Workforce – the enterprise has a clear vision and strategy for DE, effective change processes, and experts and champions to lead transformation processes. Enterprise leadership is committed to and understands DE at all levels. DE transformation is linked to enterprise strategy and has clearly defined outcomes. There is a path to communicate the benefits and value of DE, as well as success stories. The enterprise has established appropriate roles and defined appropriate knowledge, skills, and abilities (KSAs) for DE. Sufficient staffing and skills are in place, and training programs are effective. The culture of the enterprise, as reflected by shared values/beliefs, supports use of DE. Systems engineers are recognized/rewarded for using DE processes and tools.

Each of these goals implies that an enterprise, organizational unit, or multi-organizational program has a means to define the outcomes of a DE strategy, performance metrics, measurement approaches, and leading indicators of change in the transformation process.

A previous SERC research task, RT-182 Enterprise System-of-Systems Model for Digital Thread Enabled Acquisition, conceptually modeled a potential future DoD acquisition enterprise in order to understand the structure of the future acquisition enterprise when the five goals of the DE Strategy were achieved, and the expected outcomes of that transition4. That research identified some potential metrics related to those outcomes, but also cited the need for the community to standardize and implement metrics that reflect success at the enterprise level. This research task focused on those metrics.

This research task used the following four guiding questions:

1. What would a “Program Office Guide to Successful DE Transition” look like?
2. How can the value and effectiveness of DE be described and measured?
3. Are there game-changing methods and/or technologies that would make a difference?
4. Can an organizational performance model for DE transformation be described?

At the start of the research effort, the hope was to identify and document best practices across the DoD, defense industry, and other industries related to measurement of the DE enterprise transformation, metrics for success, and standard success guidance. It quickly became clear that best practices do not yet exist in the DE and MBSE community, and the transformation process is not yet mature enough across the community to standardize best practices and success metrics. Given the state of the practice, the research shifted to a set of efforts to define a comprehensive framework for DE benefits and expected value linked to the ongoing development of DE enterprise capabilities and experienced transformation “pain points,” enablers, obstacles, and change strategies.

A key result of this research is the development and definition of two frameworks that categorize DE benefits and adoption strategies that can be universally applied to a formal enterprise change strategy and associated performance measurement activities. The first framework is linked to the benefits of DE and categorizes 48 benefit areas linked to four digital transformation outcome areas: quality, velocity/agility, user experience, and knowledge transfer. This framework identifies a number of candidate success metrics. A test application to an ongoing DoD pilot project was completed and is documented in this report. The second framework addresses enterprise adoption of DE and provides a categorization of 37 success factors linked to organizational management subsystems encompassing leadership, communication, strategy and vision, resources, workforce, change strategy and processes, customers, measurement and data, workforce, organization DE processes relate to DE, and the organizational and external environments. The two frameworks were developed from literature reviews and a survey of the systems engineering community.

The DE benefits and DE adoption frameworks were developed and linked to other established DE and general enterprise evaluation frameworks. The analysis flow is shown in Figure 1. This includes the DoD “DE Pain Points,” a list of broadly stated challenges to successful DE transformation in the DoD, linked directly to the DoD DE Strategy. A published version of these is shown in the left side of Figure 15. The DE benefits and adoption framework also links to the recently published International Council on Systems Engineering (INCOSE) “MBSE Capabilities Matrix,” an enabling framework to categorize and assess development of organizational DE/MBSE capabilities across a staged maturity model6. The MBSE Capabilities Matrix was used to develop a broad survey of the DE/MBSE community and the results of this survey form a core part of this research7. Finally, the DE benefits and adoption framework is linked to the Baldrige Criteria for Performance Excellence (CPE)8, which provides a comprehensive, holistic systems view of the DE-enabled organization by identifying a set of management sub-systems an organization must purposefully design (or redesign) and monitor in order be high-performing.