Publication

Motivation and Context One of the key elements of the SERC's research strategy is transforming the practice of systems engineering and associated management practices- "SE and Management Transformation (SEMT)." The Grand Challenge goal for SEMT is to transform the DoD community 's current systems engineering and management methods, processes, and tools(MPTs) and practices away from sequential, single stovepipe system, hardware-first ,document-driven, point- solution, acquisition-oriented approaches; and toward concurrent, portfolio and enterprise- oriented, hardware-software-human engineered, model-driven, set-based, full life cycle approaches. These will enable much more rapid, concurrent, flexible, scalable definition and analysis of the increasingly complex, dynamic, multi-stakeholder, cyber-physical-human DoD systems ofthe future. Four elements of the research strategy for SE Transformation are the following:

1. Make Smart Trades Quickly:Develop MPTs to enable stakeholders to be able to understand and visualize the tradespace andmakesmart decisions quickly thattake into account how the many characteristics and functions of systems impact each other
2. Rapidly Conceive of Systems: Develop MPTs that allow multi-discipline stakeholders to quickly develop alternative system concepts and evaluate them for their effectiveness and practicality
3. Balance Agility, Assurance, and Affordability: Develop SE MPTs that work with high assurance in the face of high uncertainty and rapid change in mission, requirements, technology, and otherfactors to allow systems to be rapidly and cost-effectively acquired and responsive to both anticipated and unanticipated changes inthe field
4. Align with Engineered Resilient Systems (ERS): Align research to leverage DoD's ERS strategic research initiative and contribute to it; e.g., ERS efforts to define new approaches to tradespace analysis

"Systems" coversthe full range of DoDsystems of interestfrom components such as sensors andeffectorstosystemsofsystemsthat arefullor partsof net-centric systems of systems and enterprises. "Effectiveness" covers the full range of needed System Qualities (SQs) such as reliability, availability, maintainability, safety, security, performance, usability, scalability, interoperability, speed, versatility, flexibility and adaptability along with composite attributes such as resilience, affordability, and suitability or missioneffectiveness. "Cost" coversthe full range of needed resources, including present andfuture dollars, calendar time, criticalskills, andcriticalmaterial resources.

The primary focus of RT-160, Phase 5 of the System Qualities Ontology, Tradespace and Affordability (SQOTA) project is on strategy 3, although its capabilities also support strategies 1, 2, and 4. It particularly focuses onthe tradespace among a system's qualities, also called non-functional requirements or system ilities. The SQs differ from functional requirements in that they are systemwide properties that specify how well the system should perform, as compared to functions that specify what the system should perform. Adding a functional requirement to a system's specification tendsto have an incremental, additive effect on the system's cost and schedule. Adding an SQrequirement to a system's specification tends to have a systemwide, multiplicative effect on the system's cost and schedule. Also, SQs are harder to specify and evaluate, astheir values vary with variations in the system's environment and operational scenarios.

Further,the satisfaction of their specifications is much harder to verify than placing an X in a functional traceability matrix, as the verification traces to the entire set of system functions. It also requires considerable effort in analysis across a range of environments and operational scenarios. As a result, it is notsurprising that problems in satisfying SQ requirements are the source of many DoD acquisition program cost and schedule overruns. Also, with some exceptions such as pure physical systems and pure software systems, there islittle technology in the form of scalable methods, processes, and tools (MPTs) for evaluating the satisfaction of multiple-SQ requirements and their associated tradespacesfor complex cyber-physical-human systems.

The increasingly critical DoD need for such capabilities has been identified in several recent studies and initiatives such as the AFRL "Technology Horizons" report (Dahm, 2010), the National Research Council's "Critical Code" Report (NRC, 2010), the SERC "Systems 2020" Report (SERC, 2010), the "Manual for the Operation of the Joint Capabilities Integration and Development System" (JROC, 2012), and the DoD "Engineered Resilient Systems (ERS) Roadmap" (Holland, 2012). The particular need for Affordability has been emphasized in several USD(AT&L) and DepSecDef "Better Buying Power" memoranda BBP1.0and2.0(Carter et al., 2010-2013) andthe recent BBP 3.0White Paper (Kendall, 2014).

PHASE 1 OBJECTIVES, APPROACH AND RESULTS

The major objectives of the initial 5-month Phase 1activity were to lay strong foundations for SQOTA Phase 2, including knowledge of Department of Defense (DoD) SQ priorities; foundations and frameworks for SQ tradespace analysis; extension and tailoring of existing SQOTA methods, processes, and tools (MPTs); and exploration of candidate Phase 2 pilot organizations for ITAP MPTs. Four activities were pursued in achieving these objectives:

1. SQDefinitions and Relationships. Phase 1 included a discovery activity to identify and analyze DoD and other ility definitions and relationships, and to propose a draft set of DoD-oriented working definitions and relationships for the project.
2. SQ Foundations and Frameworks. This effort helped to build SQOTA foundations by elaborating key frameworks (process-based, architecture based, means-ends based, value-basedL anticipating further subsequent elaboration via community efforts.
3. SQ-Oriented tool demos and extension plans. This effort created initial demonstration capabilities from strong existing SERC SQ analysis tool sets and explored piloting by user organizations in the DoDS ervices.
4. Program management and community building. This effort included coordinating efforts with complementary initiatives in the DoD ERS, and counterpart working groups in the International Council for Systems Engineering (INCOSE), the Military Operations Research Society (MORS), and the National Defense industry Association (NDIA)

The Phase 1resultsfor activities 1and 2 included initial top-level sets of views relevant to SQ tradespace and affordability analysis that provided an initial common framework for reasoning about SQs,similar in intent to the various views provided by SysML for product architectures and DoDAF for operational and architectural views. The views included definitions, stakeholder value-based and change-oriented views, views of ility synergies and conflicts resulting from ility achievement strategies, and a representation scheme and support system for view construction and analysis.

Phase 1also determined that strong tradespace capabilities were beingdeveloped for the tradespaceanalysisofphysicalsystems. However,basedonsourcessuchastheJCIDSsurvey of combat commanders' tradespace needs, itfound that major gaps existed between commanders' SQ tradespace needs and available capabilities for current and future cyber- physical-human systems.The SERC also characterized the benefits and limitations of using existing tools to address SQ tradespace issues, via collaboration with other leading organizations inthe DoD ERS tradespace area,such asthe Army Engineer Research and Development Center (ERDC) and TARDEC organizations, NAVSEA, the USAF Space and Missile Systems Command; DoD FFRDCs such as Aerospace, Mitre, and the Software Engineering Institute; and Air Force and Navy participants via the SERC Service academies AFIT and NPS.

PHASE 2 OBJECTIVES, APPROACH AND RESULTS

As a result, the focus of Phase 2 was to strengthen the conceptual frameworks underlyingSQ tradespaceandaffordabilityanalysis,andtoapplythemethodsandtools identifiedand extended inPhase 1on problemsrelevantto DoD, usingthe information availablefrom development of a largeweapon systems and large automated information systems. The SERC worked with system developers directly and via participation and leadership in Government and industry working groups in such organizations as INCOSE, NDIA, and the Army-led Practical Systems and Software Measurement organization, to gain a deeper shared understanding of the strengths and limitations of the tradespace tools and methods developed under Phase 1 and elsewhere.

Task 1: /TAP Foundations and Frameworks. Phase 2 activities expanded the set of SQs represented in the tradespace, organized them into a more orthogonal value-based, means- ends hierarchy, obtained initial results in identifying and quantifying the synergies and conflicts resulting from strategies to optimize individual SQs, and developed prototype tools for representing and applying the results.

Task 2. iTAP Methods and Tools Piloting and Refinement. The SQ-oriented tool demos performed in Phase 1also led to Phase 2 interactions with DoD organizations, particularly TARDEC and NAVSEA, interested in their applicability in enhancing their systems engineering capabilities. These interactions led to refinements of existing methods and tools to address set- based vs. point design of ground vehicles and ships, and on extensions from physical systems to cyber-physical-human systems and to affordability analysis. Further interactions leading to piloting engagements included AFIT's use ofthe CEVLCC life cycle cost model and related T-X Training System Tradespace Analyses. The pilot program involved advanced pilot training aircraft, simulators and course instructional elements. Its pilot organizations were the Air Force life Cycle Management Center and the Air Education and Training Command. GTRI's Framework for Assessing Cost and Technology (FACT) was extended beyond its initialsupport of USMC, and attracted several Army and Navy programs interested in piloting, extending, and tailoring its capabilities to other domains.

Task 3. Next-Generation, Full-Coverage Cost Estimation Model Ensembles. A third area of engagement starting from exploratory discussions in Phase 1 was a new task to develop Next- Generation, Full-Coverage Cost Estimation Model Ensembles, initially for the space domain, based on discussions and initial support from the USAF Space and Missile Systems Center (SMC). Phase 2 work on this topic involved several meetings with SMC and the Aerospace Corp. with USC and NPS to set context and initial priorities. These included addressal of future cost estimation challenges identified in the SERC RT-6 Software Cost Estimation Metrics Manual developed for the Air Force Cost Analysis Agency, and prioritization of research efforts based on strength of DoD needs and availability of DoD-relevant data. Exploratory activities were pursued with respect to a seeping of full-coverage of space system flight, ground, and launch systems; hardware, software and labor costs; and system definition, development, operations, and support costs, along with explorations of sources of data for calibrating the models.

PHASE 3 OBJECTIVES, APPROACH AND RESULTS

Task1:SQFoundations and Frameworks. MIT'sPhase 2 research refined aSQssemantic basis for change-related SQs. and developed prototype tools for formal analysis of the results. Phase 3 extended the SQs semantic basis for change-related SQs, resulting from continuing literature review of SQs, collaborative work on formalization of the basis, and experience in applying the basis in historical cases. Progress and adjustments to the basis have been made as a result of feedback from other academic researchers, and specifically in MIT- UVa collaboration in their efforts on formalization and development of a REST (representational state transfer) web-based service implementation. This resulted in an expanded and more explicit representation for the semantic basis, as well as motivating the need to create a translation layer for practical use of the basis. Phase 3 also refined the SQ definitions, reviewed existing SQ definition standards, developed an initial SQs ontology reflecting the reality that the ilities have multiple definitions varying by domain, and multiple values varying by system state, processes, and relations with other ility levels. Phase 3 also expanded the initial 4x4 synergies and conflicts matrix into a full 7x7 inter-ility-class synergies and conflicts matrix, and 7 smaller intra-ility-class synergies and conflicts matrices.

Task 2: SQ-Oriented tool demos and extension plans. Phase 2 effort created initial demonstration capabilities from strong existing SQ analysis toolsets and explored piloting by user organizations, via collaboration with other leading organizations in the DoD ERS tradespace area. Phase 3 broadened and deepened these initial contacts, including with such organizations as the Army Engineer Research and Development Center (ERDC) and TARDEC organizations, NAVSEA,the USAF Space and Missile Systems Command; DoD FFRDCs such as Aerospace, Mitre, and the Software Engineering Institute; and Air Force and Navy participants via the SERC Service academies AFIT and NPS. In particular, WSU and PSU advanced the SQOTA coordination with the ERS NAVSEA group, working with them to define the specific tradespace approaches and priorities for enhanced set-based design for ERS, that will complement and extend the tool and procedures they have been using. TARDEC was actively engaged as a partner for co-development, piloting and transition into use. The GTRI FACT-related capabilities were strongly co-funded and enhanced by and for the Army Engineer R&D Center (ERDC), other Army, Navy, and further USMC programs, including strengthening and extension of the infrastructure for supporting and extending the initial FACT capabilities.

Task 3: Next-Generation, Full-Coverage Cost Estimation Model Ensembles. Based on the exploratory needs and data assessments in Phase 2, a Phase 3 workshop including Air Force, Navy, aerospace industry, andSERC researchers concluded thatthere were strong needsfor better estimation of operations and support costs, but that the data available lacked adequate cost driver information, except in in the software area. The workshop recommended that the most promising initial areas to pursue would be for software development, systems engineering, and the use of systems engineering cost drivers to improve estimation of system development costs. Further research and workshops identified further sources of data and some shortfalls in current models in these areas, and developed requirements and draft frameworks for the next-generation models. These have been used in Phase 4 to develop and calibrate prototype modelsfor systems and software engineering cost estimation models, and to pursue research in the use of the systems engineering model to better estimate system development costs.

PHASE 4 OBJECTIVES, APPROACH AND RESULTS

Task 1 .SQFoundations and Frameworks. Rather than attempt a breadth-first elaboration of the 176 SQ Synergies and Conflicts strategies in the 7x7 matrix, including its ontology elements of Referents, States, Processes, and Relations for each strategy, the USC ontology-based research id a depth first research effort on a particular SQ that touches all of the four major SQ categories. This SQ is Maintainability. It clearly drives Life Cycle Efficiency, as typically at least 75% of a system's Total Cost of Ownership is spent on operations and maintenance. It is one of two means for achieving Changeability, involving external change vs. the internal change accomplished by Adaptability. It is clearly key to Dependability, as Maintainability in terms of Mean Time to Repair (MTIR) is the key relation between Reliability in terms of Mean Time Between Failures (MTBF) and Availability in the relation Availability= MTBF I(MTBF + MTIR). And the key systems aspects being depended-upon are primarily the components of Mission Effectiveness. 

This depth-first approach thus provided insights on the overall Product Quality ontology structure without having to consider all of the 176 strategies in depth. The insights resulted in changes to the SQterminology, as shown in the main description of the Phase 4 Results. Examples are changing Resource Utilization to Life Cycle Efficiency,to be more compatible with the Better Buying Power terminology, and changing Flexibility to Changeability, to be better aligned with the MIT Quality In Use ontology structure.

The MIT Quality In Use ontology structure was refined to address further semantic aspects, and requirements for a translation layer to facilitate its use were developed, as elaborated in the main Phase 4 results section. Similarly,the U.Virginia Phase 4 research on formalizing both the MIT and USC ontologies is elaborated inthe main Phase 4 results section. An initial semantic diagram relating the USC and MIT terms and relationships is also presented in the main Phase 4 results section.

Task 2. SQ-Oriented tool demos and extension plans. The USC depth-first exploration of Maintainability identified the need for a better balance of attention during the system acquisition phase between optimizing on system acquisition cost- effectiveness and optimizing on system life-cycle cost-effectiveness, particularly for software, due to the major differences between software and software logistical aspects. This led to the development of a proposed framework of Maintainability Readiness Levels for Software - Intensive Systems. Again, details are provided in the main Phase 4 results section.

Other Task 2 Phase 4 Objectives, Tasks and Results summaries will be provided later

TASK 3: NEXT-GENERATION, FULL-COVERAGE COST ESTIMATION MODEL ENSEMBLES

The Task 3 Phase 4 Objectives, Tasks and Results summaries will be provided later.