Publication

One of the key elements of the SERC’s research strategy is transforming the practice of systems engineering and associated management practices – “SE and Management Transformation (SEMT).” The Grand Challenge goal for SEMT is to transform the DoD community’s current systems engineering and management methods, processes, and tools (MPTs) and practices away from sequential, single stovepipe system, hardware-first, document-driven, point-solution, acquisition-oriented approaches; and toward concurrent, portfolio and enterprise-oriented, hardware-software-human engineered, model-driven, set-based, full life cycle approaches.

These will enable much more rapid, concurrent, flexible, scalable definition and analysis of the increasingly complex, dynamic, multi-stakeholder, cyber-physical-human DoD systems of the future. Four elements of the research strategy for SE Transformation are the following:

1. Make Smart Trades Quickly: Develop MPTs to enable stakeholders to be able to understand and visualize the tradespace and make smart decisions quickly that take into account how the many characteristics and functions of systems impact each other

2. Rapidly Conceive of Systems: Develop MPTs that allow multi-discipline stakeholders to quickly develop alternative system concepts and evaluate them for their effectiveness and practicality

3. Balance Agility, Assurance, and Affordability: Develop SE MPTs that work with high assurance in the face of high uncertainty and rapid change in mission, requirements, technology, and other factors to allow systems to be rapidly and cost-effectively acquired and responsive to both anticipated and unanticipated changes in the field

4. Align with Engineered Resilient Systems (ERS): Align research to leverage DoD’s ERS strategic research initiative and contribute to it; e.g., ERS efforts to define new approaches to tradespace analysis.

“Systems” covers the full range of DoD systems of interest from components such as sensors and effectors to full systems that are part of net-centric systems of systems and enterprises. “Effectiveness” covers the full range of needed system quality attributes or ilities, such as reliability, availability, maintainability, safety, security, performance, usability, scalability, interoperability, speed, versatility, flexibility, and adaptability, along with composite attributes such as resilience, sustainability, and suitability or mission effectiveness. “Cost” covers the full range of needed resources, including present and future dollars, calendar time, critical skills, and critical material resources.

The primary focus of RT-113, ilities Tradespace and Affordability Project (iTAP) is on strategy 3, although its capabilities also support strategies 1, 2, and 4. It particularly focuses on the tradespace among a system’s Ilities, also called non-functional requirements or system quality attributes. The ilities differ from functional requirements in that they are systemwideproperties that specify how well the system should perform, as compared to functions that specify what the system should perform. Adding a functional requirement to a system’s specification tends to have an incremental, additive effect on the system’s cost and schedule. Adding an ility requirement to a system’s specification tends to have a systemwide, multiplicative effect on the system’s cost and schedule. Also, ilities are harder to specify and evaluate, as their values vary with variations in the system’s environment and operational scenarios.

Further, the satisfaction of their specifications is much harder to verify than placing an X or a URL address in a functional traceability matrix, as the verification involves up to the entire set of system functions. It also requires considerable effort in analysis across a range of environments and operational scenarios. As a result, it is not surprising that problems in satisfying ility requirements are underaddressed in early phases and are the source of many subsequent DoD acquisition program cost and schedule overruns. Also, with some exceptions such as pure physical systems and pure software systems, there is little technology in the form of scalable methods, processes, and tools (MPTs) for evaluating the satisfaction of multiple-ility requirements and their associated tradespaces for complex cyber-physical-human systems.

The increasingly critical DoD need for such capabilities has been identified in several recent studies and initiatives such as the AFRL “Technology Horizons” report (Dahm, 2010), the National Research Council’s “Critical Code” Report (NRC, 2010), the SERC “Systems 2020” Report (SERC, 2010), the “Manual for the Operation of the Joint Capabilities Integration and Development System” (JROC, 2012), and the DoD “Engineered Resilient Systems (ERS) Roadmap” (Holland, 2012). The particular need for Affordability has been emphasized in several USD(AT&L) and DepSecDef “Better Buying Power” memoranda BBP 1.0 and 2.0 (Carter et al., 2010-2013) and the recent BBP 3.0 White Paper (Kendall, 2014).

Here is a summary of the current and projected iTAP contributions in support of the objectives of Better Buying Power 3.0, relative to BBP 3.0 White Paper citations in italics. Details of the contributions are provided in the later section of the report.

Continue to set and enforce affordability constraints. Strengthen and expand “should cost” as an important tool for cost management. Building on previous results in RT-6 (Air Force Software Cost Modeling) and RT-18 (Valuing Flexibility), iTAP Phase 3 developed a framework and initial quantitative results for tradespace analysis of acquisition and total ownership should-costs vs. schedule, functionality, and reliablility, and an overall framework and initial population of sources of synergy and conflict among cost, schedule, and other system quality attributes. Future plans include upgrading the models to reflect emerging trends such as flexibility to accommodate increasingly rapid change, adaptability to accommodate autonomy and smart devices, usability to accommodate social networking and labor force evolution, and interoperability to accommodate increasingly interconnected systems of systems.

Employ appropriate contract types, but increase the use of incentive-type contracts. “formulaic incentives” show a high correlation with better cost and schedule performance. As above, Phase 3 has been developing cost-schedule-performance tradespace models to provide stronger formulas on which to define formulaic incentives.

Increase the use of performance-based logistics (PBL). As documented in Appendix A, Enclosure A of the 19 Jan 2012 JCIDS Manual, a survey of combat commenders on critical logistics attributes identified the key quality attributes for logistics performance as Responsiveness, Sustainability, Attainability, Flexibility, Survivability, and Economy, along with several other attributes. Phase 3 is developing an ontology of such attributes to provide a stronger basis for defining logistics performance.

Use Modular Open Systems Architecture to stimulate innovation. SERC has been collaborating with TARDEC and NAVSEA in Phase 3 to define sound properties for set-based design as a stronger way to achieve mission-supportive open systems architectures, and to define related set-based design processes.

Provide clear “best value” definitions so that industry can propose and DoD can choose wisely: providing industry with information on the value, in monetary terms, of higher levels of performance than minimally acceptable or threshold levels. As above, Phase 3 has been developing cost-schedule-performance tradespace models to provide stronger formulas on which to define the value of higher levels of performance.

Improve our leaders’ ability to understand and mitigate technical risk: to minimize the likelihood of program disruption and to maximize the probability of fielding the desired product within reasonable time and cost. Phase 3 research has been extending, applying, and refining an iterative Epoch-Era approach to address sources of uncertainty and risk. A related SERC project, Quantitative Risk, is developing improved methods for identifying and quantifying leading risk indicators. The set-based design approach goes beyond design for acquisition, to use areas of requirements uncertainty as foci for architecting sets of requirements likely to needed post-acquisition.

PHASE 1 OBJECTIVES, APPROACH, AND RESULTS

The major objectives of the initial 5-month Phase 1 activity were to lay strong foundations for ITAP Phase 2, including knowledge of Department of Defense (DoD) ility priorities; foundations and frameworks for ITAP analysis; extension and tailoring of existing ITAP methods, processes, and tools (MPTs); and exploration of candidate Phase 2 pilot organizations for ITAP MPTs.

Four activities were pursued in achieving these objectives:

1. Ility Definitions and Relationships. Phase 1 included a discovery activity to identify and analyze DoD and other ility definitions and relationships, and to propose a draft set of DoD-oriented working definitions and relationships for the project.

2. iTAP Foundations and Frameworks. This effort helped to build iTAP foundations by elaborating key frameworks (architecture-based, change-based, process-based, means-ends based, value-based), anticipating further subsequent elaboration via community efforts.

3. Ility-Oriented tool demos and extension plans. This effort created initial demonstration capabilities from strong existing ITA analysis toolsets and explored piloting by user organizations in the DoD Services.

4. Program management and community building. This effort included coordinating efforts with complementary initiatives in the DoD ERS, and counterpart working groups in the International Council for Systems Engineering (INCOSE), the Military Operations Research Society (MORS), and the National Defense industry Association (NDIA).

The Phase 1 results for activities 1 and 2 included initial top-level sets of views relevant to ilities tradespace and affordability analysis that provided an initial common framework for reasoning about ilities, similar in intent to the various views provided by SysML for product architectures and DoDAF for operational and architectural views. The views included definitions, stakeholder value-based and change-oriented views, views of ility synergies and conflicts resulting from ility achievement strategies, and a representation scheme and support system for view construction and analysis.

Phase 1 also determined that strong tradespace capabilities were being developed for the tradespace analysis of physical systems. However, based on sources such as the JCIDS survey of combat commanders’ tradespace needs, it found that major gaps existed between commanders’ ility tradespace needs and available capabilities for current and future cyber-physical-human systems. The SERC also characterized the benefits and limitations of using existing tools to address ility tradespace issues, via collaboration with other leading organizations in the DoD ERS tradespace area, such as the Army Engineer Research and Development Center (ERDC) and TARDEC organizations, NAVSEA, the USAF Space and Missile Systems Command; DoD FFRDCs such as Aerospace, Mitre, and the Software Engineering Institute; and Air Force and Navy participants via the SERC Service academies AFIT and NPS.

PHASE 2 OBJECTIVES, APPROACH, AND RESULTS

As a result, the focus of Phase 2 was to strengthen the conceptual frameworks underlying ilities tradespace and affordability analysis, and to apply the methods and tools identified and extended in Phase 1 on problems relevant to DoD, using the information available from development of a large weapon systems and large automated information systems. The SERC worked with system developers directly and via participation and leadership in Government and industry working groups in such organizations as INCOSE, NDIA, and the Army-led Practical Systems and Software Measurement organization, to gain a deeper shared understanding of the strengths and limitations of the tradespace tools and methods developed under Phase 1 and elsewhere.

Task 1: ITAP Foundations and Frameworks. Phase 2 activities expanded the set of ilities represented in the tradespace, organized them into a more orthogonal value-based, means-ends hierarchy, obtained initial results in identifying and quantifying the synergies and conflicts resulting from strategies to optimize individual ilities, and developed prototype tools for representing and applying the results.

Task 2. iTAP Methods and Tools Piloting and Refinement. The Ility-oriented tool demos performed in Phase 1 also led to Phase 2 interactions with DoD organizations, particularly TARDEC and NAVSEA, interested in their applicability in enhancing their systems engineering capabilities. These interactions led to refinements of existing methods and tools to address set-based vs. point design of ground vehicles and ships, and on extensions from physical systems to cyber-physical-human systems and to affordability analysis. Further interactions leading to piloting engagements included AFIT’s use of the CEVLCC life cycle cost model and related T-X Training System Tradespace Analyses. The exploratory-use program involved advanced pilot training aircraft, simulators and course instructional elements. Its user organizations are the Air Force Life Cycle Management Center and the Air Education and Training Command.

Task 3. Next-Generation, Full-Coverage Cost Estimation Model Ensembles. A third area of engagement starting from exploratory discussions in Phase 1 was a new task to develop Next-Generation, Full-Coverage Cost Estimation Model Ensembles, initially for the space domain, based on discussions and initial support from the USAF Space and Missile Systems Center (SMC). Phase 2 work on this topic involved several meetings with SMC and the Aerospace Corp. with USC and NPS to set context and initial priorities. These included addressal of future cost estimation challenges identified in the SERC RT-6 Software Cost Estimation Metrics Manual developed for the Air Force Cost Analysis Agency, and prioritization of research efforts based on strength of DoD needs and availability of DoD-relevant data. Exploratory activities were pursued with respect to a scoping of full-coverage of space system flight, ground, and launch systems; hardware, software and labor costs; and system definition, development, operations, and support costs

Phase 3 Objectives, Approach, and Results Overview

Details of each task’s past results, Phase 3 results, and future plans are in the detailed reports of each organization participating in the task.

Task 1: ITAP Foundations and Frameworks. Phase 2 refined an ilities semantic basis for change-related ilities. and developed prototype tools for formal analysis of the results. Phase 3 extended the ilities semantic basis for change-related ilities, resulting from continuing literature review of ilities, collaborative work on formalization of the basis, and experience in applying the basis in historical cases. Progress and adjustments to the basis have been made as a result of feedback from other academic researchers, and specifically in MIT- UVa collaboration in their efforts on formalization and development of a REST (representational state transfer) web-based service implementation. This has resulted in an expanded and more explicit representation for the semantic basis, as well as motivating the need to create a translation layer for practical use of the basis. Phase 3 also refined the ility definitions, reviewed existing ility definition standards, developed an initial ilities ontology to address the current shortfalls in existing ility-related standard and guidelines. These generally do not address the reality that the ilities have multiple definitions varying by domain, and multiple values varying by system state, processes, and relations with other ility levels. Phase 3 also expanded the initial 4x4 synergies and conflicts matrix into a full 7x7 inter-ility-class synergies and conflicts matrix, and 7 smaller intra-ility-class synergies and conflicts matrices. Detailed results are in the USC, MIT, and UVa sections.

Task 2: Ility-Oriented tool demos and extension plans. Phase 2 effort created initial demonstration capabilities from strong existing ITA analysis toolsets and explored piloting by user organizations, via collaboration with other leading organizations in the DoD ERS tradespace area. Phase 3 broadened and deepened these initial contacts, including with such organizations as the as the Army Engineer Research and Development Center (ERDC) and TARDEC organizations, NAVSEA, the USAF Space and Missile Systems Command; DoD FFRDCs such as Aerospace, Mitre, and the Software Engineering Institute; and Air Force and Navy participants via the SERC Service academies AFIT and NPS. In particular, we have advanced our coordination with the ERS NAVSEA group, working with them to define the specific tradespace approaches and priorities for enhanced set-based design for ERS, that will complement and extend the tool and procedures they have been using. We anticipate access to a public-release copy of the ship design option datasets developed as part of their ERS set-based-design experiment. We initiated discussions with NAVAIR and with the DARPA/TARDEC Ground Experimental Vehicle – Technology (GXV-T) project regarding piloting iTAP MPT. We have begun to develop a network model of the interdependencies within and between the TARDEC Ground System Architecture Framework and Performance Specification Framework, and coordination with TARDEC in this research. TARDEC is actively engaged as a partner for co-development, piloting and transition into use. Detailed results are in the Wayne State, GTRI, Penn State, AFIT, NPS, and USC sections.

Task 3: Next-Generation, Full-Coverage Cost Estimation Model Ensembles. Based on the exploratory needs and data assessments in Phase 2, a Phase 3 workshop including Air Force, Navy, aerospace industry, and SERC researchers concluded that there were strong needs for better estimation of operations and support costs, but that the data available lacked adequate cost driver information, except in in the software area. The workshop recommended that the most promising initial areas to pursue would be for software development, systems engineering, and the use of systems engineering cost drivers to improve estimation of system development costs. Further research and workshops have identified further sources of data and some shortfalls in current models in these areas, and have developed requirements and draft frameworks for the next-generation models. These will be used in Phase 4 to develop and calibrate prototype models for systems and software engineering cost estimation models, and to pursue research in the use of the systems engineering cost model to better estimate system development costs. Detailed results are in the USC, NPS, and AFIT sections.

Complementary Funding

During Phase 3, RT-113 contributed significantly to the SERC objective of obtaining complementary funding to its SERC core support. The Army Engineer Research and Development Center sponsored MIT research in SE knowledge capture, and provided over $1million to GTRI for tradespace tools research. GTRI also obtained significant from the Navy for tradespace tools research. USC and UVa obtained complementary funding from NSF to extend the ilities tradespace theory and foundations. AFIT and NPS provided complementary support of their SERC research via participation of their faulty and students in their SERC research.