Data Science Approaches to Prevent Failure in Systems Engineering
Systems Engineering and Systems Management Transformation
Dr. Karen Marais
Dr. Bruno Ribeiro
Anecdotes and statistics on the failures of systems engineering have become a sure-fire way of attracting attention and lamentation during presentations. No-one is immune to the failure disease and in particular past success is no guarantee of future performance—organizations that have succeeded spectacularly in one project will fail just as spectacularly in the next project.
For example, many major United States defense programs are challenged to meet time, budget and performance expectations. The consumer goods sector also has many failures, Samsung Galaxy Note 7 fires, or the Ford Explorer rollover problems.
In response to these challenges, new systems engineering methods, processes, and tools are continuously proposed and implemented, including numerous new methods of risk identification, tracking, and management. Yet the frequency of failures shows no signs of decreasing, and, meanwhile, engineering creativity in large complex systems seems to be stifled. Rather than the revolutionary creations our 20th century counterparts foresaw appearing in the 21st century, we find ourselves focused on evolutionary improvements.
Most systems engineering failures, even those in new, one-of-a-kind high-tech systems, do not involve previously unknown phenomena, or black swans (Sorenson and Marais, 2016). As appealing as the black swan metaphor is, the real reasons for most failures are, in fact, rather prosaic and predictable white swans. Sorenson and Marais identified a set of 23 “real reasons”, ranging from “conducted poor requirements engineering” to “created inadequate procedures”.
In complex development projects, neither traditional engineering management data nor big data analysis is able to consistently and accurately pinpoint issues. Failures, even though they may appear simple in retrospect, are often the result of a complex network of decisions, many of them locally and temporally rational. Modeling and predicting such complex events requires complex models; and complex models need large amounts of historical data to give accurate predictions and insights; cutting-edge projects do not have an abundance of historical data. More (and possibly better) data are needed.
In these complex scenarios, there is a potential to augment existing engineering management data with “wisdom of the crowd” information. Wisdom of the crowd (WoC) refers to the hypothesis that the collective opinion of a large number of non-experts (e.g., a novice engineer) is a better signal to the health of a project than the opinion of a single expert (say, an experienced manager). For instance, employees give their best assessment of the timeline and budget of a project, given their knowledge of the system. These assessments are then combined with a machine learning algorithm to predict the probability that the project will be successful or the system will fail. Unfortunately, with bonuses and salaries depending on contracts, it is challenging to ensure employees truthfully report their project estimates.
- Poster - RT 166: Formal Methods in Resilient Systems Design using a Flexible Contract Approach
- Poster - RT 182: Enterprise Systems-of-Systems Model for Digital Thread Enabled Acquisition
- Poster - Formal Methods in Resilient Systems Design using a Flexible Contract Approach
- Poster - Meshing Capability and Threat-based Science & Technology Resource Allocation