Publication

RT-19, Research on Building Education & Workforce Capacity in Systems Engineering, is a research study whose goal is to understand the impact on student learning of and career interest in Systems Engineering (SE) through a set of diverse capstone courses that expose students to authentic Department of Defense (DoD) problems and engage them in learning and practice of systems engineering. SE Capstone courses were developed and piloted during the 2010-11 academic year (and beyond) in eight civilian universities and six military institutions affiliated with the Systems Engineering Research Center (SERC). The strategic goal addressed by this research is to better understand how differing course designs, structures, materials, instructional practices, and other inputs, such as the involvement of DoD and industry mentors, impact student learning and career interest in SE. This research explored methods and approaches to augment the SE workforce for future DoD and related industry workforce needs in order to inform future investments for the purpose of institutionalizing and scaling up effective methods. This research encompassed a 20-month, three-phase effort from March 1, 2010 to October 31, 2011, including planning, course implementation, and analysis. Institutions were selected for participation through a competitive application process based on a set of criteria developed in consultation with the sponsor, and partners were awarded a subcontract of approximately $200,000 for development, implementation, analysis, and reporting on their SE Capstone project. 

According to final reports submitted by principal investigators, 330 and 257 students participated in RT19-sponsored SE Capstone courses in the fall 2010 and spring 2011 semesters, respectively. Many institutions enrolled the same students for both semesters, but a few, such as the University of Maryland, enrolled a new cohort of students in the spring, bringing the total number of students impacted to more than 360. Approximately half were undergraduates, of whom the majority were fourth year seniors. Of the graduate students, most were first year students, with small percentage post-graduates participating in roles such as project manager.

Four topic areas illustrating authentic DoD problems were presented for student teams’ projects. Problem area #1, low-cost, low-power computer solutions (see Table 2 for more complete description), was the most heavily subscribed topic, with more than half the projects addressing this problem area. Problem areas were selected, in part, based on expertise of participating faculty and institutional resources, and on availability of DoD and local experts. Institutions organized their teams in different ways: the most common structure included several teams working on several different design problems.

A majority of the universities relied on the expertise of systems engineering faculty to lead or contribute to the conceptualization, development, and implementation of the course, but many other faculty were involved as well, particularly from mechanical engineering and computer science departments. At 11 institutions, faculty came from at least three separate engineering disciplines, literally embodying the multi-disciplinarity of an SE team. Nearly two-thirds of the 14 projects were planned and implemented by teams of two or three faculty members, but four projects included four or more faculty. Only one institution (US Naval Academy) developed a Capstone course that was planned and taught by a single faculty member.

The research team gathered the following data in order to analyze the impact of the SE Capstone project on student learning of SE, student interest in SE careers, and student awareness/interest in authentic DoD problems: pre/post student surveys; pre/post case study analysis by students; and student blog posts. In addition, this report also contains input gathered from the July SE Capstone conference, review and analysis of final reports submitted by principal investigators, as well as papers, publications, and posters developed by faculty, researchers, and students.

Many faculty used customized assessments and other means (e.g., student participation in competitions) to assess student outcomes (See Appendix B for description of course materials/student deliverables and internal assessments). Through semantic analysis of students’ constructed responses on definitions of systems engineering (administered in pre- and post-course surveys) as compared to two expert definitions, larger gains were observed for undergraduates and students with no prior SE experience than for students who self-identified with prior SE knowledge. Students with no prior SE experience not only showed larger gains, but also they ended with a slightly higher percentage than the group as a whole.

The research team used an analytic rubric to measure changes in the level of complexity of student thinking using systems engineering knowledge from pre- to post-course on the case study analysis of the Bradley Fighting Vehicle. A life cycle model was used to map units of competency from the SPRDESE/PSE Competency Model into lower (definitional) or higher (development, deployment) categories of analysis (Sage, 2000, p. 166). For the entire set of matched responses, statistically significant increases were recorded for all categories of competencies combined, and for categories B and C (denoting more sophisticated reasoning). Students with prior SE experience had higher initial and final scores, but the difference between the two groups’ (students with and without prior SE experience) was smaller by the post-test, suggesting that the SE Capstone courses positively impacted student learning of SE, particularly for those students without SE experience. The increases were statistically significant for both groups of students.

Finally, weekly posts to weblogs (“blogs”) and a final post were required of all teams in order to provide qualitative data and insights into the changes in the level of complexity of students’ thinking about SE as applied to their Capstone project. Blogs were used in a variety of ways by partner institutions; therefore, generalizations about the RT-19 student population cannot be made from this data source. Student blog posts described phases of the SE design process; included project artifacts and media files; and described challenges teams encountered during the project. These included such issues as making design tradeoffs; providing adequate security for their (wireless) products; relying too much on knowledge or technical skills of one team member with a specific area of expertise; setting reasonable and achievable goals for product design within a school year; communication challenges in an interdisciplinary team with different perspectives and varying levels of expertise; and managing time and design constraints. The most common student responses about the most challenging aspects of the project included managing the dynamics of a multi-disciplinary group and communication problems.

Overall, 82% of responding students felt their group produced a successful product. Of those who did not feel their projects were successful, lack of resources and time were the most frequently cited reasons. 

A goal of the SE Capstone courses implemented in RT-19 was to increase student awareness of the diversity of problems addressed by the DoD. From pre- to post-survey, changes from very general to more specific types of problems identified by students, including greater use of SE terminology, were observed. The problem area that increased the most in students’ awareness was energy-related, particularly energy efficiency and green energy, while the area that decreased the most was weapons and weapon systems.

Another goal of SE Capstone courses was to increase student interest in: SE careers generally; SE careers in government; and SE careers in industry. Post-survey means for the entire population of matched pre-/post-survey responses increased in all three categories, although these increases were not statistically significant. For those students with prior SE experience, Q1 (general) and Q3 (industry) increased, while the mean for Q2 (government) decreased. For the matched group without prior SE experience, the means for all three questions increased, with the mean for Q3 (industry) increasing the most. Further analysis of students’ Likert Scale responses show more subtle differences in the level of interest (from low to high) among the various subgroups analyzed. Eighty percent of students who responded to an open-ended question asking whether they would pursue a career in systems engineering career and, if so, why, stated that they would, and many indicated this would be some time in the future after gaining experience in their chosen engineering discipline. The remaining 20 percent who responded they would not consider a career in SE listed a preference for pursuing their chosen engineering career as the main reason. Overall, a large majority of students who answered an openended question about the applicability of systems engineering to future engineering studies and plans (64 of 67) agreed that SE provided a useful framework and broad perspective needed to manage complex engineering challenges.

The recruitment, involvement, and impact of DoD and industry mentors is an aspect of the SE Capstone project which deserves special emphasis in the analysis of the overall project in light of the intensive efforts made to connect mentors with student teams, the voluntary nature of mentors’ roles, and implications for sustainability and scaling up. All Capstone partner institutions had a DoD mentor, and about half had additional mentors. Mentors played the role of clients as well as technical experts, guiding students toward solutions. Lack of role definition of mentors was cited as problematic by PIs; preference was expressed for DoD mentors to serve as clients. Lack of and late start of mentor involvement with student teams, as well as varying levels and frequency of communication between mentors and students were cited as challenges. Beneficial impacts of mentor involvement were reported by PIs when communication was frequent and specific, particularly in the case of design reviews. Defense prime contractors who served as mentors were reported to provide a different perspective than DoD mentors, chiefly, by representing “the solution viewpoint” and “saving student teams from exploring too many blind alleys.” Student rankings of DoD and industry mentors’ usefulness in learning about and applying SE in their courses were in the mid- to low range for all but one institution, as compared to six other course inputs such as lectures and other team members. In considering sustaining mentor relationships and scaling up, it will be important to examine value of mentors, the sustainable features of mentor relationships, and the characteristics of particularly strong mentor relationships.

Through site visits to SE Capstone universities in spring 2011, a team of sponsor representatives identified a set of promising practices—approaches which were present in universities where students demonstrated higher levels of communication, analysis, and awareness of the SE process during the site visits. Although limitations of the data and the scope of RT-19 do not allow for analysis of correlations between these promising practices and the SE Capstone models that may have led to greater student outcomes, these promising practices have informed the research being undertaken through RT-19A, the Pilot for Scaling Up and Sustaining Effective SE Capstone Practices. A graphical representation of the presence (or lack thereof) of these promising practices among all participating RT-19 universities appears as Appendix E. 

In order to institutionalize aspects of the SE Capstone project post-DoD funding, it will be necessary to address the critical challenges faced by faculty who are responsible for implementing these projects. Some challenges identified by faculty resulted from the accelerated startup and logistical demands associated with a new course. Here, issues such as recruitment, material and assessment development, coordination of schedules among students in different engineering disciplines and coordination with external mentors were commonly cited challenges. Other challenges were associated with establishing a broad, overarching SE framework in the context of traditional departmental academic structures. Course expectations, grading policies, and formation of teams representing different disciplines were cited as issues, as were negotiating the optimal balance between SE content knowledge and disciplinespecific technical expertise among students and faculty and identifying manageable project scope for the given instructional period. Lastly, models for sustaining and institutionalizing SE Capstone projects were proposed, including fee-based programs in which students work on a problem from industry or government as a contractor. 

 Findings and Recommendations:

Limitations in the data and the many approaches and variables used in the 14 pilot courses prevent statistical correlations with student outcomes and “optimal” course designs. However, the following summary of findings are grounded in data collected through RT-19: 

Analysis of student definitions of systems engineering showed that participating students were able to use general systems engineering terminology almost as well as experts but that they still had some way to go in employing more technical systems engineering language. However, those with the most to learn—undergraduates and those with no prior system engineering experience—improved the most, particularly in terms of technical language.

Analysis of the Bradley Fighting Vehicle case study showed that students increased in their ability to identify problems that mapped to specific systems engineering competencies, particularly those related to the technical elements, but that they were less likely to mention the “soft” competencies like communication and leadership. 

The blogs, where used well, showed students working through the phases of the design process and struggling with various technical and communication issues along the way. 

Students enjoyed the real-world nature of the projects—both in terms of building an artifact that might be used and in terms of the SE project context (budget constraints, interdisciplinary teams, experts as mentor)—and that they appreciated the contribution that the systems engineering perspective brought to their work. 

SE Capstone courses do not appear to have had a major impact on the students’ immediate career plans, it must be noted that many had their immediate post-college plans in place and that a large majority of both undergraduates and graduate students believed that they might choose careers in systems engineering sometime in the future. 

Recommendations for future implementations and future research include:

1. Develop a methodology to prioritize and rank the student attributes and outcomes most likely to meet DoD and defense industry needs in the near term (0-5 years) and longer term.
2. Examine the presence, depth, and characteristics of implementation of the promising practices through case study analysis (a component of research included in RT-19A); correlate, where possible, to the highest priority student attributes described in (1), above.
3. Distill the attributes of effective DoD and industry mentor relationships through further analysis of “what worked” and what did not. Investigate the incentives and rewards for mentors to continue involvement with university partners.
4. Make very explicit the goal of attracting students to DoD careers in systems engineering in coursework and other communications; provide technical assistance and other materials to mentors and faculty.
5. Leverage the experience and expertise of the RT-19 and RT-19A to build and expand a learning community of SE Capstone stakeholders (engineering institutions, clients, and mentors).
6. Consider piloting new approaches to sustain the SE Capstone project, including the creation of an online repository of potential DoD problem areas and clients along with a “venture fund” that would provide small grants of $5,000-$10,000 for materials and access to DoD problems and clients for institutions that already organize Capstone projects.
7. Publicize in relevant professional journals, education media, and the general media the contributions of SE Capstone design teams to the development of solutions critical for our military and our nation’s security.
8. Conduct a longer-term study (1-5 years) tracking RT-19 participants and their career choices and employment trends.