Publication

RT-19A, Research on Building Education & Workforce Capacity in Systems Engineering, is the second phase of a two-year research study whose goal is to understand the impact of diverse capstone courses that exposed undergraduate and graduate engineering majors to authentic Department of Defense (DoD) problems and engaged them in the learning and practice of systems engineering, and outcomes related to systems engineering careers and interest. Over an 18-month, three-phase effort from April 2011 to September 2012 that encompassed course planning, implementation, and analysis, participating RT-19A schools and the research team explored methods and approaches to augment the systems engineering workforce for future DoD and related industry workforce needs. 

The strategic goals addressed by this research are twofold: to understand the institutional challenges and successes in the adoption of core elements of successful systems engineering capstone projects; and to examine the contexts and program characteristics leading to highly successful student team-developed products and artifacts that respond to authentic Department of Defense (DoD) problem areas. To produce the following report, the research team gathered data from student pre and post surveys in order to analyze the impact of the systems engineering capstone project on student learning of systems engineering, student interest in systems engineering careers, and student awareness/interest in authentic DoD problems. In addition, this report also contains input gathered from surveys submitted by PIs and mentors, and from observations and interviews taken from a systems engineering capstone conference June 2012. 

Institutions were selected for participation through a competitive application process based on a set of criteria developed in consultation with the sponsor, and partners were awarded a subcontract for the development, implementation, analysis, and reporting on their systems engineering capstone project. Altogether, sixteen schools were selected to participate in the RT19A effort: six Systems Engineering Research Center (SERC) member universities, four service academies, and six partner schools. In the first year of this study fifteen systems engineering capstone courses were developed and implemented at six military institutions and eight civilian universities affiliated with the Systems Engineering Research Center. Ten of those schools returned to participate in this year’s effort. 

The capstone courses were organized around SPRDE-SE systems engineering competencies and selection of Department of Defense problem areas. Five topic areas illustrating authentic DoD problems were presented for student teams’ projects. Problem areas #2 and 4 (see Table 3 for more complete description) were the most researched topics, with more than half the projects addressing one of the two problem areas. Selection of problem areas was based on student research interest, expertise of participating faculty, or the decision to continue capstone designs from the prior year. Institutions organized their teams in different ways. The most common structure included several teams each working on a subsystem.

According to final reports submitted by principal investigators, 306 and 339 students participated in RT-19A-sponsored systems engineering capstone courses in the fall 2011 and spring 2012 semesters, respectively. Many institutions enrolled the same students for both semesters. An estimated 198 students worked on DoD problem areas, or 64.7% of students enrolled in the spring courses. Of these, 154 were undergraduates, 38 were graduate students, and 1 was a postgraduate student. The population was over three-quarters male and 20% female. Thirty percent of the students surveyed were systems engineering majors, followed by mechanical engineering majors (25%). Other student majors included electrical, industrial, software, and civil engineering; computer science; and engineering management. Only half of the students reported working in multidisciplinary teams prior to their capstone experience.

Fifty-one faculty members participated in the development, delivery, and assessment of RT-19A courses, almost the same number as participated in RT-19 (50), with the highest percentage from Mechanical Engineering departments, followed by Systems Engineering. This year’s faculty also came from Industrial, Electrical, Civil, Mechanical, Systems, Software, Ocean Engineering, and Computer Science. Eight schools included faculty participants from more than one engineering discipline.

Over the course of two semesters, students enrolled in the capstone courses created a number of physical prototypes, summarized in Table 5, that responded to their DoD problem areas. Overall, 75% of responding students felt their team produced projects that successfully fulfilled requirements; showed proof-of-concept; encouraged multidisciplinary, intergroup communication and coordination; and demonstrated their understanding of systems engineering concepts, from the initial design and requirements determination stages, to final prototype testing. Of those who did not feel their projects were successful, lack of resources and time were the most frequently cited reasons. The students attributed parts delays; the inability to build an operational prototype or to complete specific phases of the project, such as testing; communication between team members from different disciplines; and communication over distance as project problems. PIs cited technical issues with modifying off-the-shelf (COTS) software and hardware (e.g., Microsoft Kinect, batteries); time management; delays in parts acquisition; budget limits; and funding delays as challenges to student prototype construction.

A goal of the systems engineering capstone courses implemented in RT-19A was to increase student awareness of the diversity of problem areas addressed by the DoD. From pre- to postsurvey, there was an increase from 14% to 18% of students who listed what were clearly systems engineering issues (“requirements management,” “project scheduling,” “systems integration,” “predictive decision algorithms”). Research related to military field needs (materials research, troop protection, expeditionary housing, water filtration, improved IED detection, and lightweight armor) increased the most in students’ awareness. 

Another goal of the systems engineering capstone courses was to increase student interest in systems engineering careers generally; systems engineering careers in government; and systems engineering careers in industry. A comparison was made between the means of the baseline survey respondents and post-survey matched group in all three categories. Results indicated that the matched group was biased toward systems engineering careers from the beginning, with higher mean scores on the baseline survey than the larger group of respondents. Postsurvey means for the entire population of matched pre/post-survey responses decreased in all three categories, although these decreases were not statistically significant and none of the means were less than “3,” indicating a moderate interest in systems engineering careers. Further analyses of students’ responses show more subtle differences in the level of interest (from low to high) among the various subgroups analyzed. Where there was change was in the mean scores for those who chose 1, 2, or 3 on the 5-point scale in the baseline survey. This change was statistically significant in the direction of increased student interest in becoming a systems engineer for government.

An important component of the capstone experience was the inclusion of mentors who played multiple roles as technical experts guiding student teams toward solutions and risk assessment; as reviewers at interim and final design presentations; as clients who helped determine requirements; and as career advisors. PIs reported participation of over forty mentors from Department of Defense offices and other leading defense industry corporations (a full list is included in Appendix B of the report). All institutions that implemented capstone courses had DoD-assigned mentors in place before the start of the school semester, with the exception of two partner schools that did not have assignments, and one lead institution that utilized an advisory board of industry professionals. Similar to DoD-assigned mentors, industry mentors worked with students in all schools, excluding the two aforementioned partner schools, on specific problem areas (e.g., assistive or immersive training technologies, systems assurance, among others). Both DoD and industry mentors visited students on campus periodically, attended design reviews, and communicated with teams through email, phone, and video and teleconference. 

Three-quarters of mentor survey respondents gave student projects high rankings for meeting their goals. Mentors reported wanting both formal and informal opportunities to communicate with students; however, scheduling conflicts were cited as the primary barrier to increased engagement. Almost 90% of surveyed students felt that mentor feedback had helped them with their projects. Students recommended that mentors interact with teams earlier in the semester; guide teams towards inquiry-based solutions; and set realistic expectations for projects. Beneficial impacts of mentor involvement were reported by PIs when communication was frequent, specific, and initiated early in the semester. Three-quarters of PIs interviewed in the final survey described mentorships as highly successful and efficient; in one instance, the intervention of a mentor was critical to a partnership, providing much needed clarification and encouragement for a student team that struggled to understand its role in providing systems assurance for another school located several time zones away. 

Through site visits to systems engineering capstone universities in spring 2011, a team of sponsor representatives had identified nine promising practices—approaches that were present at universities where students demonstrated higher levels of communication, analysis, and awareness of the systems engineering process during the site visits. This year, all institutions incorporated three or more of the practices into their capstone courses. A graphical representation of the presence (or lack thereof) of these promising practices among all participating RT-19A universities appears in Table 32. The formation of cross-disciplinary teams, regular involvement with mentors, and attendance by mentors at student design reviews were practices adopted by nearly all of the participating schools. The recommendation to organize the fall lecture-based course, and to commence prototype design in the spring, was not implemented. PIs reported that they worked on DoD problem areas and simultaneously delivered engineering instruction in order to accommodate the academic calendar and also to coordinate research, materials, and personnel.

Another defining characteristic of the RT-19A capstone experience was scaling up to include five partnerships between a total of eleven schools. The report describes in detail capstone partnerships conducted over distance between service academies, civilian schools, and schools with and without systems engineering programs. The five partnership models were eachqualitatively different, with models organized around the co-development of prototypes between teams or the delegation of one part of the systems engineering process to a remotely based team. Another type of partnership dealt exclusively with faculty professional development at two schools and had no direct student team collaboration. The coordination of communication among students in different engineering disciplines and geographical locations was one issue that impacted many of the partnerships that were geographically and temporally distant. Timely funding to support resource and capacity; existing faculty relationships between partnering schools; understanding of differences in school culture; consistent communication between partnering faculty members; complementary knowledge/skills and complementary research interests to ensure that all areas of expertise are covered within the collaboration; and student interest in DoD problem areas all contributed to successful partnerships. PIs reported that partner schools benefited from initial “meet and greet” sessions with students, mentors, and faculty to promote the collaboration; however, such relationships had to be continually encouraged and maintained.

Findings and Recommendations:

Limitations in the data and the many approaches and variables used in the 15 capstone courses prevent statistical correlations with student outcomes and “optimal” course designs. However, the following summary of findings are grounded in data collected through RT-19A: 

Students enjoyed the real-world nature of the projects—both in terms of building an artifact that might be used and in terms of the systems engineering project context (budget constraints, interdisciplinary teams, experts as mentors)—and that they appreciated the contribution that the systems engineering perspective brought to their work. Mentorships and partnerships were an integral part of this year’s effort, and required management by faculty to coordinate communication and increase student content knowledge of systems engineering concepts. 

Systems engineering capstone courses do not appear to have had a major impact on the students’ immediate career plans, although it must be noted that many had their immediate post-college plans in place and that a large majority of both undergraduates and graduate students believed that they might choose careers in systems engineering sometime in the future. Although significant planning toward logistics management, funding, personnel, curriculum design, and other major decisions is required prior to capstone course implementation, the majority of stakeholders in the research study (students, faculty/PIs, and mentors) reported benefits that ranged from increasing students’ exposure to the systems engineering process through the investigation of real-life problem areas; interaction with mentors from a variety of industries; and the facilitation of prototype design in multidisciplinary teams and remote collaborations.

Benefits of these school partnerships include:

• Schools that do not have systems engineering gain access to schools with systems engineering expertise
• Students at schools with only one engineering major are able to work in multidisciplinary teams
• Students have access to a wider variety of student skills and abilities when forming teams
• Students are exposed to a wider diversity of teammates
• Students are exposed to a wider variety of mentors
• Students at civilian schools gain access to military commands and to DoD problem areas
•  Students learn the benefits and difficulties of working at a distance

New challenges introduced by these partnerships include:

• Students at different schools may have different academic calendars, be in different time zones, and therefore have difficulty coordinating schedules, meetings, delivery timelines, etc.
• Students may have difficulty communicating at a distance
• Students who cannot meet face-to-face may have difficulty learning trust, determining roles, and developing collaborations 

The report concludes with some suggestions for how these partnerships might be facilitated on a national scale in the future.