Archives

INCOSE Human Systems Integration Conference 2021

The Human Systems Integration Conference 2021 (HSI2021), an INCOSE Conference, will be held virtually from 16 to 19 November, 2021. This first virtual edition of the HSI INCOSE Conference brings a fantastic environment to learn new things and to discuss your work with the worldwide Human Systems Integration community. Working together with the International Ergonomics Association (IEA) and Association for Computing Machinery (ACM) societies provides the opportunity for networking with members of other related professional societies, bringing a richness to the experience.

General Chair of INCOSE HSI2021, Guy André Boy
General Co-Chair representing IEA, Michelle Robertson
Check the event website regularly for the latest news regarding this event.

SERC TALKS: “Progress in Test and Evaluation of AI-enabled Systems in the DoD”

DOWNLOAD SLIDES

Speaker: Dr. Yevgeniya “Jane” Pinelis, Chief, Test, Evaluation, and Assessment, Department of Defense Joint Artificial Intelligence Center (JAIC)  | la***********@vt.edu;SE*******@st*****.edu?subject=SERC Talks w/ Dr. Jane Pinelis – October 29″>CONTACT

Abstract
Though the objectives of independent government test and evaluation (T&E) are similar between traditional and AI-enabled systems (AIES), the science, practice, skills, and infrastructure necessary to evaluate these data-dependent systems do require an update. Over the last two years, the DoD Joint AI Center (JAIC) has led the DoD test community in identifying and filling the gaps in T&E of AI. We have made great progress, in partnership with other DoD test stakeholders, academia, and industry. Together, we have put forward T&E frameworks, tools, and contracting approaches. This talk will discuss our existing challenges, current and aspirational solutions, and the way ahead for the DoD T&E of AIES.
Bio
Dr. Jane Pinelis is the Chief of the Test, Evaluation, and Assessment branch at the Department of Defense Joint Artificial Intelligence Center (JAIC). She leads a diverse team of testers and analysts in rigorous test and evaluation (T&E) for JAIC capabilities, as well as development of T&E-specific products and standards that will support testing of AI-enabled systems across the DoD.
Prior to joining the JAIC, Dr. Pinelis served as the Director of Test and Evaluation for USDI’s Algorithmic Warfare Cross-Functional Team, better known as Project Maven. She directed the developmental testing for the AI models, including computer vision, machine translation, facial recognition and natural language processing. Her team developed metrics at various levels of testing for AI capabilities and provided leadership empirically-based recommendations for model fielding. Additionally, she oversaw operational and human-machine teaming testing, and conducted research and outreach to establish standards in T&E of systems using artificial intelligence.
Dr. Pinelis has spent over 10 years working predominantly in the area of defense and national security. She has largely focused on operational test and evaluation, both in support of the service operational testing commands and also at the OSD level. In her previous job as the Test Science Lead at the Institute of Defense Analyses, she managed an interdisciplinary team of scientists supporting the Director and the Chief Scientist of the Department of Operational Test and Evaluation on integration of statistical test design and analysis and data-driven assessments into test and evaluation practice. Before, that, in her assignment at the Marine Corps Operational Test and Evaluation Activity, Dr. Pinelis led the design and analysis of the widely publicized study on the effects of integrating women into combat roles in the Marine Corps. Based on this experience, she co-authored a book, titled “The Experiment of a Lifetime: Doing Science in the Wild for the United States Marine Corps.”
In addition to T&E, Dr. Pinelis has several years of experience leading analyses for the DoD in the areas of wargaming, precision medicine, warfighter mental health, nuclear non-proliferation, and military recruiting and manpower planning.
Her areas of statistical expertise include design and analysis of experiments, quasi-experiments, and observational studies, causal inference, and propensity score methods.
Dr. Pinelis holds a BS in Statistics, Economics, and Mathematics, an MA in Statistics, and a PhD in Statistics, all from the University of Michigan, Ann Arbor.
Moderator: Dr. Laura Freeman, SERC Research Council Member, Director of the Hume Center’s Intelligent Systems Lab and Assistant Dean for Research in the College of Science, and Research Associate Professor in the Department of Statistics at Virginia Tech  |  SE*******@st*****.edu?subject=SERC Talks w/ Darryl Ahner – August 18″>CONTACT

More information on all SERC Talks can be found here. Thank you!

AI4SE and SE4AI Workshop 2021

Sponsored & organized by U.S. Army DEVCOM Armaments Center (AC) Systems Engineering Directorate and Systems Engineering Research Center (SERC)

The objective of this workshop was to bring together key stakeholders from across industry, government and academia to explore specific outcomes that resulted from the initial 2020 meeting. The agenda for the October 2021 workshop was developed with the intention to curate panel discussions and facilitate group breakout sessions that produce actionable applications of Artificial Intelligence for Systems Engineering (AI4SE) and Systems Engineering for Artificial Intelligence (SE4AI) as well as relevant research ideas that align with DEVCOM AC and SERC AI/SE priorities.

DOWNLOAD AGENDA

TOPICS OF INTEREST:

  • AI/ML Technology Evolution 
  • Augmented Intelligence, Cognitive Assistants and Decision Aiding Tools
  • Theoretical Foundations of Automation & Human-Machine Teaming 
  • Augmented Engineering, Using AI to Augment Engineering Design 
  • Digital Engineering Infrastructure and Methods in Support of AI
  • Advances in Explainable AI 
  • Advances to Address Cognitive Bias in AI 
  • Uncertainty Quantification and Relation to Safety Boundaries in AI
  • V&V and T&E of Learning Based Systems 
  • Applications of Digital Twins to Improve Machine Learning
  • Data Collection and Curation for AI in Cyberphysical Systems
  • AI/ML Applications That Address Complexity of Creating, Editing, and Annotating Models
  • Workforce Development for AI4SE/SE4AI
  • Other Current Applications of AI and ML for SE 
  • Other Current Applications of SE for AI and ML

Keynotes

October 20

Dr. Mica Endsley - President, SA Technologies

Dr. Mica Endsley is President of SA Technologies and is the former Chief Scientist for the US Air Force. She has also held the positions of Visiting Associate Professor at MIT in the Department of Aeronautics and Astronautics and Associate Professor of Industrial Engineering at Texas Tech University. She was formerly an Engineering Specialist at the Northrop Corporation. Dr. Endsley is a Fellow and Past-President of the Human Factors and Ergonomics Society. She received a Ph.D. in Industrial and Systems Engineering from the University of Southern California. Dr. Endsley is a recognized world leader in the design, development and evaluation of systems to support human situation awareness (SA) and decision-making, and the integration of humans and automation. She has authored over 200 scientific articles and is the co-author of Analysis and Measurement of Situation Awareness and Designing for Situation Awareness.

October 21

Ms. Stephanie Possehl - Acting Deputy Director for Engineering and Director for Engineering Policy and Systems, OUSD(R&E)

Ms. Stephanie Possehl is the Acting Deputy Director for Engineering (DD, ENG) and Director of Engineering Policy and Systems within the Office of the Under Secretary of Defense for Research and Engineering.  She is serving as the Department of Defense’s Chief Engineer for Advanced Capabilities.  She sets engineering policy for the DoD with a focus on embracing modern engineering best practices and driving technical rigor.  She also leads DoD-wide communities of practice dedicated to advancing engineering practice and championing efforts such as digital engineering, modular open systems approach, and many traditional specialty engineering focus areas.  She leads mission integration management and supports Major Defense Acquisition Programs through engineering and test assessments.  She is the functional leader for the Engineering & Technical Management and Test & Evaluation acquisition workforces.  She also has responsibility for the Joint Hypersonics Transition Office and Strategic Intelligence Analysis Cell.

Ms. Possehl has more than 25 years of Federal service with the Department of Defense, including program management positions with the Missile Defense Agency (MDA) supporting Aegis Ballistic Missile Defense and the Sea-Based Terminal program.  Before joining MDA, she taught senior-level program management and systems engineering courses at the Defense Acquisition University.  Her previous engineering and project management assignments were in support of the U.S. Navy STANDARD Missile Program Office and the Naval Surface Warfare Center/Dahlgren Division.  She began her career as a contractor providing engineering support to DoD programs.

Ms. Possehl holds a bachelor of science in mechanical engineering and a master of science in technology management, both from the University of Maryland.  She is a member of the Acquisition Professional Community and is DAWIA Level 3 certified in Program Management and Engineering.  She received the MDA Medal for Exceptional Civilian Service (2020), Navy Superior Civilian Service Award (2018), and Navy Meritorious Civilian Service Award (2015).

October 20, 2021 | KEYNOTE | SERC AI ROADMAP | PANEL: THE AI DISRUPTION

KEYNOTE: New Challenges for Systems Engineering: AI Systems in a Human World
Dr. Mica Endsley, SA Technologies

SERC AI Roadmap
Mr. Tom McDermott, SERC at Stevens Institute of Technology

DOWNLOAD PRESENTATION

PANEL | The AI Disruption: The Challenges for Specialty Engineering Posed by AI
Led by Dr. Jason Cook, US Army DEVCOM Armaments Center

DOWNLOAD PRESENTATION

October 20, 2021 | TRACK 1: V&V OF AI SYSTEMS

Test and Evaluation of Reinforcement Learning
Dr. Ali Raz, George Mason University
DOWNLOAD PRESENTATION

Bringing Reliability, Prognostics, and Testing to Machine Learning
Dr. Tyler Cody, Intelligent Systems Lab, Virginia Tech, Hume Center for National Security and Intelligence

DOWNLOAD PRESENTATION

Concept-of-Operations for Testing AI Systems: Fast-time Emergent Scenario Simulation (FTESS)
Dr. Lance Sherry, Center for Air Transportation Systems Research at George Mason University

DOWNLOAD PRESENTATION

Systems Theoretic T&E Framework for Learning Based Systems
Mr. Paul Wach, Virginia Tech

DOWNLOAD PRESENTATION

Towards a Tool for Composing, Evaluating, and Tracking Systems Validation Arguments
Dr. Daniel Shapiro and Dr. Bryan Mesmer, The University of Alabama in Huntsville

DOWNLOAD PRESENTATION

Army Software Suitability Statement Criteria for AI
Mr. Adam Hilburn, US Army ACC-NJ, Picatinny Arsenal

DOWNLOAD PRESENTATION

October 20, 2021 | TRACK 2: LIFECYCLE ADAPTATION & RESILIENCE

Analyzing Cyber Attack Impacts and Defense Strategies Using Machine Learning
Mr. Daniel Colvett, University of Alabama in Huntsville
DOWNLOAD PRESENTATION

Using Natural Language Processing to Extract Science Traceability from Mission Concept Documentation
Mr. Benjamin Simpson, Department of Aerospace Engineering, Texas A&M University
DOWNLOAD PRESENTATION

Cognitive Assistance for Problem Formulation in Tradespace Exploration
Mr. Gabriel Apaza, Texas A&M University
DOWNLOAD PRESENTATION

SE4AI: Design of Neural Network Architectures to Support AI and Machine Learning Formalisms Working Side-by-Side as a Team
Dr. Mark Blackburn, Stevens Institute of Technology
DOWNLOAD PRESENTATION

Creating a Digital Twin of an Insider Threat Detection Enterprise Using Model Based Systems Engineering
Mr. James Lee, George Mason University
DOWNLOAD PRESENTATION

October 21, 2021 | KEYNOTE | DEVCOM AC SED | PANEL: WORKFORCE IN AI

KEYNOTE: Insights on AI and SE
Ms. Stephanie Possehl, Acting Deputy Director for Engineering and Director for Engineering Policy and Systems, OUSD(R&E)

[No slides]

Systems Engineering, Artificial Intelligence, and Machine Learning at the Systems Engineering Directorate
Mr. Roshan Patel, US Army DEVCOM Armaments Center
DOWNLOAD PRESENTATION

PANEL: AI and the Engineering Workforce
Led by Mr. Tom McDermott, SERC

October 21, 2021 | TRACK 3: AUGMENTED ENGINEERING

TEXT2MODEL: A Conceptual Model-Based Knowledge and Intelligence Management System
Professor Dov Dori, Technion, Israel Institute of Technology
DOWNLOAD PRESENTATION

Using Natural Language Processing (NLP) in Requirement Engineering: Extracting Models From Text
Dr. Carlo Lipizzi, Stevens Institute of Technology
DOWNLOAD PRESENTATION

Augmented Intelligence for Competitively Bid Proposal Evaluation
Dr. Shane Esola, US Army DEVCOM Armaments Center
DOWNLOAD PRESENTATION

Creating MBSE Model Content from Non-Standard Sources
Mr. Matthew Cotter, The MITRE Corporation
DOWNLOAD PRESENTATION

Executive Hosts:

  • Dr. Dinesh Verma – SERC Executive Director, Stevens Institute of Technology
  • Dr. Jason Cook – US Army DEVCOM Armaments Center
  • Dr. Dan DeLaurentis– SERC Chief Scientist, Purdue University

Research Highlight: Improving the STEM Pipeline

Dr. Ruth David, Stevens Institute of Technology
Jeff Wilcox, Stevens Institute of Technology
Dr. Dinesh Verma, Stevens Institute of Technology
View Project Page

The increased integration of technology in our lives and greater need for individuals in the workforce who are able to take on STEM (science, technology, engineering, and mathematics)-related jobs is well documented. The Department of Defense (DoD) formalized its strategic activities around STEM education with the foundation with Defense STEM Education Consortium (DSEC) in 2019 to cultivate a diverse and sustainable STEM talent pool ready to serve the nation and evolve the DoD’s competitive edge. [1]  There have likewise been a number of STEM initiatives in the US at the national, state, and local levels. Despite the attention to and the growth seen in some STEM areas, however, as of 2020 only about 7% of the 153-million member US workforce worked in STEM fields. [2]

To explore some of the persistent issues impacting STEM education and career choices, the Systems Engineering Research Center (SERC) has launched new research that will help to shape the STEM pipeline in the United States. Led by SERC/AIRC (Acquisition Innovation Research Center) Fellows Dr. Ruth David and Mr. Jeff Wilcox, this research task will focus on developing collaborative DoD-industry activities that will: create a technical workforce able to receive the security clearance needed to meet defense missions; support educational opportunities for defense sector personnel in both government and industry; and increase educational opportunities for veterans and military dependents.

Dr. Dinesh Verma, Executive Director of the SERC and AIRC, explained, “As we look into the future, our dependence on kids opting for academic preparation and careers in STEM becomes more and more important to us given the nature of jobs in the future and our national security needs. It is a critical endeavor for us as a nation to ensure more of our kids have the option and are comfortable pursuing a STEM major and STEM-oriented career.”

To address this, the team will focus on activities based on metrics and education best practices to ensure maximum effectiveness. The team will look at existing incentives and propose new incentives for the government and defense industrial base to participate in STEM education and outreach, and workforce development opportunities. The team will also explore how collaborative STEM education and outreach activities support long-term employee retention or occupational retraining, and ways to measure the success of the proposed initiatives.

The goal for this research is to create a small number of high-impact, realizable recommendations that are easy to implement and actually used by the DoD and the DIB. In particular, the team hopes to identify ways in which collaboration allows the DoD and DIB to deliver impacts that each may not be able to achieve separately.

To learn more about this task, click HERE.

[1] Best. 2021. “Department of Defense STEM Education and Outreach.” Building Engineering & Science Talent (BEST) website. Accessed 6 October 2021. Available at: https://www.bestworkforce.org/what-we-do
[2] US Bureau of Labor Statistics. 2021. “Data: Employment in STEM Occupations”. Accessed 4 Oct 2021. Available at: https://www.bls.gov/emp/tables/stem-employment.htm

Research Highlight: AIRC Launches Research in Agile Acquisition

Dr. Gregg Vesonder, Stevens Institute of Technology
Dr. Nicole Hutchison, Stevens Institute of Technology

The Office of the Under Secretary of Defense for Acquisition & Sustainment (OUSD(A&S)) has created a series of related tasks in response to Congressional requirements to study and develop policy options and recommendations on how the Department of Defense (DoD) can use agile program and project management concepts in non-software acquisition programs. As part of this effort, Drs. Gregg Vesonder and Nicole Hutchison are leading the development of a report that describes the history of agile processes as viewed by multiple disciplines and industries that documents the agile journey-related experiences of organizations. This task will distill core agile process-related principles from across technical and process domains and industries, and will provide guidance to defense acquisition programs and processes beyond software. The resulting report will define “what good looks like”, the primary indicators of progress, and the steps the DoD might take to expand agile methods across its portfolio.

The task will focus on exploring how the principles of Development and Operations (DevOps) can be used in the DoD. DevOps has made a major difference in software development across the world and most especially in how it is deployed. Part of the approach of DevOps is doing things in small increments: releasing smaller pieces of functional software and testing these thoroughly and automatically. This task will help transition highly successful DevOps work into the DoD framework by looking at how procurement and acquisition is – and can be – done.

When completed and transitioned, the results of this task could have a major impact on DoD acquisitions. “If we can make [the process] more efficient and much more responsive, it would result in higher quality and quicker acquisition of weapon systems,” says Dr. Vesonder. It also could have a positive effect on the DoD industrial base and broader industry. Dr. Vesonder further explained, “This could create virtuous feedback: DoD discovers how to apply DevOps in acquisition and that could be fed back into industry, who could use the lessons learned to address and improve upon unique problems. In DevOps, sharing and iteration are critically important – as you share, you get stronger.”

To learn more about the AIRC, click HERE.