Archives

SERC TALKS: “How is T&E Transforming to Adequately Assess DOD Systems in Complex Operating Environments?”

This was the final Talk of the three-part series on “Test and Evaluation” curated and moderated by: Dr. Laura Freeman, SERC Research Council Member, Director of the Hume Center’s Intelligent Systems Lab and Assistant Dean for Research in the College of Science, and Research Associate Professor in the Department of Statistics at Virginia Tech. You can view the recording of the Talk below. Prior Talks in the series are as follows:

Speaker: Dr. Sandra Hobson, Deputy Director for Strategic Initiatives, Policy and Emerging Technologies, Office of the Director, Operational Test and Evaluation, Office of the Secretary of Defense | CONTACT

Abstract
The Director, Operational Test & Evaluation (DOT&E) is senior advisor to the Secretary of Defense on operational test and evaluation (OT&E) and live fire test and evaluation (LFT&E) in the Department of Defense (DoD). Ultimately, the office of DOT&E is responsible for ensuring testing is adequate to confirm operational effectiveness, suitability, survivability, and lethality of defense systems in combat use. The technologies we are testing today and will in the future require that we rethink classical approaches to T&E. As the National Defense Strategy notes, “We cannot expect success fighting tomorrow’s conflicts with yesterday’s weapons or equipment.” Similarly, we need to modernizing T&E assessment tools, infrastructure, and expertise. In this talk, I will discuss the critical role DOT&E plays in ensuring we provide systems that work to our warfighters. I will discussion how the DOT&E Science and Technology Strategic Plan is focusing on making advancements in current T&E CONOPS in the technology areas of Software, Cybersecurity, and Next-Generation capabilities. I will also discuss the need for an integrated T&E lifecycle and how digital transformation plays a key role in changing how we do business. Finally, transformation will require us to engage the T&E workforce in education and training on new methods. New partnerships with communities pioneering new methods (to including the SERC network in academia) will assist in our transformation.
Bio
Dr. Hobson was appointed to the Senior Executive Service in 2014 as a Deputy Director in the Office of the Director, Operational Test and Evaluation, within the Office of the Secretary of Defense. She is currently the Deputy Director for Strategic Initiatives, Policy and Emerging Technologies focused on advancing operational and live fire test and evaluation tools, methods and capabilities to meet the test and evaluation demands of the future.
Prior to this appointment, Dr. Hobson supported the Director, Operational Test and Evaluation, in two other capacities: first as a Research Staff Member at the Institute for Defense Analyses and then as an Aircraft Systems and Weapons Staff Specialist in the Office of the Director, Operational Test and Evaluation. During these tenures, she provided guidance and direction on the survivability and lethality evaluation of major Department of Defense aircraft and weapons acquisition programs to include the F-35 Joint Strike Fighter, the Air Force KC-46A aerial tanker, and the Navy P-8A multi-mission maritime aircraft.
Earlier in her career, Dr. Hobson worked as a Research Associate resident at the National Institute of Aerospace/NASA Langley Research Center. She defined and executed an interdisciplinary research that included the application of smart materials to actively control boundary layer flow demonstrating the ability to advance aerospace vehicle concepts and technologies. She authored articles, scientific publications and studies to explore the application of this research towards building more fuel efficient aircraft. Other duties included Aerospace Engineering Department lecturer and Airbus North America consultant.
Dr. Hobson earned her Bachelor of Science degree in Aerospace Engineering from the United States Naval Academy and her Doctor of Philosophy degree in Aerospace Engineering from A. James Clark School of Engineering at University of Maryland. Dr. Hobson is a recipient of the National Fellowship for Exceptional Researcher awarded by UNESCO (the United Nations Educational, Scientific and Cultural Organization), and two Secretary of Defense Medals for Meritorious Civilian Service.
DOWNLOAD SLIDES
Moderator: Dr. Laura Freeman, SERC Research Council Member, Director of the Hume Center’s Intelligent Systems Lab and Assistant Dean for Research in the College of Science, and Research Associate Professor in the Department of Statistics at Virginia Tech  |  CONTACT

More information on all SERC Talks can be found here. Thank you!

SERC TALKS: “Progress in Test and Evaluation of AI-enabled Systems in the DoD”

DOWNLOAD SLIDES

Speaker: Dr. Yevgeniya “Jane” Pinelis, Chief, Test, Evaluation, and Assessment, Department of Defense Joint Artificial Intelligence Center (JAIC)  | CONTACT

Abstract
Though the objectives of independent government test and evaluation (T&E) are similar between traditional and AI-enabled systems (AIES), the science, practice, skills, and infrastructure necessary to evaluate these data-dependent systems do require an update. Over the last two years, the DoD Joint AI Center (JAIC) has led the DoD test community in identifying and filling the gaps in T&E of AI. We have made great progress, in partnership with other DoD test stakeholders, academia, and industry. Together, we have put forward T&E frameworks, tools, and contracting approaches. This talk will discuss our existing challenges, current and aspirational solutions, and the way ahead for the DoD T&E of AIES.
Bio
Dr. Jane Pinelis is the Chief of the Test, Evaluation, and Assessment branch at the Department of Defense Joint Artificial Intelligence Center (JAIC). She leads a diverse team of testers and analysts in rigorous test and evaluation (T&E) for JAIC capabilities, as well as development of T&E-specific products and standards that will support testing of AI-enabled systems across the DoD.
Prior to joining the JAIC, Dr. Pinelis served as the Director of Test and Evaluation for USDI’s Algorithmic Warfare Cross-Functional Team, better known as Project Maven. She directed the developmental testing for the AI models, including computer vision, machine translation, facial recognition and natural language processing. Her team developed metrics at various levels of testing for AI capabilities and provided leadership empirically-based recommendations for model fielding. Additionally, she oversaw operational and human-machine teaming testing, and conducted research and outreach to establish standards in T&E of systems using artificial intelligence.
Dr. Pinelis has spent over 10 years working predominantly in the area of defense and national security. She has largely focused on operational test and evaluation, both in support of the service operational testing commands and also at the OSD level. In her previous job as the Test Science Lead at the Institute of Defense Analyses, she managed an interdisciplinary team of scientists supporting the Director and the Chief Scientist of the Department of Operational Test and Evaluation on integration of statistical test design and analysis and data-driven assessments into test and evaluation practice. Before, that, in her assignment at the Marine Corps Operational Test and Evaluation Activity, Dr. Pinelis led the design and analysis of the widely publicized study on the effects of integrating women into combat roles in the Marine Corps. Based on this experience, she co-authored a book, titled “The Experiment of a Lifetime: Doing Science in the Wild for the United States Marine Corps.”
In addition to T&E, Dr. Pinelis has several years of experience leading analyses for the DoD in the areas of wargaming, precision medicine, warfighter mental health, nuclear non-proliferation, and military recruiting and manpower planning.
Her areas of statistical expertise include design and analysis of experiments, quasi-experiments, and observational studies, causal inference, and propensity score methods.
Dr. Pinelis holds a BS in Statistics, Economics, and Mathematics, an MA in Statistics, and a PhD in Statistics, all from the University of Michigan, Ann Arbor.
Moderator: Dr. Laura Freeman, SERC Research Council Member, Director of the Hume Center’s Intelligent Systems Lab and Assistant Dean for Research in the College of Science, and Research Associate Professor in the Department of Statistics at Virginia Tech  |  CONTACT

More information on all SERC Talks can be found here. Thank you!

SERC TALKS: “What Does Test & Evaluation Mean in a Digital Engineering Enabled World?”

DOWNLOAD SLIDES

Speaker: Dr. Darryl K. Ahner, P.E., Director, Scientific Test and Analysis Techniques Center of Excellence (STAT COE), Air Force Institute of Technology (AFIT) | CONTACT

Abstract
Testing has often been looked at as a (un)necessary evil. In order for testing to be value added, it must not take too much time, provide quality information, and provide timely information to appropriate affect systems engineering and acquisition decisions. In this talk we will discuss the demand for testing events, the Scientific Test and Analysis Techniques process (since STAT is mandated in DoD policy anyway), and what test planning may look like in a digital engineering environment. Examples of efficient and effective test planning will also be discussed, both those that were conducted during development and those developed post deployment.
Bio
Dr. Darryl K. Ahner, P.E. is the Director, Scientific Test and Analysis Techniques Center of Excellence (STAT COE) and was instrumental in shaping its graduate-level test engineering operational concept leading to its establishment by the Office of the Secretary of Defense in 2012. He is also a Professor of Operations Research, Program and Curriculum Chair of the Data Analytics and Data Science programs, and Director of the Center for Operational Analysis at the Air Force Institute of Technology. As Director of the STAT COE, he is responsible for executing the Center's mission to provide independent advice and assistance to designated major acquisition programs in the application of scientific test and analysis techniques for test planning and the resulting data analytics. The Center also conducts externally funded applied STAT research, provides STAT education, and develops best practices and case studies in the employment of STAT. Dr. Ahner is a licensed Professional Engineer. He has written over 50 technical articles, made over 70 technical presentations, and has led several technical working groups. He has been a faculty member at the Air Force Institute of Technology (AFIT) since 2010, where his research emphasizes optimization of test & evaluation, autonomous systems, big data analytics, reliability, stochastic models, simulation, and military operations research applications. Prior to AFIT, he taught for five years at the United States Military Academy at West Point and served for three years as the research director for an Army analysis center at the Naval Postgraduate School. In addition to being a member of the Institute for Operations Research and the Management Sciences (INFORMS), and International Test and Evaluation Association (ITEA), Dr. Ahner has been an active member of the Military Operations Research Society (MORS), most recently serving as Vice President for Professional Development and Vice President (Secretary of the Society) on the Executive Committee. A 1990 graduate of the United States Military Academy at West Point, Dr. Ahner retired from the US Army in 2012. During this time he was awarded the Legion of Merit, an award usually reserved for general officers and a Special Act Award for leading modeling, simulation, and analytic research efforts. His academic studies include Rensselaer Polytechnic Institute, where he earned a M.S. in applied mathematics and a M.S. in operations research and statistics, and Boston University where he earned his doctoral degree in systems engineering while a Charles Stark Draper Laboratory Fellow. Dr. Ahner’s awards include the 2020 Colonel Charles A. Stone Award in recognition of an individual who has made outstanding contributions to furthering the AFIT mission, member of the 2019 U.S. Air Force Analysis Team of the Year Honorable Mention, 2018 Journal of Defense Analytics and Logistics Best Paper for “Finding the Fuel of the Arab Spring Fire: A Historical Data Analysis,“ the 2016 Richard H. Barchi Prize for the best paper given at the Military Operations Research Symposium for “Suitability Analysis of Continuous-Use Reliability Growth Projection Models”, the 2017 Affiliate Society Council of Dayton Outstanding Engineers & Scientists Award, the 2014 E. Grosvenor Plowman Award for Best Paper at Council of Supply Chain Management Professionals Global Conference for "Exploring the Use of the Upper-Semi Variance as a Robust Estimator For Calculating Safety Stocks," the 2012 Colonel Charles A. Stone Award, a 2008 Special Act Award including a $5000 Cash Award presented by a 4-star general in recognition of meritorious scientific achievement, and the 2007 Wilbur B. Payne Award in recognition of the best Army analytical technical study.

More information on all SERC Talks can be found here. Thank you!

NOTE TO PARTICIPANTS: SERC Talks broadcasts is now using the Zoom for Government Webinar platform. Prior to the SERC Talk, we encourage all participants to install and test the platform to exercise full virtual capabilities. Installation is NOT required to join the session, as Zoom is accessible via web browser.

Download Zoom for Government client | Zoom for Government FAQ | Zoom for Government: How to join from your browser

More information on all SERC Talks can be found here. Thank you!