Authors:
Michael Lingg, PhD, Array of Engineers, Timothy J Kushnier, DEVCOM GVSC SEC, Rodolfo Proenza, DEVCOM GVSC SEC, Howard Paul, Array of Engineers, Brendan Grimes, Array of Engineers, Emory Thompson, Array of Engineers
Abstract
Hardware/software integrated system ensures a system will operate as intended
in the same configuration it will be used in the field. Manual system testing can be a very slow and error prone process, as well as being incapable of testing interfaces that humans cannot interact with. Many existing solutions exist to introduce test hardware into the loop for verifying systems, but most of these solutions provide a separate component for each hardware interface. This paper presents an approach for a single integrated system that can test all hardware interfaces of a system under test, managed by a single controller. This test system provides the capability to abstract away the hardware being tested so a test developer can develop tests while only understanding the manual interfaces of the system being tested. We show that this approach can provide a significant acceleration to the time to execute tests, as well as improving the reliability, and consistency of the tests.
Citation: M. Lingg, T. Kushnier, R. Proenza, H. Paul, B. Grimes, E. Thompson, ”Multi-level Hardware-In-The-Loop Test API For Hardware-Software Integration Testing,” In Proceedings of the Ground Vehicle Systems Engineering and Technology Symposium (GVSETS), NDIA, Novi, MI, Aug. 16-18, 2022
1. Introduction
Integrated hardware/software systems are invaluable in modern equipment. While pure hardware systems often require complete replacement to expand or add new capabilities, integrated hardware/software systems can adapt to new features being added to the system with updated software. This update capability provides two primary benefits. First the ease of adding new features can extend the life of the equipment, which may otherwise become obsolete as technology advances. Second, software updates can allow a system to adapt to a rapidly changing environment. Over the air software updates allow equipment to be repurposed, or provided extended capabilities, in the field [1].
The flexibility provided by the integrated hardware/software system can lead to increased complexity when testing the system. During initial development, software unit testing may be performed on a PC without the actual hardware, and some hardware functionality can be tested with a basic test stand. However these separate test methods often fail to find bugs that show up when the software and hardware are integrated together. Integration testing of an integrated hardware/software system can be performed by a human working with the interface and observing the outputs of the system, but these systems increasingly have multiple components communicating with each other. In most cases, the interactions between multiple components use hardware communication protocols that require specific hardware to interface with, delaying testing until all systems are available to work together, or requiring a hardware test interface. Automated hardware testing can allow for a greater number of tests to be executed in a shorter period of time, allowing for a greater number of boundary conditions to be tested, or simply reducing the amount of time
required for testing.
Automated hardware in the loop testing can allow for earlier testing during development. As soon as hardware is available, the hardware in the loop tests can be ready to run as the software implementation becomes more complete. This earlier testing allows the developers to receive feedback on their implementation well before the end of development. As development continues, the test suite can continue to be used for regression analysis, to ensure that while new features are implemented correctly, existing functionality is not broken. The same hardware in the loop test system can also be adapted to be used on the production line, ensuring all functionality works as expected on each unit as it is produced. Finally the automated hardware in the loop testing can be used to test new features on an identical hardware software integration system back at the shop, before deploying to the field. The updates can be quickly tested for regression to ensure all features work as expected, providing high confidence that deploying the software will have the desired effect.
The focus of this paper is to provide a hardware in the loop test approach that reduces the time necessary to fully test an integrated hardware/software system, with a particular focus on reducing the time, and technical knowledge required, to develop tests. To provide a proper framework we will next look at three examples for testing components of an integrated hardware software system, and how test developers and the rest of the system would interact with these components. Then we will look at existing hardware in the loop testing solutions. Finally we will present a new all in one hardware in the loop test approach. Then look at test performance results showing the benefits of this system in time to execute a suite of tests, and reducing weeks long manual tests to days. Finally we will discus the consistency provided by the test interface, and automation of test execution.
2. Background
First we will look at some possible components of an integrated hardware/software system to be tested. In these examples we consider how these components work in a complete system, with no test hardware connected to the system under test. After defining how each component interacts with the system controller, an example system with all components will be described, including manual test examples.
2.1 Simulating a Multi-Function Display
The first component looked at is a multi-function display (MFD). In the complete system, inputs from the MFD would lead to the system activating other components of the system, or displaying data from other components of the system, for example vehicle climate control. The MFD communicates with the system controller via a Controller Area Network (CAN) bus [2]. The system controller will send a periodic Heartbeat message that includes the current screen the MFD is expected to display, and the system will go into a fault mode if the MFD does not acknowledge the heartbeat. Any button press on the MFD is sent as a single message from the MFD over the CAN bus. The button press message will include the identifier of the button pressed.
A basic test for this example would start with powering up the system with the MFD attached, then verify after a time limit that a fault is not displayed on the screen. Next a button would be pressed, then the user will verify the display changes to the appropriate screen within a time limit.
2.2 Testing Control of a Speaker
The second component for the system is a speaker. This component is rather simple on the surface, the speaker is commanded by the system to output a given frequency at a given volume. The manual test is extremely simple, if rather imprecise. Have a human listen to the speaker and decide if it sounds right. Another possible option is testing the speaker sound output with a microphone and analyzing the signal. The microphone would require a fairly controlled environment if a precise measurement of the sound output is desired.
2.3 Testing Reading of a Thermocouple
The final component is a thermocouple that outputs a voltage to be translated by the system into a temperature. There are limited simple test methods to test this with any accuracy. A thermometer can be placed next to the thermocouple to measure the present temperature, and a fan, or heater, could be used to decrease or increase the temperature, but the thermocouple and thermometer may change temperature at different rates. An expensive temperature chamber can test the system with high precision, but the size of the system under test may limit practical use of a temperature chamber.
2.4 Combined System
With these components we can define how the system works as a whole to act as a high temperature alarm. The temperature sensor provides the current temperature to the system, and the speaker sounds an alarm if a high temperature condition exists. The MFD provides an alarm menu system to set the temperature threshold for the alarm condition.
A simple manual test procedure might be to power up the system with all hardware connected, verify the main menu is displayed on the MFD, not a Fault menu. Press the button to enter the alarm menu. Press the increase or decrease temperature button until the temperature threshold is one unit below the current temperature, and verify the alarm
sounds. Press the increase temperature button twice, and verify the alarm does not sound.
2.5 Existing Solutions
While human testing of systems often has low up front costs, the time required for human testing can quickly add up, and often humans are simply not equipped to properly test many digital and analog systems. For example a human can verify the menu switching for the MFD, but if the MFD is disconnected a human cannot determine if, or when, the system entered fault mode. Simple hardware solutions can be tailored to specific tests. An oscilloscope can be connected to the CAN bus, but then the human tester will need to decode the CAN signals and most oscilloscopes have a limited capture window. A better solution for this test application is hardware that can read the digital signal, and includes the capability to process and decode the data.
National Instruments (NI) provides an excellent range of hardware test products [3]. Among other products is a CAN analyzer [4] that could work nicely for testing the MFD example. Total Phase also produces a similar analyzer [5]. NI also provides the LabVIEW [6] program that allows their hardware products to be programmed for specific operation, and to analyze data read by their hardware.
As powerful as these existing products are, they tend to focus on the signals that the NI
product is designed to work with. To use these devices, a test developer will need to understand the hardware signals involved. For our MFD example, understanding what the system should do from a user perspective is not sufficient, the test developer will also need to understand what CAN messages are being communicated, and sometimes details of
how the CAN protocol works. Additionally while some of these existing hardware test products can be connected together to simulate, and/or read, multiple signals in parallel, the products are typically sold separately and require the test developer to determine how to connect the devices and program them to work together.
An alternative is creating an integrated system that manages all necessary hardware interfaces in a single unit. Benefiting this approach, costs of producing circuit boards have decreased significantly. Today a circuit board capable of connecting to all sensors and communications ports of a small to medium system can be printed, even in small quantities, for a unit cost equivalent to tens of man hours of a human tester’s time. This makes integrating all of the necessary elements to test a system into a single board, that includes its own processing unit, very cost effective. With all of the testing components integrated together, a test scripting interface can be developed that abstracts away the low level details of the components being tested. This abstraction means tests can be developed that only requires high level knowledge of how the system works, no longer requiring test developers to understand the protocols involved. Next the details of such a system will be discussed.
3. Methods
In this section a Multi-level Hardware-In-the-Loop (MHIL) test system will be described, from here on referre