Author: Emory Thompson
Overview of the Array of Engineers API
An application programming interface, or API for short, is simply a software bridge that enables two applications to communicate with each other. With a growing specialization and interest in delivering automated testing at Array of Engineers, we quickly discovered how a robust and multi-leveled API was essential for delivering successful automated testing solutions. Taking a project from requirements to fully tested software deliverables can be a daunting task. Therefore, one of the major goals of automated test hardware, test software, and customer-specific test environments is to make this task easier by simplifying software testing and requirements coverage.
For our team at Array of Engineers, developing a multi-leveled API for our Automated Test Controller, Automated Test Software, and various test environments proved to be a challenging climb. However, due to an immensely innovative team, constant collaboration, and some down time as a result of the pandemic, our fully integrated API now brings a very complex test functionality to a simplified level, making it easier for testing and requirements coverage. For those other teams out there who are struggling with their test APIs, we would like to share what worked well for us.
How it Works
The Array of Engineers' API system for our Automated Test Controller (ATC) is multi-leveled to provide abstraction and simplification through an integrated project harness, which is initialized at the start of each test script. The API sets up all of the basic functionality and gives us access to high level functions to perform various tasks.
Here is a breakout of how we use and access our API at basic levels:
FPGA IP Cores: At the lowest level we have the FPGA and IP cores. These IP cores allow us to perform basic tasks such as:
Send and receive CAN/I2C/SPI/UART protocols and control the settings associated with them (such as baud rate or stop bit)
Turn on/off and define the output levels for a digital-to-analog converter (DAC)
Collect analog-to-digital converter (ADC) data including collection rate data size and multiplex connections
RTOS: These FPGA cores are controlled by a RTOS processor to perform both simple and more complex tasks such as:
Send a message repeatedly
Start collecting data at a specific time
Collect data (both ADC and communication) into a cohesive log file for analysis
Respond to stimuli. For example, we set up the RTOS to respond based on incoming data
Automated Test Controller (ATC) Generic Harness: The RTOS processor in turn is controlled via ethernet communication, with specific functionality for how to accomplish this provided via the generic ATC harness. This harness is written in python and includes functionality to:
Set up the ATC ports and settings
Retrieve communication logs back for analysis
Perform data analysis is supported via template parsing functionality that allows us to use a config file to describe what the raw data means
Project Harness: The project harness will then extend from the generic harness and is designed based on the specific needs of the system to be tested. The project harness includes an initialization that performs the basic system setup such as setting up the various communication ports, as well as creating any repeating messages and triggered reactions (such as those described above). This harness also gives access to functions written and named to align with how one might physically interact with the device. Functions such as foot_pedal.connect(True) or user_interface.press_button(button_name).
Test Scripts: Finally we use the project harness to create a test script to accomplish a specific goal usually with the intent of providing coverage for a given requirement. Also included within this test would be tracing statements. These statements help ensure that we claim the coverage we are achieving and we are clear about what we are covering in the test. It also helps to ensure we cover each step along the way, thus bringing us full circle from the requirements, through the testing solution apparatus, and finally to functional tests with clear and explicit coverage of those requirements.
Example Project Harness Functions
Ex1. foot_pedal.connect(True):
In this example, a device foot pedal is set up as an RS232 communication line. When the function is called the ATC will begin communicating to show that a foot pedal is now connected. This process includes a call and response initialization with the device under test (the call and response process is created as part of the harness initialization) that occurs to show that the foot pedal is connected and working properly. At this point, we are now simulating a foot pedal via RS-232 and sending a heartbeat message every 50ms stating that no buttons are pressed.
Ex2. foot_pedal.press_button(button_name=“pedal_left”, button_value=255):
This example uses the same foot pedal described in example 1 with 3 foot pedals each of which can be pressed to a value between 0 and 255 where 0 is not pressed and 255 is pressed all the way to the floor. In this case, the heartbeat message contains the data telling us what buttons are being pressed and how much. This data is stored on the ATC and compiled by the RTOS to send via FPGA every 50ms. So in this case the project harness contains configuration data telling us that if we want to manipulate “pedal_left” we can change the value of byte 1 in a given message stored on the ATC. So we send a message telling the ATC to update the data associated with the foot pedal press data from “00 00 00” (all buttons not pressed) to “FF 00 00” (left pedal fully pressed). From here on until changed again, the RTOS will use this updated value whenever it sends out the heartbeat message and thus simulate that the left pedal is pressed.
Ex3. user_interface.press_button(button_name):
In this example whenever we wish to “press a button” via the user interface we must send a CAN message with ID 1000 that includes 1 byte of information that is the button value.
During harness initialization we create a CAN byte frame that includes a single byte of data that is the button value. This CAN frame is NOT sent at this time.
button_name is the name of the of a button that is associated with a lookup table in the config file in the project harness that then is associated with a specific value, for example button_name=”volume_up” which is then found in the lookup table and is shown to be the value 23.
At this point we send an update to the ATC to change the value of the button that we want to send to 23.
Then we update the ATC to tell it to send the CAN Frame with that information one time.
Ex4. adc.capture (“ON”):
In this example, we have a function that comes directly from the generic harness as beginning the ADC capture is likely to be necessary and applies in much the same way across all projects where it simply means to begin capturing ADC data. In this case, the ADC module will need to be included and initialized by the project harness to ensure capture settings are what we are looking for, but the function to begin capturing data would not be unique.
Example Tracing Statements:
The following are several examples of how coverage could be claimed for various types of requirements. All trace statements in the test will result in an object being created in the results XHTML file that can be retrieved in bulk from all tests performed for requirements coverage analysis (RCA).
Ex1. Req-1: The SW shall set foot_pedal status to True when a foot pedal is connected otherwise set foot_pedal status to False
In this situation the foot_pedal status corresponds to a specific bit within a CAN message that we can view the state of. So we create a test claiming Req-1: Full Coverage where we check that the foot pedal status is false on initialization, then we connect a simulated foot pedal (akin to example shown earlier) and ensure it goes to true then we disconnect the foot pedal and show it goes to false.
Ex2. Req-2: The SW shall set GPIO to True when on screen 1 or screen 2
In this situation we only need to cover the true case, but it is not efficient to have a single test that goes to both screen 1 and screen 2. So we end up with 2 tests; Req-2: Partial Coverage: Covers for screen 1 and Req-2: Partial Coverage: Covers for screen 2. For this case, the coverage is split but it is still easy to keep track of for RCA.
Challenges:
There were a lot of challenges and pitfalls along the way with developing and integrating an API capable of supporting our data acquisition and automated testing process. Whether it be that a given FPGA IP Core turned out to be more finicky than we expected, or ensuring data mapping worked correctly for when the ADC transmits hundreds of MB of data into RAM in a fraction of a second, or ensuring the RTOS would be able to perform complex tasks without requiring additional input from the host computer, our team came together to hurdle numerous challenges. In the end, these were all solvable problems and we are extremely excited to see it all come together.
Benefits and Strengths:
The main strength of this sort of harness environment and working with Array of Engineers is the flexibility and end product simplicity. It doesn’t take someone steeped in product knowledge or programming to grasp that press_button(“home”) means we press the home button. And by abstracting all the way to these types of simple functions we create a system that empowers new team members to ramp up to development quickly, with simple language. This is achieved while still retaining access to the underlying core functions that give your experts the freedom and flexibility to focus on more complex issues.
And while it might be simple to work with, the vast capabilities allow for numerous input and output device simulations across various communication protocols and up to 14 ADCs and 4 DACs for handling the most complex of devices under test.
What Comes Next:
RCA Tool Suite: A suite of tools to help pull all of those trace statements from the test results to allow for easy analysis. This will also help with tasks like ensuring that all tests were run on the correct software and hardware.
ATC Lite: A digital only version of the ATC for situations where analog measurements might not be necessary.
ATC-Linker: In extreme situations you might need to be able to connect to 4 CAN networks, or 8 RS-232 lines, or 60 GPIOs or 25 ADCs or … For these types of extreme cases we are excited to be working on a method to enable multiple ATCs to be synchronized together for more connection and simulation capacity.
Emory Thompson is a staff software engineer at Array of Engineers, where he performs software development, testing, and verification for various partners in the aerospace and medical device industries. Emory has over 10 years of engineering experience and plays a vital role in developing and enhancing automated testing systems at Array of Engineers.