Troubleshooting Automation Software

I can’t tell you how many times I have received an email or text message saying, “It doesn’t work!” Well, what doesn’t work? Can you give me any details? So I thought I would put together some basic troubleshooting steps followed by what you should add to your cryptic message of “It doesn’t work!”

I want to start by saying, computers do EXACTLY what they were instructed to do each and EVERY TIME. They follow the instructions they were given. Computers get old and can break, but software doesn’t wear out.

1. Repeat Tests – Most errors are found when a test point fails or the results don’t look right. When this happens, the first thing you should do is repeat the test a few times. If the results are consistent, then you know the error is systematic. If the results are inconsistent, then the problem is random and will be harder to track down and find a solution.
2. Check Standards − The next step is always to “check your standards!” The more accurate the reference standard, the more things can go wrong. Double-checking your reference standard should be common practice. Checking a reference standard is a pretty simple task. Simply find a comparable standard in the lab and compare the two. Grab a comparable digital multimeter and measure the output to see if your reference standard is a calibrator outputting DC voltage. This isn’t a calibration; this is just a sanity check, and it helps identify an error before it affects several calibrations.
3. Check Settings − Next, check the equipment settings against the manual. This is usually an easy step if the software stopped on the test point in question. If the software stops, then you should be able to turn the output on and verify the measurement. At this time, you can check the instrument-specific setting against the manual.
4. Record if Possible − You can also start an I/O trace and repeat the test. This will allow you to see the commands getting sent to the instruments. Most I/O trace tools will allow you to save the trace and send it with a support ticket. Another option is to record a video of the UUT and the reference standards as the calibration runs. The I/O trace shows the commands paired with a video of the equipment can really make remote troubleshooting easy.
5. Check Specs − The next check should be the test limits. Check the limits in the automation to the written calibration procedure. Then, check the written procedure test limits against the equipment’s published specifications.

Hopefully, after following these five steps, the problem has been identified. But even if the exact problem is still unknown, you have gathered enough information to formulate an intelligent email that will help get to a resolution.

When you write the email, include the results of the repeated tests. If you just say it fails, I will respond with “equipment sometimes fails calibration.” If you say, “I ran it several times, and the numbers are all over the place,” or “consistently low,” then I have an idea of what to look for in the code. If your email says, “I checked the manual, and the device settings are wrong; the instruments need to be configured as follows,” then changes can be made quickly, and an update is usually that day. The same goes for errors in the test limits. Don’t assume the manufacturer’s manual’s calculated test limits are correct – I can’t tell you how many errors we have found.

Finally, if you have a video or I/O trace, you can attach it to the email. That makes it easier to track down problems. I hope this helps my customers and other automation engineers get the data they need from their users.