How do you automate tests against real embedded hardware?

Tools
1

How do you automate tests against real embedded hardware? (genuinely curious, not selling anything)

I’m building a hardware-in-the-loop test tool and want to understand how people are actually solving this before I go too far down the wrong path.

Specifically: a box (Raspberry Pi-based) that sits on your bench or in a rack, exposes UART / I²C / SPI / GPIO / JTAG to the network, and lets you drive it from Robot Framework keywords over gRPC — so you can run hardware tests from CI the same way you run software tests.

Before I build too much, I’d love honest answers to a few questions:

  1. How do you test against real hardware today? (Serial scripts? LabGrid? pytest + custom fixtures? Something else entirely?)
  2. What’s the most painful part of that setup? Flaky connections, per-board glue code, only works on one dev’s machine, something else?
  3. Have you tried Robot Framework for hardware testing? If yes — what stopped you or slowed you down?
  4. If a networked, rack-mountable box just worked with RF out of the box — would your team pay for it? Ballpark: ~€480/box/year. Who would actually approve that purchase — you, your manager, procurement?
  5. What would make you NOT buy something like this? (This one is the most useful for me.)

Feel free to answer here or shoot me a PM if you’d rather talk privately. No pitch, no follow-up spam — I’m in the “figure out if this is worth building” stage and your input is genuinely the most valuable thing right now.

Thanks in advance.
Tim

2

Hi Tim,

I’ve not done anything like this myself, but I remember there was a talk about this at a previous robocon (2024 I think), check the RobotFramework Youtube Channel and hopefully you’ll be able to find it

Also it might be worth checking the Embedded section of awesome-robotframework

Also, you can create keywords in python, and I know python has modules for I²C & SPI and probably the others, so if running robot framework on the Raspberry Pi is an option, that might be the easiest option.

Hopefully that help

Dave.

3

I’ve done some device testing and main issues i’ve had:

  • Need for remote lights out (LOM). Not a huge problem to tackle but its always there depending how the device under tests is powered.
  • iconnectivity depending on the device. Serial ? Serial over usb ? ethernet ? Ethernet over usb ? (and then the rest of the stuff you mentioned like GPIO, JTAG)
  • And regarding the previous point - how many devices are connected to testing. If there are multiple, connectivity ? How is the device marked as “in use” and released after a run.

One scenario where dedicated box would have been nice would be to have TFTP (or similar server) that would allowed flashing / firmware upload but everything i said really depends on what sort of device is currently being tested..

In the past, i had bunch of SBC’s that used uboot to boot into OS. If i had to update the kernel or uboot itself, had to add keyboard to the device, boot it manually and at the start of boot process, enter the fixed keyboard presses so that uboot would go into flashing mode .. Having some usb device that would act as a fake KVM that could be wired to multiple devices would have helped to automate that process ..