ParallelRunner 0.4.0: run a keyword in parallel inside one test case, with a single clean log.html

Hi all! I’d like to share a library I’ve been working on.

The problem: a test case that checks many independent things that are mostly waiting on I/O (API endpoints, DB records, health checks) runs them one by one in a FOR loop. Using threads usually ends badly: BuiltIn().run_keyword isn’t thread-safe, and robot.api.logger messages from worker threads get lost or mixed up.

What ParallelRunner does: it runs a Python library keyword on a thread pool, once per item or N times. Each call buffers its own log messages. When all calls have finished, the messages are replayed in order, so log.html reads like a normal sequential run. Results come back in call order.

*** Settings ***
Library    ParallelRunner
Library    my_library.MyLibrary

*** Test Cases ***
Verify Agents In Parallel
${agents}=    Create List    1    2    3    4    5    6    7    8    9
${results}=    Run Parallel Scenarios
...    keyword=Verify Agent Data
...    library=my_library.MyLibrary
...    for_loop_iterable=${agents}

Numbers (bundled offline demos, measured with 0.4.0):

  • 9 checks of 1 s each: 9.0 s sequential → 1.0 s with 9 workers (3.0 s with the default 4).
  • It complements pabot rather than replacing it. pabot parallelizes suites and tests across processes; ParallelRunner parallelizes a loop inside one test with threads. On 4 suites × 8 calls: plain robot 33.1 s, pabot 13.6 s, ParallelRunner 5.2 s.

What’s in 0.4.0:

  • The keyword fails if any call fails, after all calls finish and their logs are written, and the error lists each failed item. fail_on_error=False returns a PASS/FAIL status per item instead.
  • call_timeout=10s: a per-call limit, so one hung call doesn’t block the batch. A keyword can opt in to a _cancelled event to stop cleanly.
  • Robot’s own [Timeout] interrupts the batch and still writes the partial logs.
  • Arguments are converted by type hints, like a direct keyword call.
  • All Robot log levels (TRACE to ERROR) are replayed.
  • 116 unit tests and 82 acceptance tests, run in CI on Python 3.9–3.13 and tested against Robot Framework 5, 6 and 7. Also tested with pabot (by suite and with --testlevelsplit).

Limits, to be upfront:

Feedback, issues and ideas are very welcome, especially if you’ve run into this in your own suites.

Hi @cristiangarciavd,

Welcome to the community,

This is a nice looking library, thank you for contributing it :heart:

One question you didn’t mention is how this compares to AsyncLibrary? i.e. when would you use ParallelRunner in preference to AsyncLibrary? perhaps the answer is related to the buffering of logs?

Also you mentioned that ParallelRunner only works with python keywords for now, are you planing to support user defined robot keywords in a future release?

Dave.

Thanks a lot, and great questions! :folded_hands: :people_hugging:

AsyncLibrary: I hadn’t actually compared them closely before :eyes: , so now I tried it with some help from Claude to test different scenarios and compare. It’s more capable than I expected. With a FOR loop of Async Run calls and one Async Get on the handles, it runs any keyword in parallel, user keywords included, and it buffers and replays each call’s output so log.html keeps the full keyword tree. So you’re right that logging is part of the story, but both libraries handle it. The real difference (I think) might be in the use case:

  • AsyncLibrary is the better fit when the work lives in user keywords (it’s great to be able to use the robot keywords there), or when you want to start something in the background, run other steps, and collect it later.
  • ParallelRunner is for fanning out one Python keyword over a list (or N times) as a single batch. In addition, there are a few points that might help in some specific scenarios:
    • you set the worker count (AsyncLibrary seems to use Python’s default pool size, min(32, CPUs + 4))
    • call_timeout abandons a hung call and frees its slot for the next item, while Async Get’s timeout stops waiting but the call keeps its thread
    • results come back in call order, and the keyword fails once with a list of every failed item. With fail_on_error=False you get every call’s outcome instead (the return value or the error), so one failed call doesn’t cost you the results of the others (no manual per-handle loop needed)
    • the log is lighter: messages only instead of the full keyword tree per call, so output.xml stays smaller with thousands of calls. The trade-off is less step-by-step detail.

User keywords: not planned for now. ParallelRunner calls the Python method directly and only writes to the log from the main thread, through the public logger API (avoiding to patch Robot’s internals). That’s why it needs the _logger parameter, but it also means less to maintain and less exposure to internal changes between Robot versions. It’s tested on Robot 5, 6 and 7. For user keywords today, I’d point people to AsyncLibrary though :eyes:

Thanks for the idea, by the way :people_hugging: . I’ve added a short comparison to the README so it’s easier to find: GitHub - cristiangarciavd/robotframework-parallelrunner: Run a loop inside a single Robot Framework test case in parallel — with one clean `log.html`, not a merge of many. · GitHub

Update for anyone using this library: 0.5.0 adds two things related to call_timeout and one change you may need to act on.

  • call_retries (with retry_delay / retry_backoff) retries a failed or timed-out call before it counts as failed. Each result now has an attempts key.
  • Threads left running by timed-out calls are now capped by ROBOT_MAX_ABANDONED_THREADS (default: twice ROBOT_THREAD_WORKERS), so many hung calls no longer pile up threads and connections.

Before upgrading:

  • Python 3.10+ is required now; on 3.9, pip keeps installing 0.4.0.
  • Keyword arguments named call_retries, retry_delay or retry_backoff are no longer forwarded to your keyword.