Dear community,
An very impactful idea we brought into our test tooling decades ago was a third result state, called unknown, which we found crucial for truly accurate testing outcomes. While the current core implementation of Robot Framework categorizes test results as either passed or failed, we believe it’s time to reimagine this binary approach and bring in a new unknown state, enhancing the expressiveness and accuracy of Robot Framework’s test results.
Why Unknown Matters: More Precise Test Results
Currently, a test result marked as failed signals that the feature or function under test did not work as expected. But what if the test wasn’t even able to reach the feature due to an unrelated issue? For example:
- The device under test was unreachable.
- A library failed to load, rendering the test inoperable.
- Syntax errors or undefined variables halted the test execution.
- …
In these cases, marking the result as failed can be misleading, as it doesn’t necessarily indicate that the feature itself was flawed—only that an error prevented the test. The unknown state clarifies this ambiguity by designating such test outcomes with an appropriate label, thus providing greater transparency and insight into the nature of test results.
How Unknown Will Work in Robot Framework
With the unknown state integrated, Robot Framework core will differentiate between test failures and incidents where the test could not proceed. This additional precision will not only help clarify the nature of issues encountered but also prevent unnecessary analyzing and or debugging of features that weren’t even tested due to preconditions not being met. Finally it brings clarity and saves time and money.
Developers will be able to invoke this state through BuiltIn().unknown(...) explicitly, allowing you to flag tests with an unknown status when necessary. This can be particularly useful for edge cases or complex test suites where errors outside of the feature under test may frequently arise.
Considering Compatibility with Existing Tools
We are very aware that introducing this unknown state will have side effects on existing tools that post-process result data. However, we’re confident that the increased accuracy in test reporting far outweighs the initial challenges. By adopting this state, teams can make clearer distinctions in test outcomes, reducing misinterpretation and supporting more effective triage and debugging.
Join Us in Building This Feature
We invite you to experiment with the RobotFramework AIO (All In One) OSS project., where the unknown state has already been prototyped, and share your feedback with us.
Below our idea how we propose to implement the unknown state. We would implement and provide a pull-request for the latest 7.x develop branch based on your proposals. We would also maintain the feature:
unknown state |
PR Link |
|---|---|
| Base Implementation | View PR #20 |
Thank you,
Thomas