Retry Mechanism on Flaky Test

Robot Framework
1

Hi,

I wanted to have your though about having a mechanism that allows to retry a full test when it is failing. Let say same as Wait Until Keyword Succeed but for a test
Something like
[Retry] 3 500ms
Which will retry the test 3 times with a waiting period of 500ms.
In the scope of backend application that has a lot of external dependencies which could fail at some point we need to stabilise our CICD. Having a retry will allow a dependency to be down for a short period of time.
WDYT?

2

If the state of SUT changes in anyway and test is re-run, there’s a pretty good change that the rest of the retries will fail too. For example, consider this:

  1. suite setup → starts application, its state is Z
  2. test setup → sets the application into state X
  3. test case → clicks few buttons to get into state Y, then fails …

Now, how would step 2 know the state of the SUT and to have conditional for setting up the SUT from Y to X instead of “normal” Z to X … Would make things more complicated.

Robot does already have this sort of mechanism. Once you run your full suite and you do have the output.xml from it - you can use --rerunfailed commandline argument, place the output to a new file and after that, you can use rebot to combine the re-run to original report …