Wait Until Keyword Succeeds (WUKS) and API testing

Robot Framework
1

I’ve been using WUKS in the context of API testing where the general work flow is as follows:

  • Submit an async API transaction and receive a transaction ID
  • Use WUKS and subsequentially run a query on the transaction ID’s status
  • If all goes well, continue with the test.

Unfortunately, the part that WUKS has to process is rather a large one as the API itself is fairly complex. For each call to WUKS, the underlying API-specific robot code receives:

  • a variable name. That entry will be extracted from the API response’s JSON opject
  • a positive list and a negative list of values for that variable. Explanation see below
  • additional positive and negative lists API call’s http response (e.g. 200 for http200)

Additionally, the actual async operation can take up to two hours to complete. Yes, two hours.

The WUKS result is considered to be positive/successful if:

  • the variable’s value can be found in the JSON response and its value can be extracted AND
  • the variable’s value is contained in the positive list AND
  • the http response code’s value is contained in the positive list

The WUKS result is considered to have failed completely if:

  • the variable’s value can be found in the JSON response and its value can be extracted AND
  • (the variable’s value is contained in the negative list OR
  • the http response code’s value is contained in the negative list)

Any other error situation is considered to be a minor error which will trigger a new WUKS cycle.

However, when the test knows that its async option has failed completely, I need to break free from the WUKS cycle and abort it. To my knowledge, I could fail the WUKS part of the test in the following two ways:

  • Fail keyword. This will not help me as it will simply trigger another WUKS cycle. In case I have specified a retry interval of 120 times with a sleep cycle of one minute, my test will terminate - but only after 120 minutes.
  • Fatal Error keyword. This is the option that I have chosen. The moment the test recognises that the WUKS test step has failed (and that there is no further sense in waiting until the test auto-terminates), I do trigger a Fatal Error and the test terminates. Works like a charm.

So far so good. The issue that I want to get clarification on is that the Fatal Error keyword does not only terminate the current test but ALL other tests within that same suite. So if I have a test file where the very first test case’s API fails and causes a Fatal Error, the remaining test cases of the same suite will no longer be processed.

What I’m essentially looking for is a keyword that permits me to break free from the WUKS part of my test and terminates the current test at the same time - but permits Robot to continue processing of the remaining tests in my suite. From a ‘failure severity’ perspective, is there a keyword which has a higher severity than Fail and a lower severity than Fatal Error?

Alternatively, is there a better way for breaking free from a WUKS test and have it the current test terminated in a negative manner?

2

So far so good. The issue that I want to get clarification on is that the Fatal Error keyword does not only terminate the current test but ALL other tests within that same suite. So if I have a test file where the very first test case’s API fails and causes a Fatal Error , the remaining test cases of the same suite will no longer be processed.

About what you mentioned above, maybe you could create a special condition to avoid waiting so long. For example, add a sleep of 60s in case of failure and make your WUKS with 120x 1s. It’s not the perfect solution, but your test will not be stuck for so long.

Another suggestion here… when you are executing the tests, you could separate then in 2 executions. First run all the stable/critical tests and then the unstable/non-critical. You can use different files with one of then having the first line as --non-critical followed by all the tests or using tags for the 2 different executions. I know there is a way to put both results in the same html report, but I don’t remember what is the syntax.

Good luck!
Evaldo.

3

Thank you Evaldo. Unfortunately, none of your suggestions are going to work for my use case. An average test completion takes between 15 and 20 minutes but CAN take up to 120 minutes (for this test case; others may take longer than that). Additionally, if I decide to request the API status in a shorter interval, the API maintainer team is going to kill me :slight_smile:

For completeness sake, I’ve opened up a Github issue related to this topic: https://github.com/robotframework/robotframework/issues/4077. The only workaround that I have been able to determine (but haven’t tested yet):

  • Positive Outcome exits the WUKS keyword successfully with a “success” return code (e.g. global variable)
  • Negative Outcome exits the WUKS keyword successfully with a “failed” return code
  • Just like before, a “test is still running” use case triggers a soft fail which will then cause a new WUKS loop.

With this scenario, code residing outside of the WUKS keyword could then evaluate the test’s result - and terminate it with a FAIL, if necessary. This approach is far from ideal but should work. I will go for this approach if there is no better solution.

4

As per https://github.com/robotframework/robotframework/issues/4077, I will go for external code residing outside of the WUKS keyword which will handle the issue.

1 Like