Hi all,
I have been trying to find an answer for this but no success until now.
I have a Test Case with a For loop, and some of the Keywords I’m using return Timeout from time to time, how can I handle this timeout gracefully? For now I’m using TRY/EXCEPT but is there another way, because if I do that, the iteration is marked as passed, and I want it to be marked as failed.
Thanks
If you want to retry the Keywork multiple times until it passes, you can also use Wait Until Keyword Succeeds
I tried that one, but unfortunately it says:
The execution is not continued if the failure is caused by invalid syntax, timeout, or fatal exception.
I would like for the Test to be marked as fail, and the For Loop to start over with the next iteration
EDIT posted without refreshing the page, so possibly something below has already been mentioned.
Hi
I guess if you want to see it marked as a failure, then remove the TRY/EXCEPT and let the FOR fail?
But there are many ways to handle it falling over in a FOR; and it really depends on how you want to handle it. It sounds as though you have wanted it to continue, but it also shows that it has failed.
IF not ${status} CONTINUE or IF not ${status} BREAK and then you can continue the loop if needed or break from the loop…${TEST STATUS} and set the test status yourself, as a note: This can only be set in the teat teardownI really would be looking more at the first point, but if you expect an error, then a TRY/EXCEPT or Run Keyword and Return Status though there are many other ways, but if you want the Fail to show, then you could either let it Fail, run the keyword FAIL or set the test status yourself if you wanted to continue executing the tests.
If you were to share the code and line where the exception was thrown, then you’d be able to possibly get a more definitive approach.
I hope this at least helps 
I caught up with the above; from the sounds of it, breaking or continuing with the Run Keyword And Set Status within the FOR instead of using the TRY/EXCEPT (but you could still use the TRY/EXCEPT), and setting the ${TEST STATUS} after the test as finished excuting would be what you may be after.
The ${TEST STATUS} sounds interesting, but that would mark the entire test as failed, and I would like for the For Loop iteration to be marked as fail and for the loop to proceed.
The problem with the TRY/EXCEPT is that the iteration is marked as pass.
I’m running a test of something that might fail, and I want to take a look at the log and see exactly when it failed.
Other solution for my problem would be if it was possible to split the iterations in Test Cases, but I haven’t found a solution for that also.
Hi Ramiro,
All the previous suggestions are good, personally I like to keep things simple, which library are you using? Most libraries that have a timeout also have a way to adjust the length of time for the timeout.
I’d suggest the most graceful way is to adjust the timeout, if the object/element you want is timing out after 30 seconds because sometimes it takes 48 or 56 seconds to load, increase the timeout to 120 seconds, these keywords will usually continue as soon as the item is there, if it appears in 13 seconds your script only waits 13 seconds, but if it takes 56 seconds it’ll wait 56 seconds or timeout at 30 seconds depending on what you set.
The only catch with setting a long timeout is the script is if the element never loads it can cause the script to be stuck a long time, I usually recommend don’t set it longer than 600 seconds (10 min), but hey I’m a performance tester, so I’ve seen systems get painfully slow, and it’s very rare user indeed that would patiently wait longer than 10 minutes.
Some examples of keywords for doing this:
Hope that helps,
Dave.
Ultimately you’d control which test you set as FAIL in this way, it wouldn’t set all as you’d be able to set a Global variable within the FOR, so the idea would be to set the Run Keyword And Return Status to that keyword that is throwing the time out, the return value will be either true or false.
Then the next line down you’d have an IF block, so if it is false you’d step in, set your global variable and then call CONTINUE the line after and finish with an END then the loop will continue.
Within the teardown you’d move your current into a singular keyword so it can be called, and drop an IF at the end to check if that global variable holds false, if so, then set ${TEST SATUS} to FAIL and then set the global variable back to ${EMPTY} or ${True} afterwards so then any test there after would not be set to FAIL or it could be set back in the test startup.
But this would be my least favourite approach over fixing the why to it being flaky.
Tbh, it’s still not clear if your happy for this line to fail which has been touched on and for which @damies13 as expanded on, so if you are expecting it to fail but happy to continue then I don’t really quite understand why you’d want to set as failed, you’d be best just letting it fail and or looking at making that part of your test not flaky.
All the best.
I’m using the Test Case timeout functionality and not a specific library.
In this case, I’m stress testing HW meaning that the timeout can occur if the HW is faulty and doesn’t respond in time. I just want to mark the FOR iteration as failed, and start a new iteration.
With TRY/EXCEPT it’s almost what I want, and I can log in the console that the iteration is failed, but I would like the log report to be clear on how many iterations failed and which ones.
Run Keyword And Return Status is also not handling timeouts according to the documentation 
I dont think its possible to catch “test case” timeout with try/except blocks. That would be an issue if someone set’s up a test case timeout but then on test case itself one could override that by just catching and ignoring it. Could be wrong thought about this.
Also worth pointing out that robot is single threaded – if actual timeout occurs while a blocking keyword is executing - that will not be cancelled before execution of that keyword ends. Thats why its quite typical to have timeouts implemented at library level for actions that could be blocking the execution. Eg, if you implement your own keywords, you should consider writing timeout support for those keywords by yourself and then its
I would like for the Test to be marked as fail, and the For Loop to start over with the next iteration
because if I do that, the iteration is marked as passed, and I want it to be marked as failed.
Those combined are a bit weird but if i understand what you want to do;
*** Keywords ***
My Keyword
[arguments] ${index}
Log To Console My Keyword: ${index} # 0-9
IF ${index} == 6
FAIL Index cant be 6
END
*** Test Cases ***
Testink
${should_fail} Set Variable ${False}
FOR ${index} IN RANGE 10
${res} Run Keyword And Return Status My Keyword ${index}
Log To Console RES: ${index} ${res}
IF ${res} == ${False}
${should_fail} Set Variable ${True}
END
END
IF ${should_fail} == ${True}
FAIL This testcase failed because at one point "My Keyword" threw error
EN
But
*** Keywords ***
Check Board Boots
[Documentation] Check if MCU starts to boot
[Arguments] ${cycle}
[Timeout] 10s
${read} Read Until Single Board Booted
Append to file ${cycle}_mcu.log ${read} encoding=UTF-8
Loop Boot Cycle
[Documentation] Boot cycle loops
[Arguments] ${index}
Temperature Reached ${temp_low}
Read Measurement
Check Board Boots ${index}
*** Test Cases ***
Power Cycle System 100 Times
FOR ${index} IN RANGE 100
Power System
TRY
Loop Boot Cycle ${index}
EXCEPT AS ${error_message}
Log to Console Boot ${index} failed: ${error_message}
ELSE
Log to Console Boot ${index} success
END
Power Off System
Sleep 30
END
This a rough draft of what I have, the serial port is waiting for a message, and if the message doesn’t come in 10 seconds the keyword returns with Timeout.
I would like to catch that timeout and mark it as a failure, the problem is if I don’t catch it with the TRY/EXCEPT the entire For Loop stops and I would like it to continue running until the end, and have each iteration clearly marked pass or fail accordingly
Ideally I would have 100 Test Cases, instead of iterations, but I haven’t found any way to do that.
Hi Ramiro,
Ah ok, it’s a little clearer what you;re trying to acheive now,
So PASS and FAIL are not really something that applies to a loop, but rather a keyword or test result.
The way Robot Framework is designed when a keyword fails the test fails and stops end of test.
Try / Except and keywords like Run Keyword And Return Status are ways to avoid the test stopping and failing when a keyword fails, allowing you to handle the error condition and continue.
What you are doing is a little like going against how Robot Framework was designed (at least my understanding of it’s design)
What you have is a workaround for that and will give you the 100 polls you’re seeking but not the reporting of the result. There are 2 ways I can see that will give you better reporting
This is a bit hacky, In your Except and Else sections you can use Set Tags to create a tag Boot ${index} success, not ideal as you’ll get 100 tags.
You could also use Set Suite Metadata to increment the count of Boot failed and Boot passed, this would give you the counts in the metadata section of the report but not which iterations failed
This approach works more inline with how Robot Framework was designed, make a new keyword that you will use as a Test template:
Power Cycle System
[Arguments] ${index}
Power System
Loop Boot Cycle ${index}
Power Off System
Sleep 30
Then you can either use Data-driven style and have 100 test lines, or you could try Templates with FOR loops to call the template test 100 times (I’ve never done this so just referring to the doco here) something like this:
*** Test Cases ***
Template with FOR loop
[Template] Power Cycle System
FOR ${index} IN RANGE 100
Power Cycle System ${index} ${index}
END
If I got that right your report should have 100 tests each reporting PASS / FAIL and give you the nice summary data
Hopefully something here helps,
Dave.
Quick win without changing much at all given the provided snippet (but I’d still replace that TRY/EXCEPT for a Run Keyword And Return Status and continue)
Log to Console Boot ${index} failed: ${error_message}
To:
Log Boot ${index} failed: ${error_message} level=WARN
You’d then get the output similar as below:
A warning over a fail based on what you provided sounds more fitting given you’re letting it continue to run regardless of a fail; the log level can only be added to Log not Log To Console
I’ve tried the template but I still get only 1 test reported per Loop, I guess I might have screwed up something.
Also I think the code should look like this, the For loop already calls the template inside the loop without calling the Keyword again
*** Test Cases ***
Template with FOR loop
[Template] Power Cycle System
FOR ${index} IN RANGE 100
${index}
END
Thanks, I think I will use this solution. But I still think I can’t use the Run Keyword And Return Status since I have a lot of Keywords that can return timeout errors and that is not caught by it.