Fail whole test suite if a test case fails without affecting following test suites

Robot Framework
1

Hello,

I run tens of test suites each containing several test cases in a row. If a test case in a test suite fails I need to fail or skip all the remaining test cases in that test suite without affecting following test suites.

I am planning to use Test Setup to fail or skip the testcase if any previous test has failed by checking a suite variable which is set in Test Teardown if the testcase failed.
I’m just wondering if there is a better way to do this?

Br,
Sami

2

Hi Sami,

  1. Not sure if this will help, you could try Fatal Error
    It says: “The test or suite where this keyword is used fails with the provided message”
    but it also says: “Stops the whole test execution.”
    It seems like you want something in between Fail and Fatal Error
  1. my other suggestion would be to use Set Suite Variable.
    set a variable to false in the *** Variables *** section
    in the test that the others depend on set it to true as the last line
    in the other tests after this important one check if the variable is true before continuing
    if it’s not one specific test but they all need to pass sequentially (I don’t remember where it is in the documentation, but it’s recommended that test cases shouldn’t be dependent on each other) you could set it to false when a test starts and back to true as the last step in the test, alternatively you could use a Test teardown to set it to false

Hopefully something in that is helpful,

Dave.

3

Hi Dave,

Thank you for your response.
You’re right. Based on my experiments Fatal Error terminates the whole test execution i.e. all the following test suites will fail also. Fail keyword causes only the current test case to fail. So something between them would be perfect in my case.

I know that tests should not be dependent on each other but in my case (embedded system testing) test cases would be very big and complicated. That’s why I prefer to use test cases which sometimes are dependent on each other in a same test suite.
I will continue with suite variable based solution.

Br,
Sami

1 Like
4

A little bit late to the conversation, but I wanted to share my own solution.

I also have dozens of test suites to run, where I want to skip the suite if there are any failures, but continue with the remaining test suites. I also wanted the results of all suites conveniently in a single report.
Using robot --exitonfailure command will skip the current test suite when a failure occurs (good), but also skip any remaining test suite after the current test (bad).

My solution is to write a simple bash or python script executing each suite independently with the --exitonfailure option, then combining all results into a single report with rebot.

Something like this in bash (run.sh):

#!/usr/bin/env bash
robot -d results/subdir_01 --exitonfailure test_suites/my_suite_01
robot -d results/subdir_02 --exitonfailure test_suites/my_suite_02
robot -d results/subdir_03 --exitonfailure test_suites/my_suite_03
robot -d results/subdir_04 --exitonfailure test_suites/my_suite_04
robot -d results/subdir_05 --exitonfailure test_suites/my_suite_05
rebot --outputdir results results/*/*.xml

Then in terminal,
> ./run.sh

Or something like this in Python (run.py):

import os

commands = [
   "robot -d results/subdir_01 --exitonfailure test_suites/my_suite_01"
  ,"robot -d results/subdir_02 --exitonfailure test_suites/my_suite_02"
  ,"robot -d results/subdir_03 --exitonfailure test_suites/my_suite_03"
  ,"robot -d results/subdir_04 --exitonfailure test_suites/my_suite_04"
  ,"robot -d results/subdir_05 --exitonfailure test_suites/my_suite_05"
  ,"rebot --outputdir results results/*/*.xml"
]

if __name__ == '__main__':
  for line in commands:
    os.system(line)

Again, in terminal:
> python run.py