How to catch suite setup failure in listener interface 3

Robot Framework
1

I assumed this to be simple, but apparently it is not. I have start_suite and start_test listeners in use and before starting the first test case in the suite, I need to get information in Python code that suite setup has failed. So when the first test starts, the start_test(data, result) result object should contain indicator that suite setup failed, but I cannot get it. Also, if I store the start_suite(data, result) result object in a class variable and inspect it later after suite setup is run, it does not have indication that suite setup has failed.

So how to achieve this?

2

Hi Japi,

If I remember correctly, the start_suite listener would run before suite setup runs and the start_test listener would run after suite setup but if suite setup failed I doubt robot would even get to the start_test listener, so from that I guess you could infer that if start_test listener is running suite setup passed (not ideal, and doesn’t help you with the fail case)

suite setup executes a keyword, it’s quite likely this keyword is unique to suite setup? if so you could potentially use an if statement in an end_keyword listener to check if you are running the suite setup keyword, but unfortunately start_keyword and end_keyword listeners are not implemented in listener interface 3, so you’d need to use a v2 listener for that. The good news is you can have V2 and V3 listeners run in the same robot test, I think they need to be in different listener files but you can include both listener files and it will work (I’ve done it before when I needed features from both listener interfaces)

Hope that helps,

Dave.

3

Thanks Dave, I will consider using V2 listener in a suite setup keyword. That is not definitely an ideal solution tho.

Actually Robot will run the V3 start_test listener even though the setup failed, but for some reason the default state for test case is always “FAIL” even if the suite setup passed. I think that the test status will update to PASS (or the true value) only in end_test listener. I just can’t understand how the default value can be FAIL.

4

The TestCase result object has FAIL status by default and it’s only set to PASS (or SKIP) when the test has finished. This is to play safe, it would be odd if a result of a test that hasn’t yet finished would be PASS. Arguably a better default would be NOT RUN that we nowadays use with keywords and control structures. Changing the default would be easy, but it would require making sure we handle such a test status gracefully elsewhere. For example, suite status is got based on test statuses and the new status needed to be taken into account there. Do you see concrete benefits from changing the default? If yes, I could take a look what it would require.

If we’d use NOT RUN status by default with tests, then we could use FAIL in cases where the test cannot be run like a suite setup failure or invalid syntax. That ought to help in your case, and would be a concrete benefit of using the NOT RUN status, but it would then require changing the code that we actually use the FAIL status in such cases.

You should, however, be able to access suite setup status in start_test via test.parent already now. Something like below would also take higher level suites into account:

def start_test(self, data: running.TestCase, result: result.TestCase):
    if self._suite_setup_failed(result.parent):
        # do something

def _suite_setup_failed(self, suite: result.TestSuite):
    if suite.setup.failed:
        return True
    if suite.parent:
        return self._suite_setup_failed(suite.parent)
    return False
6

Thanks Pekka,

Conceptually your code example has potential to work and I was hopeful it would, but unfortunately in my case start_test() “result.parent.failed” is always True, it does not change if suite setup fails or passes. Also it seems that there is no “suite.setup.failed” attribute in result.TestSuite, only “suite.failed” exists. I am using robot framework v6.0, hopefully there is no defects related to listener functionality in that version. If not, I need to investigate my own code and create a simpler barebone listener testing environment to check how it behaves.

7

Tested using barebone example:

from robot import result
from robot import running
from robot.api import logger

class ListInterface:

ROBOT_LIBRARY_SCOPE = 'GLOBAL'
ROBOT_LISTENER_API_VERSION = 3

def __init__(self):
    self.ROBOT_LIBRARY_LISTENER = self

def my_key(self):
    logger.info(f"Moikka!", also_console=True)

def start_suite(self, data: running.TestSuite, res: result.TestSuite):
    data.doc = 'Documentation set by listener.'

def start_test(self, data: running.TestCase, res: result.TestCase):
    logger.info(f"\nSTATUS = {res.parent.setup.status}", also_console=True)

And using test case file:

*** Settings ***
Library    ListInterface
Suite Setup    MySetup

*** Keywords ***
MySetup
    Log    Passing Suite Setup

*** Test Cases ***
Test1
    Log    Running Test Case
    My Key

Produces logging:

STATUS = FAIL

which is printed from start_test() → 

logger.info(f"\nSTATUS = {res.parent.setup.status}", also_console=True)

So is this considered as a defect, or should I just accept that this is the way it is?

8

This looks pretty strange especially if you are running just a single suite. If you were running a directory and a certain suite wouldn’t have setup, then the suite.setup.status is FAIL by default. That’s not great, the default status should be NOT RUN. It’s too late to change that anymore in RF 6.1, but we could/should change it in RF 7. If the status of a setup that’s run successfully is FAIL, that’s even worse and should be still changed in RF 6.1.