Using a Listener to Stop a Test When a Flag Is Set

Robot Framework
1

Hey everyone, I’m hoping for some help with this!

I’m working with a Robot Framework test suite that interacts with a WebSocket server using a custom Python library. This library has an expectedErrorSeen flag, which is set to True when an expected error is see in the web socket reply. Im using a test template and there the error can happen at different keywords in the test.

What I Want to Achieve:

  • Whenever expectedErrorSeen is set to True, stop the test immediately but mark it as passed.
  • This should work for any keyword, not just a specific one.

I wrote the following listener to detect when expectedErrorSeen is True in the library and i tried to stop the execution using BuiltIn().pass_execution() function. I get the following errror and the keywords in the test are executed.

Calling method ‘end_library_keyword’ of listener HarrierListener.py’ failed: Expected error encountered, skipping test.

Test Template
Send Test Email <— error could happen here if it does i don’t want the rest of the test to run
Process Release Approval Of Email
Verify Test Email In Recipients Inbox
Retransmit Test Message <— error could happen here if it does i don’t want the rest of the test to run
Verify Retrasmited Message In Senders Folder
Process Release Approval Of Retransmitted Message
Verify Retrasmited Message In All Recipients Folder

from robot.libraries.BuiltIn import BuiltIn
from robot import result, running
from robot.api.interfaces import ListenerV3
from pprint import pprint

class HarrierListener(ListenerV3):

def __init__(self):
    self.expected_error_seen = False
    self.test_started = False

def start_test(self, data, result):
    """Reset expected error flag at the start of each test."""
    self.test_started = True
    harrier_ws= BuiltIn().get_library_instance("HarrierUtilities.SMIME.HarrierWebSocket")
    setattr(harrier_ws, "expectedErrorSeen", False)

def end_keyword(self, data: running.Keyword, result: result.Keyword):
    """Check for the expected error flag after every keyword execution."""
    if self.test_started == True:
        # Retrieve the HarrierWebSocket instance and check for expectedErrorSeen
        harrier_ws= BuiltIn().get_library_instance("HarrierUtilities.SMIME.HarrierWebSocket")
        if hasattr(harrier_ws, "expectedErrorSeen") and harrier_ws.expectedErrorSeen:
            self.expected_error_seen = True
            BuiltIn().pass_execution("Expected error encountered, skipping test.")

My question is:
is this even possible using a listener or do i need to look at managing this another way.

2

Hi Willem,
in my opinion this should work.

Are you able to run/debug this thing locally?
I would be interested in the result.status during the end_keyword call.
Is it PASS or is it FAIL ? Maybe it needs to be adjusted.

To Debug:
If its a Listener Library and you are using RobotCode, you can enable the Debug: Attach Python option and set a breakpoint in your end_keyword function.

If it is a normal listener, you can add a Debug Configuration to VS Code like
{
“name”: “Robot Framework: HarrierListener”,
“type”: “python”,
“request”: “launch”,
“module”: “robot”,
“args”: [“–listener”, “MyListener.py”, “-d”, “results”, “${file}”],
“console”: “integratedTerminal”,
“justMyCode”: false
}
and just use run your .robot suite in VS Code using that Debug configuration.

3

But I have to admit that I never tried using BuiltIn().pass_execution in a listener and don’t know the actual effect on the remaining keywords…

4

I simply think that its not possible to use BuiltIn().pass_execution in a listener, i would probably need to change some thing in one of the models but that’s a little over my head.

Something here i think.
https://robot-framework.readthedocs.io/en/master/autodoc/robot.running.html#robot.running.model.Keyword

5

Using BuiltIn.pass_execution() like this doesn’t work. The reason is that it raises an exception and although that exception works when raised by a keyword, any exceptions raised by a listener is considered a failed listener call.

The way for listeners to affect the execution flow is setting result.status. Unfortunately for you, setting it to PASS only passes the current keyword, it doesn’t stop the whole execution. One way to accomplish that would be modifying data so that you add Pass Execution there. Not too pretty but ought to work.

Ideas how to enhance the listener API so that this would be supported are appreciated. The approach should work also with other special cases like continuable failures and stopping the whole execution.

6

Thanks for the confirmation Pekka