Add tags to keywords of builtin library based on configuration

Hello,

I am relatively new to Robot Framework, but so far, I always got the results I wanted and (almost) all of the approaches made sense… But I am struggling with one specific aspect of the various options / use-cases of “continue-on-failure”.

From the docs, I got, that I can apply this tag for example to test cases and keywords, and the tag is automatically set for teardowns. perfect for me…

But I work in an environment, where we, by norm, rely on shared validated assert functions, most often the ones in the Builtin library, starting with “Should …” (which could be better called “Shall …”, but that is besides the point)… like “Should be equal as numbers …”

Given, that we use those keywords after obtaining some value to assert, wether the obtained value is correct and as expected or not, I would like to run those keywords with the tag “continue-on-failure”, since the outcome of those assertions does (usually) not block or disturb the rest of the execution or the remainig keywords of the test (the value is just wrong, for example some reaction took too long…)…

Is there a way to add such tags by configuration to a specific list of keywords (i.e. via the robot.toml for Robot Code) ?
Currently, with Robot Code, I can use the parameter “set-tag” to add tags “on-the-fly” to tests, but not as far as I know to keywords… (robot.toml configuration settings | RobotCode)
The reason for not just changing the Builtin library is, that I do not want to make those changes permanently and everywhere and I do not want to alter the official release, but rather extend / add to them (for auditing purposes…).

Thanks for any pointers on how to go about doing this…!
Is there maybe a way to set the tag for all keywords of a certain section (like “activate continue-on-failure on all following keywords” followed later by “deactivate continue-on-failure for all following keywords”)

Cheers
Niels Göran

i may not get the point, but was curious and tested with copilot. the result is a example test case like:

** Settings ***
Library    CustomListener.py

*** Test Cases ***
TEST A
    Log To Console    HI test A will pass

E2E_TEST B
    Log To Console    HI test B will Fail
    Should Be Equal As Strings    1    2
    Should Contain    "Hello"    "World"
    Log To Console    Hi test B still continues

TEST C
    Log To Console    Hi test C will Pass

now, using a listener you can dynamically update the tags or keywords, but depends of course if you can name the test cases where you want to add “robot:continue-on-failure” and if all your “should” assertions are ment to be conditinaly soft asserted and hence updated, or just some of “shoulds…” but if yes, you may try with listener where basically the “should” assertions will be “prefixed/wrapped” with Run keyword and continue on failure, hence test will not fail here.

from robot.libraries.BuiltIn import BuiltIn

class CustomListener:
ROBOT_LISTENER_API_VERSION = 3

def start_test(self, data, result):
    """Dynamically add 'robot:continue-on-failure' tag to test cases starting with 'E2E_'."""
    if data.name.startswith("E2E_"):
        BuiltIn().run_keyword("Set Tags", "robot:continue-on-failure")
        print(f"🔹 Added 'robot:continue-on-failure' tag to test: {data.name}")

def start_keyword(self, data, result):
    """Wrap assertion keywords in 'Run Keyword And Continue On Failure'."""
    assertion_keywords = ["Should Be Equal", "Should Be True", "Should Contain",
                          "Should Not Be Equal", "Should Not Be Empty"]

    if any(data.name.startswith(kw) for kw in assertion_keywords):
        print(f"⚠️ Wrapping '{data.name}' with continue-on-failure")

        # Modify the keyword to run within "Run Keyword And Continue On Failure"
        data.name = "BuiltIn.Run Keyword And Continue On Failure"
        data.args = ["BuiltIn." + data.name] + list(data.args)

def end_test(self, data, result):
    """Log test result."""
    print(f"✅ Finished test: {data.name} | Status: {result.status} | Tags: {', '.join(data.tags)}")

so running robot --listener CustomListener.py test.robot adds tag to “E2E” test case and updates “should…” keywords

2 Likes

Edit / Extension:

Something is still mixed up…
data.name = “BuiltIn.Run Keyword And Continue On Failure”
data.args = [“BuiltIn.” + data.name] + list(data.args)
I don’t think the order is correct, since this would put the new keyword to be called (data.name) also as the first argument…

Hej hej,

thanks very much for your answer!
It pointed me exactly in the right direction…
A Listener that can also be added / extended per configuration in robot.toml (robot.toml configuration settings | RobotCode, robot.toml configuration settings | RobotCode) seems like the easiest solutions for the use-case I described…

Aside from that, I also notieced, that it would even be an option to create wrapped that wraps the orginial builtin keywords with the tag (and maybe some other additional steps)… I do not know, why I didn’t considered that a viable options before, but your solution clearly has the benefit of being able to be added later on without reworking everything…

Thanks !

Cheers
Niels Göran

1 Like

Hej hej,

maybe I am missing something here, but I do not seem to be able to get the example from the copilot to fully work…
I can alter the arguments, but I cannot alter the keyword being executed…
Any idea, if that is even possible with start_keyword?
I saw robotframework/atest/testdata/output/listener_interface/body_items_v3/eventvalidators.py at master · robotframework/robotframework · GitHub, but the focus here is to show, that start_keyword is actually not called.
I also saw robotframework/atest/testdata/output/listener_interface/body_items_v3/ArgumentModifier.py at master · robotframework/robotframework · GitHub which seem sto focus on creating keyword, rather than altering them…

Any ideas?

Cheers
Niels Göran

===

listener class looks currently like this:

from robot.libraries.BuiltIn import BuiltIn
from robot.api.interfaces import ListenerV3
from robot import result, running

class ContOnFailureForTestsKeywords(ListenerV3):

ROBOT_LIBRARY_SCOPE = 'GLOBAL'
ROBOT_LISTENER_API_VERSION = 3

# listener must be passed a string argument, which is in fact a list, where the individual strings are seperated by the ';'-sign
def __init__(self, kws : str = ''):
    self._kws = kws.split(';')
    print('listener initialized...')

def start_test(self, data: running.Keyword, result: result.Keyword):
    """Dynamically add 'robot:continue-on-failure' tag to test cases starting with 'CONT_'."""
    if data.name.startswith('CONT_'):
        BuiltIn().run_keyword('Set Tags', 'robot:continue-on-failure')
        print(f"{data.name} - Added 'robot:continue-on-failure' tag to testcase")

def start_keyword(self, data: running.Keyword, result: result.Keyword):
    """Wrap specified keywords in 'Run Keyword And Continue On Failure'."""
    if any(data.name.startswith(kw) for kw in self._kws):
        # Modify the keyword to run within "Run Keyword And Continue On Failure"
        data.args = ['Builtin.' + data.name] + list(data.args)
        data.name = 'BuiltIn.Run Keyword And Continue On Failure'
        print(data)

def end_keyword(self, data: running.Keyword, result: result.Keyword):
    if any(data.name.startswith(kw) for kw in self._kws):
        print(data)

def end_test(self, data: running.Keyword, result: result.Keyword):
    """Log test result."""
    print(f"{data.name} - Finished  | Status: {result.status} | Tags: {', '.join(data.tags)}")