Question | Exclude empty suites from reports?

Robot Framework
1

I currently have a Jenkins pipeline that is running sets of tests based on tags selected. This of course sometimes leads to empty test suites that don;t have any tests matching the tag selection like so:

image
image709Ă—403 11.9 KB

Is there any way to exclude empty test suites from either the report created by robot or the merged report created by rebot?

2

How do you execute robot?
I have never seen this behavior.

When i call robot --exclude tag suite only the suites with test cases are in the log.

Can you give more details?

Solution:
One solution that would work would be a SuiteVisitor that reworks the output.xml and deletes all suite that don´t have a test case, but i think there are easier ways.

It would work like this example in the docs.

"""Pre-run modifier that excludes tests by their name.

Tests to exclude are specified by using a pattern that is both case and space
insensitive and supports '*' (match anything) and '?' (match single character)
as wildcards.
"""

from robot.api import SuiteVisitor
from robot.utils import Matcher


class ExcludeTests(SuiteVisitor):

    def __init__(self, pattern):
        self.matcher = Matcher(pattern)

    def start_suite(self, suite):
        """Remove tests that match the given pattern."""
        suite.tests = [t for t in suite.tests if not self._is_excluded(t)]

    def _is_excluded(self, test):
        return self.matcher.match(test.name) or self.matcher.match(test.longname)

    def end_suite(self, suite):
        """Remove suites that are empty after removing tests."""
        suite.suites = [s for s in suite.suites if s.test_count > 0]

    def visit_test(self, test):
        """Avoid visiting tests and their keywords to save a little time."""
        pass

https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#example-exclude-tests-by-name

1 Like
3

I am running it using:

robot --RunEmptySuite -i tag1 -e tag2 suite

If I were to remove --RunEmptySuite it would indeed exclude them but it counts them as a failure which propagates up to Jenkins:
[ ERROR ] Suite 'Buildings' contains no tests matching tag 'dev AND About' and not matching tags 'doNotRun', 'inDevelopment' or 'Flaky'.

image
image457Ă—544 20.8 KB

I will take a look at SuiteVisitor, thank you.

4

This is a prerunmodifier of DataDriver that deletes all suites that run pass last time.
Here you see how to “parse” the results of an output.xml.

the __init__(self, original_output_xml) get the path to the output.xml.

5

rebot --prerebotmodifier DeleteEmptySuites.py --output output_new.xml output.xml

just delete the most stuff from the example code … :wink:

from robot.api import SuiteVisitor

class DeleteEmptySuites(SuiteVisitor):

    def end_suite(self, suite):
        """Remove suites that are empty."""
        suite.suites = [s for s in suite.suites if s.test_count > 0]

    def visit_test(self, test):
        """Avoid visiting tests and their keywords to save a little time."""
        pass
1 Like
6

Thanks so much for all the help!

In the end however while working through this I realized my mistake… :sweat_smile: I am currently running these tests from a shell script and I just needed to add
|| exit 0
to ignore the robot execution status. Then when I use the robot publisher in Jenkins it will appropriately set my Success, Unstable, or Failure status based on the actual tests run.

I will certainly keep in mind the suggestion though as I did not know about the prerebotmodifier at all!

7

I believe, exit 0 is windows specific. Robot offers an inbuilt switch --nostatusrc to ignore failed runs:

robot --nostatusrc -i flaky -i development .

See all options here: https://github.com/robotframework/robotframework/blob/master/doc/userguide/src/Appendices/CommandLineOptions.rst

8

Its working on windows and linux but I will give that a try tomorrow though, thank you.

9

Tried that out today, works great, thanks!