Test Plan with Advanced Tagging Logic

Robot Framework
1

Does Robot Framework support tagging tests with AND logic?

Currently we can tag a test like [tags] tag1 tag2 in which tag1 and tag2 are logical ORs. Can we tag the test plan so that it only matches runs (in my case, DUTs) that include both tag1 AND tag2?

The problem I’m facing is that I have multiple network devices I need to run tests against. I have some tests that can only run on some of these devices, and not others.

For example, I have three DUTs with the following “tags”.

DUT1:

  • RoleA
  • Type1

DUT2:

  • RoleB
  • Type2

DUT3:

  • RoleA
  • Type3

I will be running a robot command that target’s each of these DUTs and passes in the role and type parameters as tags. Essentially saying “this is the DUT to target and here is what it supports”.

I have a test that matches this criteria (RoleA OR Role B) AND (Type1 OR Type2). Given this, we should expect the test to run on DUT1 and DUT 2, but not DUT3.

how can I define the tags in the test plans to make this work? If I use the --include flag and AND on the tags, then that run will ignore tests that are just tagged with RoleA, which isn’t desirable.

2

Hi Nick,

Refer to Tagging test cases to add the tags you need to your test cases

Then refer to Tag patterns in Using command line options for how to control which tags to include or exclude, you have both AND and OR options, so I believe Robot Framework will support what you need.

Dave.

3

Thanks for the response. I think my problem is more nuanced and can’t be solved with tag patterns or the command line options. Allow me to explain.

I have some DUTs that have unique parameters, and based on these parameters, we determine which tests can or can’t run on them.

Again these are:
DUT1:

* RoleA
* Type1

DUT2:

* RoleB
* Type2

DUT3:

* RoleA
* Type3

I have some tests that have the following logic:

Test 1: Match on (RoleA or RoleB) AND (Type1 or Type2)

I can tag the test as follows:

[tags] RoleA RoleB Type1 Type2

However, these tags are all logical ORs. Meaning that as long as one of those tags matches, the test is ran.

I want to write the test tag logic like this:

[tags] (RoleA or RoleB) && (Type1 or Type2)

I then want to be able to invoke the robot CLI command for each DUT, and pass in the DUT’s parameters as tags like so:
robot -v DUT:DUT1 -v Role:RoleA -v Type:Type1 /path/to/all/robot/tests
robot -v DUT:DUT2 -v Role:RoleB -v Type:Type2 /path/to/all/robot/tests
robot -v DUT:DUT3 -v Role:RoleA -v Type:Type3 /path/to/all/robot/tests

That way, test 1 runs on DUT1 and DUT2, but not DUT3.

Using this same logic, I should be able to have a new test called Test 2 with a different tag matching pattern defined in the test plan, and with the same 3 robot cli commands as before, have Test 2 run or not run on those 3 DUTs based upon that test’s matching pattern.

So in essence, what I looking for is a way to tag tests with advanced tagging logic like this
[tags] (RoleA or RoleB) && (Type1 or Type2)

4

Hi Nick,

Perhaps if I give an example:

geruta_Nick_Heaton.robot

*** Test Cases ***
Test ZRA
    [Tags]    RoleA
    Log    This is Test ZRA

Test ZRB
    [Tags]    RoleB
    Log    This is Test ZRB

Test ZT1
    [Tags]    Type1
    Log    This is Test ZT1

Test ZT2
    [Tags]    Type2
    Log    This is Test ZT2

Test ZT3
    [Tags]    Type3
    Log    This is Test ZT3

So when I run robot including the tags, I get this:

 % robot -i RoleA -i Type1 geruta_Nick_Heaton.robot 
==============================================================================
geruta Nick Heaton                                                            
==============================================================================
Test ZRA                                                              | PASS |
------------------------------------------------------------------------------
Test ZT1                                                              | PASS |
------------------------------------------------------------------------------
geruta Nick Heaton                                                    | PASS |
2 tests, 2 passed, 0 failed
==============================================================================
Output:  /Users/dave/tmp/output.xml
Log:     /Users/dave/tmp/log.html
Report:  /Users/dave/tmp/report.html

 % robot -i RoleB -i Type2 geruta_Nick_Heaton.robot
==============================================================================
geruta Nick Heaton                                                            
==============================================================================
Test ZRB                                                              | PASS |
------------------------------------------------------------------------------
Test ZT2                                                              | PASS |
------------------------------------------------------------------------------
geruta Nick Heaton                                                    | PASS |
2 tests, 2 passed, 0 failed
==============================================================================
Output:  /Users/dave/tmp/output.xml
Log:     /Users/dave/tmp/log.html
Report:  /Users/dave/tmp/report.html

 % robot -i RoleA -i Type3 geruta_Nick_Heaton.robot
==============================================================================
geruta Nick Heaton                                                            
==============================================================================
Test ZRA                                                              | PASS |
------------------------------------------------------------------------------
Test ZT3                                                              | PASS |
------------------------------------------------------------------------------
geruta Nick Heaton                                                    | PASS |
2 tests, 2 passed, 0 failed
==============================================================================
Output:  /Users/dave/tmp/output.xml
Log:     /Users/dave/tmp/log.html
Report:  /Users/dave/tmp/report.html

Now for example if you introduced DUT4 that needed RoleB and Type3 then

% robot -i RoleB -i Type3 geruta_Nick_Heaton.robot
==============================================================================
geruta Nick Heaton                                                            
==============================================================================
Test ZRB                                                              | PASS |
------------------------------------------------------------------------------
Test ZT3                                                              | PASS |
------------------------------------------------------------------------------
geruta Nick Heaton                                                    | PASS |
2 tests, 2 passed, 0 failed
==============================================================================
Output:  /Users/dave/tmp/output.xml
Log:     /Users/dave/tmp/log.html
Report:  /Users/dave/tmp/report.html

Unless I’ve missed something is that not what you wanted to achieve?

Dave.