Mark test version to execute backward-compatible test cases

Robot Framework
1

Hi there!

I have a set of test cases, some of them have evolved as the software releases came out. I need to maintain backward compatibility, so I need to be able to execute all tests compatible with a specific version of the software. I thought of using tags to check if a test case is compatible or not, but it doesn’t work so well… Imagine the following test cases:

TC1 - v1.0
   [tags]  version=1.0
   ...

TC1 - v2.0
   [tags]  version=3.0
   ...

TC2 - v1.0
   [tags]  version=2.0
   ...

Right now, let’s say I’m testing version 2.2 of my software. I would like to run robot framework that way:

robot -i version<=2.2 myTestSuite.robot

I don’t think that’s possible… so what would be the proper way of handling this?
Looking forward to your suggestions!

1 Like
2

Interesting problem.

I only see a preprocessing strategy. You would have a test/system/python script that would do the calculations, generate a list and prepare an Arguments File for the actual test run.

Other option, to avoid tags, could be using the Metadata field at Suite level, where you can have those versions stored. Again, some calculation would need to happen, possibly at Suite Setup, where you would pass a command line variable definition with your condition.

If you want a good example of using Argument Files, you could see RIDE testrunner, because it creates those files always (but, of course, with manual selection).

2 Likes
3

Hi Bruno,

Think of the tags more like an identifying string rather than a key:value pair.

Referring to the documentation for Tag patterns and Simple patterns I came up with the following example for you:

Example robot file:

marques-bruno_1.robot

*** Test Cases ***

#
# Version 1
#
TC1 - v1.0
	[tags]  Version_1.0
	Log    TC1 - v1.0

TC1 - v1.1
	[tags]  Version_1.1
	Log    TC1 - v1.1

TC1 - v1.2
	[tags]  Version_1.2
	Log    TC1 - v1.2

TC1 - v1.3
	[tags]  Version_1.3
	Log    TC1 - v1.3

TC1 - v1.5
	[tags]  Version_1.5
	Log    TC1 - v1.5

TC1 - v1.9
	[tags]  Version_1.9
	Log    TC1 - v1.9

#
# Version 2
#
TC1 - v2.0
	[tags]  Version_2.0
	Log    TC1 - v2.0

TC1 - v2.1
	[tags]  Version_2.1
	Log    TC1 - v2.1

TC1 - v2.2
	[tags]  Version_2.2
	Log    TC1 - v2.2

TC1 - v2.3
	[tags]  Version_2.3
	Log    TC1 - v2.3

TC1 - v2.7
	[tags]  Version_2.7
	Log    TC1 - v2.7

TC1 - v2.9
	[tags]  Version_2.9
	Log    TC1 - v2.9

#
# Version 3
# 
TC2 - v3.0
	[tags]  Version_3.0
	Log    TC1 - v3.0

TC1 - v3.1
	[tags]  Version_3.1
	Log    TC1 - v3.1

TC1 - v3.2
	[tags]  Version_3.2
	Log    TC1 - v3.2

TC1 - v3.3
	[tags]  Version_3.3
	Log    TC1 - v3.3

Examples of calling this file:

Run everything:

% robot marques-bruno_1.robot                     
==============================================================================
Marques-Bruno 1                                                               
==============================================================================
TC1 - v1.0                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v1.1                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v1.2                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v1.3                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v1.5                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v1.9                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v2.0                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v2.1                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v2.2                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v2.3                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v2.7                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v2.9                                                            | PASS |
------------------------------------------------------------------------------
TC2 - v3.0                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v3.1                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v3.2                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v3.3                                                            | PASS |
------------------------------------------------------------------------------
Marques-Bruno 1                                                       | PASS |
16 tests, 16 passed, 0 failed
==============================================================================

Run only version 1.x test cases

% robot --include 'Version_1*' marques-bruno_1.robot                             
==============================================================================
Marques-Bruno 1                                                               
==============================================================================
TC1 - v1.0                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v1.1                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v1.2                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v1.3                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v1.5                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v1.9                                                            | PASS |
------------------------------------------------------------------------------
Marques-Bruno 1                                                       | PASS |
6 tests, 6 passed, 0 failed
==============================================================================

Run all version 1 test cases and all version 2 up to version 2.2

% robot --include 'Version_1.*' --include 'Version_2.[1-2]' marques-bruno_1.robot
==============================================================================
Marques-Bruno 1                                                               
==============================================================================
TC1 - v1.0                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v1.1                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v1.2                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v1.3                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v1.5                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v1.9                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v2.1                                                            | PASS |
------------------------------------------------------------------------------
TC1 - v2.2                                                            | PASS |
------------------------------------------------------------------------------
Marques-Bruno 1                                                       | PASS |
8 tests, 8 passed, 0 failed
==============================================================================

Naturally if you want to test version 2.7:

robot --include 'Version_1.*' --include 'Version_2.[1-7]' marques-bruno_1.robot

or version 3.3:

robot --include 'Version_1.*' --include 'Version_2.*' --include 'Version_3.[1-3]' marques-bruno_1.robot

Or for version 3.3 you could also do:

robot --include 'Version_[1-2].*' --include 'Version_3.[1-3]' marques-bruno_1.robot

this might be useful if you wanted to test version 8.5 because then it would be:

robot --include 'Version_[1-7].*' --include 'Version_8.[1-5]' marques-bruno_1.robot

Hope that’s what you wanted,

Dave.

1 Like
4

Thanks @damies13 ,
this is precisely what i want! I did not notice in the documentation that i could use wildcards when including tags :slight_smile:
Thanks also for taking the time for an exhaustive example ! :+1:

1 Like