Set Tags - Can it be used as a counter?

Robot Framework
1

Hi All!

I have a lot of data driven tests that can Fail or Pass in a loop.

Is it possible to use SET TAG to count the passes / failures?

I use this :

    Set Tags                Query Failed

OR
Set Tags Query Passed

At the end of the run, I will only get the value 1 for Total Tests and 1 for Failed.

What I would like is:

Query Failed 2
Query Passed 4

Thanks In Advance!

2

Hi @stevereiss

Would be worth sharing how you increment Query Failed / Query Passed? but I feel as if this may be what you question on the how-to, apologies if I’m mistaken.

This is a tricky one (arguably the report shows PASS and FAIL anyways), but as Tags cant be incremented dynamically for each test (at least without additional code thereafter Id suspect), and Tags are test-level which means you could go as far as setting a Tag against all at the Setting level, or apply them during test execution or at the start, and Tags aren’t accessible at a suite level which limits how you could handle/try to approach this, so you are limited to test case/test teardown for an approach without any additional code thereafter.

So my initial idea, just from a glance was as below:
Create a keyword that checks the test status after it’s completed by using test teardown or extend your current ones (test teardown is executed after the test case has been executed) and increment a suite or global variable ${COUNT_PASSED} / ${COUNT_FAILED}, and then apply a tag for name/count for whichever status PASS/FAIL it falls into via the Set Tag keyword, for which you can make use an Automatic variable ${TEST STATUS} within, which would be the automatic variable ${TEST_STATUS} to check for the status with a simple IF block, and thereafter use a Suite Teardown to remove all minus the uppermost value so only that shows on the report, but as you cant use retrieve @{TEST TAGS} in suite level or use them at this level, this cannot be done in what I theorized as an approach entirely.

So similar to the approach above, you could create a “dud” test that is excluded from the count but applies the Tags thereafter, and you’d be able to order these in a way that this would be the last test ran see How to execute tests in sequential order in robot framework? or similar on the forum

The only downside is a there be a test that’s more of a task shown in the report, possibly this could be excluded without losing the tag count

So an approach could be as below:

*** Settings ***
Resource             keywords.resource
Resource             variables.resource
Test Teardown        Test_Outcome_Check_Status 

*** Variables ***
${COUNT_PASSED}   ${0} 
${COUNT_FAILED}   ${0}

*** Keywords ***
Test_Outcome_Check_Status 
    IF    "${TEST STATUS}" == "PASS" and "${TEST_NAME}" != "Last Test To Assign Tags"
        ${count}=    Evaluate         ${COUNT_PASSED} + 1  
        Set Global Variable       ${COUNT_PASSED}      ${count}   
    ELSE IF  "${TEST STATUS}" == "FAIL" and "${TEST_NAME}" != "Last Test To Assign Tags"
        ${count}=    Evaluate    ${COUNT_FAILED}+1
        Set Global Variable       ${COUNT_FAILED}      ${count}        
    END
    
    IF    "${TEST_NAME}" == "Last Test To Assign Tags"
        Set Tags   Query Passed ${COUNT_PASSED}
        Set Tags   Query Failed ${COUNT_FAILED} 
    END


*** Test Cases ***
Test Case 1
    # Force fail
    Fail    Test Failed

Test Case 2
    # PASSED
    No Operation

Test Case 3
    # PASSED
    No Operation

Last Test To Assign Tags
    # Will do nothing other than as a pointer to set tags as the last test in execution
    No Operation

Outcome shows below:

image
image1072Ă—822 35.6 KB

I’m sure others with more experience could give other approaches, hope that helps nonetheless :slight_smile: but this isn’t something I’ve needed to do or probably will need to do personally, but was a fun one to have a stab at keeping within what RF gives out the box

1 Like
3

Hi Steve,

That’s what would happen if your data driver is treating each data item as a separate test.

See this simple example:

*** Test Cases ***
Test Case 1
	[Tags]	Odd
	${Num}= 	Choose A Number
	Do Something That Might Fail		${Num}

Test Case 2
	[Tags]	Even
	${Num}= 	Choose A Number
	Do Something That Might Fail		${Num}

Test Case 3
	[Tags]	Odd
	${Num}= 	Choose A Number
	Do Something That Might Fail		${Num}

Test Case 4
	[Tags]	Even
	${Num}= 	Choose A Number
	Do Something That Might Fail		${Num}

Test Case 5
	[Tags]	Odd
	${Num}= 	Choose A Number
	Do Something That Might Fail		${Num}

Test Case 6
	[Tags]	Even
	${Num}= 	Choose A Number
	Do Something That Might Fail		${Num}

Test Case 7
	[Tags]	Odd
	${Num}= 	Choose A Number
	Do Something That Might Fail		${Num}

Test Case 8
	[Tags]	Even
	${Num}= 	Choose A Number
	Do Something That Might Fail		${Num}

Test Case 9
	[Tags]	Odd
	${Num}= 	Choose A Number
	Do Something That Might Fail		${Num}


*** Keywords ***
Choose A Number
	${random}= 	Evaluate 	random.randint(0, 10)
	RETURN 	${random}

Do Something That Might Fail
	[Arguments]		${mynumber}
	IF 	${mynumber} > 6
		Set Tags  Number Failed
		Fail 	Number: ${mynumber} is too high
	ELSE

		Set Tags  Number Passed
		Log  	Number: ${mynumber} is ok.
	END

And this in the log:

image
image961Ă—309 42.2 KB

But if I understand correctly the tag could only be applied once per test, so if you’re looping over the data in the same test case you might want something like Set Suite Metadata which will let you assign a value to the metadata name

Here is a simple example of how that might work:

*** Variables ***
${PassCount} 		0
${FailCount} 		0


*** Test Cases ***
One Big test Case
	FOR		${i} 	IN RANGE 	10
		${Num}= 	Choose A Number
		Run Keyword And Ignore Error 	Do Something That Might Fail		${Num}
	END

*** Keywords ***
Choose A Number
	${random}= 	Evaluate 	random.randint(0, 10)
	RETURN 	${random}

Do Something That Might Fail
	[Arguments]		${mynumber}

	IF 	${mynumber} > 6
		${FailCount}= 	Evaluate 	${FailCount} + 1
		Set Suite Variable    $FailCount    ${FailCount}
		Set Suite Metadata  Number Failed 	${FailCount}
		Fail 	Number: ${mynumber} is too high
	ELSE
		${PassCount}= 	Evaluate 	${PassCount} + 1
		Set Suite Variable    $PassCount    ${PassCount}
		Set Suite Metadata  Number Passed 	${PassCount}
		Log  	Number: ${mynumber} is ok.
	END

And the Log:

image
image1303Ă—555 79.7 KB

Hopefully one of those will give you what you’re after,

Dave.

2 Likes
4

Hi Daryl!

Thanks for your reply and coding solution… :smiley:

The Set Tags option did not work out for me…

I was able to use the Set Suite Metadata suggestion that another user posted.
This is what worked correctly:

Set Suite Metadata Mobius Queries Succeeded ${QUERY_SUCCESS}
Set Suite Metadata Mobius Queries Failed ${QUERY_FAILURE}

To increment the counters:

Set Global Variable ${QUERY_FAILURE} ${QUERY_FAILURE+1}
Set Global Variable ${QUERY_SUCCESS} ${QUERY_SUCCESS+1}

Thanks again!

Steve

2 Likes
5

Oh that’s okay, If I was to have picked then the metadata is a much cleaner approach (admittedly didn’t even cross my mind, until it was provided as an approach), but definitely an interesting approach with tags (but they just ain’t built for that, but possible in a not so clean way but relies on test order execution in that way)

Glad @damies13 was able to provide a solution :slight_smile:

All the best

1 Like