Hi,
I am new to Robot and looking to see if it make senes to assign a score for each test to better quantify the test results. (Instead of just pass and fail)
For instance, we have hostA that is running 20 tests. If hostA fails a critical test versus a minor test, it would both yield a passing score of 19 out of 20 if I logged the test results. However, if I assign a test score for each test result, this would give me a meaningful idea how is hostA doing.
Any thoughts? I have looked into the documentation but couldn’t find much information on exporting and customizing test results.
Hi Leo,
As far as I know in robotframework the only test status’s are Pass Fail and NotRun.
As an option you could use tags, so your critical test cases can be tagged “Critical”, your high Importance test cases “High”, medium and low Importance test cases “Medium” and “Low”, alternatively you could use tags “Importance 1” to “Importance 9” or 001 to 999 however you like as tags are free text.
The result is when you put these tags on your test cases in your default robot test results you get a count of pass failed and skipped for each tag, so you’ll be able to quickly see if the failed test step was a critical test, or a low importance test
Documentation for tagging test cases
Here is an example of statistics by Tag, obviously I use tags differently, but if you if these tags were “Critical”, “High”, “Medium” and “Low” instead of “Column”, “Create”, “Delete”
and “Negative Case”
Hope that helps,
Dave.
Here is an example:
TestScoreExample.robot:
*** Settings ***
Default Tags Criticality 5
*** Test Cases ***
Test A1
No Operation
Test A2
No Operation
Test A3
No Operation
Test A4
Fail
Test A5
No Operation
Test B1
[Tags] Criticality 1
No Operation
Test B2
[Tags] Criticality 1
No Operation
Test C1
[Tags] Criticality 9
No Operation
Test C2
[Tags] Criticality 9
No Operation
Test C4
[Tags] Criticality 9
Fail
@damies13,
For some reason this landed my spam box, but yes it is helpful!
Is it possible to export the statistics to a TSDB?
TSDB? do you mean a Time Series Database? I’ll answer assuming that’s what you mean.
By default robot framework doesn’t have a plugin for that, however there are 2 ways you can do that:
-
post process the output.xml grab the data you want and send that to your TSDB, you can use any programming language you prefer to do this. The times each test step was executed is in the xml so you can use that for the timestamps in your TSDB.
-
Robot framework has a very powerful Listener Interface, basically you create some python functions in a (python) listener file, these functions need to have the names as mentioned in the documentation, as each event happens the function is called, e.g. when a test case ends the end_test function is called and you will have available variables for the test details (name, start time, end time, elapsed time, status, tags, etc) you can then choose which of these details you want and using a python library for your TSDB you can send these values to your TSDB from that function.
I have used both methods for my usage the listener interface is better for me, but RF gives you flexibility so use what ever works best for you.
Dave.
Yes, i am referring to time series database. (I forgot to whitelist the thread notification and missed it)
Looking at the Listener version, it seems version 2 would be just fine in my use case as I am not intended to change it.
Hi Leo,
Yes, Listener version 2 is what I use too, the robotframework team are aware of the functionality from version 2 that is missing in Listener version 3, so Listener version 2 remains supported in robot framework version 4, and seems it will do for quite a while.
Good luck with your project,
Dave.
1 Like
Hi Dave,
Sounds good! Thank you for the suggestion all along.
Leo
