Some of our testcases use the DataDriver library and others use standard data driven templates.
Everything works as expected with Pabot without --testlevelsplit, and running the DataDriver and regular templated testcases separate from each other with --testlevelsplit.
However, when all run together (from a common folder) each process containing a sub-set of DataDriver testcases, also runs ALL the non-DataDriver testcases! This takes a really long time.
Pabot then merges all the results and makes it look like each only ran once - but it takes a really long time.
IF I use Rebot and combine all .\pabot_results\*\output.xml files what really happened becomes obvious.
Adapting the example from github: Snooz82/robotframework-datadriver/issues/70 and inserting the same test commands into github: robotframework/WebDemo/blob/master/login_tests/invalid_login.robot
(I’m not allowed more links)
If I go to extremes and use optimize_pabot=Atomic in the Pabot testsuite, I get: Report.html:
Status:
All tests passed
Documentation:
Pabot result from 15 executions.
Start Time:
20220316 12:31:44.758
End Time:
20220316 12:32:14.538
Elapsed Time:
00:00:29.780
Total Statistics
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
All Tests
32
32
0
0
00:01:04
Combined.html:
Total Statistics
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
All Tests
207
207
0
0
00:06:56
The 25 DataDriver tests are still only run once, while the 7 “Invalid Login” tests are run 26 times.
I hope this is an obvious mistake, but if not I have a sample problem taken from those two sources. I used a clean Python 3.8 venv, and installed
pabot --testlevelsplit --pabotlib .\Test\Pabot_Test.robot
Total testing: 2 minutes 17.70 seconds
pabot --testlevelsplit --pabotlib .\Test\invalid_login.robot
Total testing: 19.99 seconds
It took almost 3x longer to run the robot files together than separately with Pabot. Yet this seems like such a common scenario, that I have to assume it is something I have configured wrong.
(venv) PS C:\Users\burrk\VSCodeProjects\Test> pabot --testlevelsplit --pabotlib .\Test\invalid_login.robot .\Test\Pabot_Test.robot
Robot Framework remote server at 127.0.0.1:8270 started.
Storing .pabotsuitenames file
2022-03-17 10:18:37.563429 [PID:14280] [2] [ID:5] EXECUTING Suites.Invalid Login.Empty Password
...
2022-03-17 10:19:59.417866 [PID:2576] [12] [ID:32] PASSED Suites.Pabot Test in 16.3 seconds
32 tests, 32 passed, 0 failed, 0 skipped.
===================================================
Output: C:\Users\burrk\VSCodeProjects\Test\output.xml
Log: C:\Users\burrk\VSCodeProjects\Test\log.html
Report: C:\Users\burrk\VSCodeProjects\Test\report.html
Stopping PabotLib process
Robot Framework remote server at 127.0.0.1:8270 stopped.
PabotLib process stopped
Total testing: 7 minutes 55.60 seconds
Elapsed time: 1 minute 23.81 seconds
(venv) PS C:\Users\burrk\VSCodeProjects\Test> pabot --testlevelsplit --pabotlib .\Test\Pabot_Test.robot
Robot Framework remote server at 127.0.0.1:8270 started.
Storing .pabotsuitenames file
2022-03-17 10:22:25.946942 [PID:23272] [0] [ID:0] EXECUTING Pabot Test.Verify Most Searched Term ${items}
...
2022-03-17 10:22:49.782654 [PID:15196] [8] [ID:25] PASSED Pabot Test in 3.4 seconds
25 tests, 25 passed, 0 failed, 0 skipped.
===================================================
Output: C:\Users\burrk\VSCodeProjects\Test\output.xml
Log: C:\Users\burrk\VSCodeProjects\Test\log.html
Report: C:\Users\burrk\VSCodeProjects\Test\report.html
Stopping PabotLib process
Robot Framework remote server at 127.0.0.1:8270 stopped.
PabotLib process stopped
Total testing: 2 minutes 17.70 seconds
Elapsed time: 25.30 seconds
(venv) PS C:\Users\burrk\VSCodeProjects\Test> pabot --testlevelsplit --pabotlib .\Test\invalid_login.robot
Robot Framework remote server at 127.0.0.1:8270 started.
Storing .pabotsuitenames file
2022-03-17 10:23:19.238232 [PID:10424] [4] [ID:1] EXECUTING Invalid Login.Invalid Username
...
2022-03-17 10:23:22.451461 [PID:10424] [4] [ID:1] PASSED Invalid Login.Invalid Username in 2.9 seconds
7 tests, 7 passed, 0 failed, 0 skipped.
===================================================
Output: C:\Users\burrk\VSCodeProjects\Test\output.xml
Log: C:\Users\burrk\VSCodeProjects\Test\log.html
Report: C:\Users\burrk\VSCodeProjects\Test\report.html
Stopping PabotLib process
Robot Framework remote server at 127.0.0.1:8270 stopped.
PabotLib process stopped
Total testing: 19.99 seconds
Elapsed time: 4.59 seconds