Running multiple test suite directories in parallel and sequential execution of test suites within test suite directory

Tools
1

Hi, all.
Is there a way to run multiple test suite directories in parallel where test suites inside each test suite directory run in sequential mode using pabot? e.g:

  • test/dir1 contains suite1.robot, suite2.robot, suite3.robot
  • test/dir2 contains suite4.robot, suite5.robot, suite6.robot
  • test/dir3 contains suite7.robot, suite8.robot

I want to run all directories in parallel (dir1, dir2, dir3 running in the same time), but test suites within dir run in sequential mode. I have created my script like this:

pabot --processes 3 test/dir1 test/dir2 test/dir3

It seems that the script is not correct. What should I write to run this case?
Thank you

2

Hi,

I think you can do something like this by using arguments files :

pabot --processes 3 --argumentfile1 argfile_dir1.txt --argumentfile2 argfile_dir2.txt --argumentfile3 argfile_dir3.txt

In each argfile_dir[i].txt you can set the needed suites and order, something like this (example for dir1) :

--suite suite1
--suite suite2
--suite suite3
test/dir1

Regards
Charlie

1 Like
3

Hi,

Just for whom that could be interested, on my side which is near from this request, I finally used the – ordering parameter with order.txt file. This gave better results and behavior than arguments for what’s expected:

{
--suite test.dir1.suite1
--suite test.dir1.suite2
--suite test.dir1.suite3
}
{
--suite test.dir2.suite4
--suite test.dir2.suite5
--suite test.dir2.suite6
}

With a command like this:

pabot --processes 2 --ordering order.txt .\test

There groups are executed on two bots/processes with sequential order in the group.

This works well also in docker (ppodgorsek one used here) by passing the $pabot_options (only --ordering order.txt in my case):

    docker run --name robotcontainer -e ROBOT_THREADS=2 -e PABOT_OPTIONS="$pabot_options" -e ROBOT_OPTIONS="$robot_options" -v ${projectfolder}\ResultsDocker:/opt/robotframework/project/Results -v ${projectfolder}:/opt/robotframework/project image-docker:latest

Regards
Charlie

1 Like
4

Hi Guys,

Above solution was working on a simple example, however I’m quite struggling on the setup I want to have.
I’m having issues and wonderings about the naming, case and correct read of the order.txt.
Sorry already for the amount of info below, but couldn’t have it simplier

To sum up, I have a structure like this :

zidework\
├── Tests\
│   ├── GLOBAL\
│   │   ├── TYPEA\
│   │   │   ├── TYPEA1.robot
│   │   │   ├── TYPEA2.robot
│   │   ├── TYPEB\
│   │   │   ├── TYPEB1.robot
│   │   │   ├── TYPEB2.robot
│   │   ├── TYPEC\
│   │   │   ├── TYPEC1.robot
│   │   │   ├── TYPEC2.robot
│   │   ├── TYPED\
│   │   │   ├── TYPED1.robot
└── order.txt

My need is to be able to group by “TYPEx”, for example A and C grouped, parallellized against B and D group. The .robot files might contain one or plus tests.

1. Standard Pabot exe and .pabotsuitenames

Simple pabot run pabot --processes 2 --dryrun . gives me this .pabotsuitenames file (OK):

--suite Zidework.Tests.GLOBAL.TYPEA.TYPEA1
--suite Zidework.Tests.GLOBAL.TYPEA.TYPEA2
--suite Zidework.Tests.GLOBAL.TYPEB.TYPEB1
--suite Zidework.Tests.GLOBAL.TYPEB.TYPEB2
--suite Zidework.Tests.GLOBAL.TYPEC.TYPEC1
--suite Zidework.Tests.GLOBAL.TYPEC.TYPEC2
--suite Zidework.Tests.GLOBAL.TYPED.TYPED1

Here already I see that the main root folder is named with “Z” uppercase instead of “z”

2. Pabot order.txt

I set an order.txt file as following (simple copy from the generated one, grouped, and verified that if order.txt is missing got an error)
pabot --processes 2 --ordering order.txt --dryrun .

{
--suite Zidework.Tests.GLOBAL.TYPEA.TYPEA1
--suite Zidework.Tests.GLOBAL.TYPEA.TYPEA2
--suite Zidework.Tests.GLOBAL.TYPEB.TYPEB1
--suite Zidework.Tests.GLOBAL.TYPEB.TYPEB2
}
{
--suite Zidework.Tests.GLOBAL.TYPEC.TYPEC1
--suite Zidework.Tests.GLOBAL.TYPEC.TYPEC2
--suite Zidework.Tests.GLOBAL.TYPED.TYPED1
}

But got pabot running error :
[ ERROR ] Suite ‘Zidework’ contains no tests or tasks in suites, failed with 252 tests:

Only way it works is renaming Zidework to zidework (lowercase). There execution is made and seems to be made by group, even though “group” is not mentioned.

3. Expected/needed structure

I set without the robot file as below which would be my behavior (to avoid update of order.txt as soon as a test is added), but got the same error. Only way it works is renaming “zidework” (case issue with z), but in this case the execution is not made by group too…

{
--suite Zidework.Tests.GLOBAL.TYPEA
--suite Zidework.Tests.GLOBAL.TYPEB
}
{
--suite Zidework.Tests.GLOBAL.TYPEC
--suite Zidework.Tests.GLOBAL.TYPED    
}

4. Simple order test

Finally I just try to verify forced order execution:

--suite zidework.Tests.GLOBAL.TYPEC
--suite zidework.Tests.GLOBAL.TYPED
#WAIT
--suite zidework.Tests.GLOBAL.TYPEA
--suite zidework.Tests.GLOBAL.TYPEB

Here it doesn’t work (tests run in parallel with no order nor wait taken in account), unless I rename “Zidework” with first letter in upper case (quite the reverse behavior as explained above).

If somebody already experienced issues like this, I would be interested in knowing how you managed this.
Already tried also to narrow down suites names in order.txt, pass --suite parameter in robot command, with no more luck. Also changed and tested encoding of file (CRLF/LF).

Thanks and regards.
Charlie