Browser library v16 seems to be a lot slower than v14

Libraries Browser
1

I’ve upgrade from v14.x to v16.x because I have a need for the ‘Get Console Log’ keyword which has been introduced in v16. Changes I made are:

  1. Updated Robotframework to 6.0.2 (was 6.0.1)
  2. Updated Robotframework-browser to 16.0.0 (was 14.1.0)
  3. Added run_on_failure=Take Screenshot And Fetch Console Log to the Browser library include in all files where it was referenced (and naturally implemented that keyword in my resource files)

Our tests are executed in a pipeline with quite consistent execution durations (was 23 minutes). After these changes duration increase to 36 minutes. Quite a big increase.

To verify if the ‘run_on_failure’ keyword was causing the increase in runtime duration, I removed this parameter and ran the tests again. Unfortunately with the same results.

Is anybody else experiencing increase in runtime duration after switching to v16 of Browser library?

Björn

2

Hi, i actually did not compare.
But can you try to compare two test cases?
Are there specific keywords that take more time?

3

Hi René,

Please find the results below. As far as I can tell it’s an increase across all keywords. I’ll revert my changes entirely (go back to v14 of Browser library) to see if this will reduce durations again.

Browser v14

image
image806×568 46.2 KB

Browser v16

image
image788×584 28.5 KB

Björn

4

Ran onother test on the same environment with Browser library v14 and went back from 36 minutes to less than 23 minutes. This confirms that the increase in duration is related to the new version of Browser Library.

Kind regards,

Björn

5

Hi Björn,

Only 3 Browser keywords are visible in those screenshots

Keyword Browser v14 Browser v16
Browser.Wait For Elements State 00:00:00.035 00:00:01.035
Browser.Click 00:00:00.588 00:00:00.542
Browser.Get Element 00:00:00.019 00:00:00.512

Interestingly Browser.Wait For Elements State is exactly 1 second longer in Browser v16, also Browser.Get Element is ~500ms longer

I wonder if any other keywords have a similar issue? Could you provide the output.xml files for both v14 and v16, or diff them with a diff tool to see if any other browser keywords have such big increases?

Dave.

6

We have checked our acceptance tests of both versions and they were more or less same.

We still will check that on our side.

Regards
René (Browser lib core-team)

7

I double checked it for 1.5 hours now.

rerunning our 650 test cases with 14 and 16
Of course there are difference in runtime, but all i saw was just because of more new tests in 16.0.0.

so going through the suites Playwright 1.25.2 (Browser 14.0.0) and Playwright 1.30.0 (Browser 16.0.0) seems to also make no difference.

In your log for example the keyword Wait For Elements State took more time, but we did not touch anything there.

At the moment i would still guess to Performance differences on the machines or the testobject

8

Hello René,

I ran the tests with both browser versions in two environments (one from within a build agent and one from my local machine). When I switched back to v14 durations decreased again. So my first reaction would be that it has something to do with the version of the Browser library. However, I will try to run some additional tests where I will try to ‘freeze’ the context as much as possible. I’ll share my results when I have them available.

Thanks for diving into this!

Kind regards,
Björn

1 Like
9

Hi,

I’ve finally been able to get some numbers together.

Scenario: run the same script from the same machine against the same environment. First time with Browser Library v14.x, second time with Browser library v16. No other package differences. Part of the difference can be related to the environment not always being equally fast, but I’ve ran these tests so many times that this can’t be coincidence anymore.

image
image1654×1074 70.7 KB

It seems like the difference is limited to a few keywords. ‘Get Element States’ and ‘Wait For Element State’ being the biggest differences.

Excel seems to have some issues with displaying / calculations on durations with ms (‘Get Text’ on v14 has an average duration of 0ms), but I’ve checked a few values manually and the difference in average duration seems to be correct.

Kind regards,
Björn

10

And the same data, but now using testarchiver and Excel to format the data.

Column Labels
1 2 Total Cumulative count Total Average duration
Row Labels Cumulative count Average duration Cumulative count Average duration
Check Checkbox 73 557,9462366 88 1070,847059 161 814,3966477
Clear Text 9 658,0833333 11 675,2222222 20 666,6527778
Click 823 586,3662496 965 574,2060869 1788 580,1019234
Close Browser 22 993,1818182 22 1378,227273 44 1185,704545
Fill Secret 22 528,3636364 22 528,7272727 44 528,5454545
Fill Text 133 530,0912607 149 530,1369225 282 530,1140916
Get Checkbox State 94 10,42553191 102 518,6666667 196 264,5460993
Get Element 209 59,67565651 224 564,4065092 433 312,0410829
Get Element Count 1 6 1 6
Get Element States 77 816,9095833 86 3171,517663 163 2039,494548
Get Page Ids 131 4,007633588 131 3,938931298 262 3,973282443
Get Text 113 14,26779101 127 968,919401 240 506,9912026
Go To 109 1068,055046 109 1047,036697 218 1057,545872
New Browser 22 367,4545455 22 382,3636364 44 374,9090909
New Context 22 10,04545455 22 8,681818182 44 9,363636364
New Page 22 5412,909091 22 6467,272727 44 5940,090909
Set Browser Timeout 254 2,791658909 262 2,79389313 516 2,792776019
Uncheck Checkbox 31 540,8064516 34 1051,764706 65 796,2855787
Wait For Elements State 253 57,13763829 275 679,7654978 528 352,8858716
Grand Total 2419 479,1802084 2674 1204,356618 5093 847,2621737
11

I expect the last one to be more accurate, because I noticed that Excel has some issues with calcuations on time values with milliseconds.

12

Hi @René ,

Did you by chance find some time to have a look at the additional figures which I posted in this topic?

Thanks in advance,
Björn

13

We’re currently using v17.2.0 of robotframework-browser. Speed is back to what is was before (or even faster). Can’t tell whether this changed when moving from v16 to v17 or earlier.

1 Like