I’ve upgrade from v14.x to v16.x because I have a need for the ‘Get Console Log’ keyword which has been introduced in v16. Changes I made are:
Updated Robotframework to 6.0.2 (was 6.0.1)
Updated Robotframework-browser to 16.0.0 (was 14.1.0)
Added run_on_failure=Take Screenshot And Fetch Console Log to the Browser library include in all files where it was referenced (and naturally implemented that keyword in my resource files)
Our tests are executed in a pipeline with quite consistent execution durations (was 23 minutes). After these changes duration increase to 36 minutes. Quite a big increase.
To verify if the ‘run_on_failure’ keyword was causing the increase in runtime duration, I removed this parameter and ran the tests again. Unfortunately with the same results.
Is anybody else experiencing increase in runtime duration after switching to v16 of Browser library?
Please find the results below. As far as I can tell it’s an increase across all keywords. I’ll revert my changes entirely (go back to v14 of Browser library) to see if this will reduce durations again.
Ran onother test on the same environment with Browser library v14 and went back from 36 minutes to less than 23 minutes. This confirms that the increase in duration is related to the new version of Browser Library.
Only 3 Browser keywords are visible in those screenshots
Keyword
Browser v14
Browser v16
Browser.Wait For Elements State
00:00:00.035
00:00:01.035
Browser.Click
00:00:00.588
00:00:00.542
Browser.Get Element
00:00:00.019
00:00:00.512
Interestingly Browser.Wait For Elements State is exactly 1 second longer in Browser v16, also Browser.Get Element is ~500ms longer
I wonder if any other keywords have a similar issue? Could you provide the output.xml files for both v14 and v16, or diff them with a diff tool to see if any other browser keywords have such big increases?
I ran the tests with both browser versions in two environments (one from within a build agent and one from my local machine). When I switched back to v14 durations decreased again. So my first reaction would be that it has something to do with the version of the Browser library. However, I will try to run some additional tests where I will try to ‘freeze’ the context as much as possible. I’ll share my results when I have them available.
I’ve finally been able to get some numbers together.
Scenario: run the same script from the same machine against the same environment. First time with Browser Library v14.x, second time with Browser library v16. No other package differences. Part of the difference can be related to the environment not always being equally fast, but I’ve ran these tests so many times that this can’t be coincidence anymore.
It seems like the difference is limited to a few keywords. ‘Get Element States’ and ‘Wait For Element State’ being the biggest differences.
Excel seems to have some issues with displaying / calculations on durations with ms (‘Get Text’ on v14 has an average duration of 0ms), but I’ve checked a few values manually and the difference in average duration seems to be correct.
We’re currently using v17.2.0 of robotframework-browser. Speed is back to what is was before (or even faster). Can’t tell whether this changed when moving from v16 to v17 or earlier.
