Few scripts fail on jenkins

Robot Framework
1

I have 20 automation scripts that are placed on GitHub and written using the Robot framework with Python, and I execute them from Jenkins. When I execute all of them together, a few of them fail, but when I execute them one by one, they all work. How can I resolve this situation so that all scripts execute and pass in one go.

2

Hi Urvashi,

Without some idea what’s causing the failures, it’s difficult to know what to suggest.

  • What are the errors you are getting when the tests fail, this is usually a pretty good clue to what’s causing the test to fail.
  • What library (libraries?) are you using?
  • What do you mean by “When I execute all of them together” as opposed to “when I execute them one by one”, this could mean quite a few different things.

Dave.

3

Hi Dave,

So I was asked to execute all the test cases from Jenkins, ,so once I execute all the scripts should execute one after another. Also, I get errors like some element not found, etc., but when I execute that failed script solo, it passes.

4

Also this,
ElementNotInteractableException: Message: element not interactable

5

Hi Urvashi,

  • What do you mean by “When I execute all of them together” as opposed to “when I execute them one by one”, this could mean quite a few different things.

When I asked this I was more looking for the command you run, or maybe that Jenkins runs.

It shouldn’t really matter whether robot was run by Jenkins, by your IDE or manually from the command line, what matters is the command line arguments to robot when you ran. Can you get the robot command used by Jenkins when the tests all worked and when some failed? The difference between them might give us a clue to what the problem is.

This sounds like a timeout type issue, when you "execute all of them together” the application is running a bit slower for some reason.
You didn’t mention which Library you are using but most of them have keywords that start with “Wait Until” or “Wait For”, so you probably need to use one of these wait keywords to wait for that element to be ready or available for interaction.

Dave.

6

What @damies13 said…

ElementNotInteractableException hints towards SeleniumLibrary. These sort of issues sometimes happen when user tries to interact with element and the state of the SUT is not expected. Running test on desktop without any formal load and things work fine but when, say, running on headless browser on jenkins slave where there’s cpu or io load, clicks might not register, responses from backend might take more time or even fail completely and so forth.

I’d say that the issue is that the actual tests are not just robust enough and someone needs to take a look how the UI interaction is done and verification for state transitions needs to be actually verified. “Wait Until Keyword Succeeds” is the typical approach in some of these sort of cases.

3 Likes
7

Generally spoken, if you run such a test in isolation, run it 10 to 20 times. Many of these type of failures don’t fail every time, they fail a few percent of the times. You might have to run a particular test a hundred times in isolation to be reasonably sure it’s stable (and thus the instability on the pipeline is caused by something else).

Are you running the tests in a Docker image when running in Jenkins? If so it can be interesting to take a look at Visual Studio Code devcontainers; if you run in a devcontainer build on the same Docker image, you’re running the tests in an environment that’s very close to the pipeline. I believe you can also assign system resources to a devcontainer, which allows you to give the devcontainer about the same “hardware” as your Jenkins node. That way, you can get the timing as close as possible to the pipeline situation.

1 Like