Robot command not exiting in Azure pipeline

Tools
1

We have this weird issue where the robot command doesn’t seem to exit when the test suite is finished. This doesn’t happen every time, but if there has been some issues during the tests that made it take longer than usual we often see this issue. The test suite is done but the pipeline doesn’t seem to pick up on it, like the robot command got stuck:

Swrs :: Test suite for the SwRS tests                                 | FAIL |
121 tests, 116 passed, 5 failed
==============================================================================
Debug:   /home/robotest/myagent2/_work/1/a/log/debug.txt
Output:  /home/robotest/myagent2/_work/1/a/log/output.xml
XUnit:   /home/robotest/myagent2/_work/1/a/log/outputxunit.xml
Log:     /home/robotest/myagent2/_work/1/a/log/log.html
Report:  /home/robotest/myagent2/_work/1/a/log/report.html

What could possibly be going on here?

2

Try to force an exit code of 0 (zero).
For example: robot (...) || exit 0

2 Likes
3

Thank you! I will try that.

1 Like
4

Oh, I forgot we already run it with something similar but the other way around:

{
  robot (…)
} || {
  echo "##vso[task.setvariable variable=TestFailure]true"
  exit 1
}

We use that TestFailure variable further down but maybe it’s this part that messes things up? Maybe it’s better to add a step after this with condition: failed() that sets the variable. But wouldn’t robot (…) || exit 0 always return code 0?

5

Yes, it will always pass. For my workflow I will handle the report on other pipeline.

I see you use a lot of curly braces, for me, and when using shell, that is to suppress output. I don’t know if that is your goal or on what scripting shell you are.

2 Likes
6

Well, I changed to this:

- script: |
      set -eu
      ls -lhtr
      source .venv/bin/activate
      robot \
        -d $(Build.ArtifactStagingDirectory)/log/
        -b $(Build.ArtifactStagingDirectory)/log/debug \
        -v hil:"${{ parameters.hil }}" \
        -v semver:"${{ parameters.semver }}" \
        -v filepath:"${{ parameters.file_path }}" \
        -v bb_dir:"$(Build.ArtifactStagingDirectory)/log/" \
        -i "${{ parameters.test_level }}" \
        -x outputxunit.xml \
        --loglevel DEBUG:INFO \
        --listener RetryFailed \
        --listener "HIL-rig/python/AddTestSpecId.py" \
        HIL-rig/tests/"${{ parameters.test_suite }}"
    displayName: Run tagged test-suite
    continueOnError: ${{ parameters.continueOnError }}

  - script: |
      echo "##vso[task.setvariable variable=TestFailure]true"
    condition: failed()
    displayName: Set TestFailure variable

But no joy, it still gets stuck sometimes (not always) even though the whole test suite has finished.

Any ideas? I don’t see how robot … || exit 0 would help me here since I need to know if the tests failed.

Edit: I’ll try to set failOnStderr: trueand see if that helps.

Edit 2: That didn’t work at all since it made the task exit with an error code even though all tests passed

8

I still don’t know what’s causing the robot command to not exit when done, I read somewhere that it might be the listeners but that seems unlikely. I did however find a workaround that seems to work:

- script: |
    set -eu
    ls -lhtr
    source .venv/bin/activate
    robot \
      -d $(Build.ArtifactStagingDirectory)/log/ -b $(Build.ArtifactStagingDirectory)/log/debug \
      -v hil:"${{ parameters.hil }}" \
      -v semver:"${{ parameters.semver }}" \
      -v filepath:"${{ parameters.file_path }}" \
      -v bb_dir:"$(Build.ArtifactStagingDirectory)/log/" \
      -i "${{ parameters.test_level }}" \
      -x outputxunit.xml \
      --loglevel DEBUG:INFO \
      --listener RetryFailed \
      --listener "HIL-rig/python/AddTestSpecId.py" \
      HIL-rig/tests/"${{ parameters.test_suite }}" &
    pid=$!
    
    # Wait for the tests to finish (outputxunit.xml is created)
    until [ -f $(Build.ArtifactStagingDirectory)/log/outputxunit.xml ]; do
      sleep 5
    done

    echo "XUnit file created!"

    # Wait a bit to give robot time to complete
    sleep 30
    
    # Kill process if still running
    if [ $(ps aux | grep -Pc "\s+$pid\s+") -gt 0 ]; then
      echo "Robot is stuck, killing it now"
      kill $pid
    else
      echo "Robot exited on its own"
    fi

    # Check if there are failed tests
    failures=$(grep -c "failure message" $(Build.ArtifactStagingDirectory)/log/outputxunit.xml)
    if [ $failures -gt 0 ]; then
      echo "There are failed tests"
      exit 1
    fi

    echo "No failed tests"
    exit 0
  displayName: Run tagged test-suite
  continueOnError: ${{ parameters.continueOnError }}

I’m sure there are better ways but if it works it’s good enough for me.

9

is the underlying OS on your test runner a windows machine ? If yes, do you happen to run any any external processes. If again, yes. This is known issue with windows and nothing to do with robot. You need to read all the input from external processes and if you don’t, issue like yours happens..

10

Nope, it’s running on a Linux machine.

11

and you are sure you have killed/stopped all possible child processes correctly ?

12

I don’t have any external/child processes. I’m not even sure what you mean with that to be honest :slight_smile: