VS Code - How to only run 'not yet run' tests from a suite?

Tools Visual Studio Code(ium)
1

Hello, I am using VS Code to run my Robot Framework tests. I mainly use the Test Explorer view (‘Testing’) for this. I use the extensions Robot Framework Language Server and Pylance + Python.

I have several .robot-files (suites) that contain several tests each. In the Test Explorer view I can start a single test or a whole suite by clicking on the play button icon (‘run test’). There’s also the option to ‘rerun failed tests’. However, I am looking for an option to only run tests from a suite that have not yet run. So I can skip the passed or failed tests from a suite but only run the ones that have not yet run. I haven’t been able to find this option and in this case I need to manually run each test case one after the other. I thought maybe I could put a breakpoint in the .robot-file and run everything below from there, but there’s no option for that.
So does anyone know if it’s possible to only run tests that have not yet run?

image001
image001522×549 18.6 KB

Another question related to this: when using the Test Explorer view and running tests, every passed test will get a green checkmark and every failed test will get a red cross. I use this to track my progress while testing the complete suite. However, sometimes things happen like VS Code will crash and all the green/red icons in front of the tests are gone. So now I have no visual clue anymore to track which tests I already ran. I was wondering where these results are stored? I guess in output.xml, but I haven’t been able to restore my results after a crash. Is there a way to use the Test Explorer view and run tests that way and somehow save/backup/restore the test results in case of a crash or other event? I know I can also use the command line to run tests and save results in reports with custom names etc, but I just like to use the Test Explorer view and press a play button and track the results by looking at the red or green icons :slight_smile: Thanks!

3

Thanks David. I’m not sure if this is what I was looking for, though. About where the results are stored: What I mean is: in the Test Explorer view, tests can be marked green for passed and red for failed. See the screenshot above. Whenever VS Code crashes, these results are gone. So I am wondering where this information is stored and how I can retrieve it. I don’t think it’s in the output.xml file? Or maybe it is, but since I run my tests fragmented (some single test runs, some suites, etc.), the output.xml file gets updated/overwritten everytime. I am not sure how to backup this file and place it back after a crash so it contains all the tests I have been running during a day (instead of only the last run before the crash). And how to make VS Code show these results again in the Test Explorer view.

Also, I like to just hit play buttons to start a test, but I only have play buttons for a single test or a whole suite, or a rerun of failed tests. I wish there was a way to hit a play button to only run ‘unrun’ tests (within a suite). Now I either have to run the whole suite again (including tests that were already passed), or manually run every unrun single test.

For context: Testing

4

Hi @JST81,

I dig a little bit about this, as I saw that from the left test panel it always opens the Test Results tabs of your terminal.
And there :

  • You can see that the tests statuses are the same
  • They are linked to the workspace you’re in
  • If you clear here the “tests” terminal, this reset only the workspace test statuses (as in a crash)

Finally I found out that those infos are stored here in .json after execution:

C:\Users\<yourname>\AppData\Roaming\Code\User\workspaceStorage\<longID>\testResults

<longID> is a random number linked (it seems) to the current test session of the workspace (remains the same prior to reset/clear of the terminal or tests results).

Then in the testResults folder, you’ll find a new .json each time your run a suite or a test.
So my guess would be to be able to make a backup of the .json, but I tried this that doesn’t seem to work:

  • Remove/backup .json (as a crash) => reload VsCode : no more statuses
  • Paste .json => reload VsCode : statuses not reloaded…
  • Run a test => a new .json is created

Maybe you can dig this way see if you can work this out.
About the not runned tests, I would dig the VsCode tasks possibilities, or eventually try to use tags added dynamically to your tests (pass, fail, notrun) to be able to rerun them easily.

Regards.
Charlie

5

This is some good info, thank you. I’m using a cloud workspace for work, and I noticed on Mondays the test results are always gone. I guess that is because the workspace is rebooted every weekend, so it seems the TestResults.json is somehow tied to a session (like you mentioned). Also after a crash of VS Code the results are gone, so it seems that also creates a new session. Let’s try to find out more about how this works and how previous test results can be restored. I find it very odd that VS Code offers this Test Explorer view, with play buttons and visual confirmation of passed or failed tests, but the results are stored in such a brittle way. It seems you have to complete your entire (set of) test suite(s) (I have over 300 tests) in one ‘session’, or else you will lose the results (after a crash or reboot). Strange.