I’m mapping a Linux shell script to Python script to run my test. The Python script/commands are sent to a hardware DUT via a UART port. I’m using Python-based keyword to generate my RF keyword. Currently, those three (3) files – two .robot files and one .py file, run well except that I don’t know how to capture the wording “test passed” or “test failed” from “output.xml” file, which will need to be reflected on the RF COMMAND LINE OUTPUT, log.html and report.html, respectively. It means that if the testing fails due to product/DUT bug(s), the overall results should show “FAIL” on RF COMMAND LINE OUTPUT, log.html and report.html, respectively (even if there are no any errors such as invalid syntax, timeouts, or fatal exceptions, ….caught by keywords).
I tried TRY/ECXEPT/ELSE/END functionality and “Run Keyword” and “Run Keyword And Continue On Failure” (I believe that I didn’t make them completely right). They didn’t capture “test passed” or “test failed” from “output.xml” file (Note: the RF COMMAND LINE OUTPUT on Linux terminal can’t display shell script results such as the wording “test passed” or “test failed”, but which can be displayed on the Hyper terminal/TeraTerm associated with the hardware DUT). Could any of you provide me with a detailed command example on how to solve this issue (I’m relatively new on RF)?
My env. – Robot Framework 6.0.1 (Python 3.8.10 on linux)
Thank you so much for your time and help in advance!
It’s really not clear what you’re trying to do here:
run 1 robot from another?
run a robot from a shell/batch file?
run a robot from a python script?
something above being run from another robot file?
something else?
I’ll try to help anyway,
If all you need to know is “did all the test cases pass or not?” then the simplest way to find out is not parse the out put text but simply read the return code, robot (robot.exe on windows) is POSIX compliant (Windows, Mac and Linux are all POSIX environments) so the return code is 0 if everything is ok and some other integer if there was an error, in the case of robot framework the Return codes are well documented.
If you do need to parse the text then it’s returned on the stdout so the calling process needs to capture the stdout and then do something with it (in robot framework I’d use something like Should Not Contain ${stdout} FAIL
Another option that might be easier is use an xml library to read output.xml and get the test status from there.
Hi Dave,
Thank you so much for your kind response!
I ran one test case .robot file (under testsuites) using a normal way “python3 -m robot -d ….”
My question is that if I don’t parse some result file to capture the keyword of “test failed” due to DUT defect(s), the RF COMMAND LINE OUTPUT, log.html and report.html always show PASS info. As I understood, for RF " Run Keyword And Expect Error" command and TRY/EXCEPT/ELSE/END functionality, the “error messages” are referred to the report.html and log.html files while my desired test results keyword (“test passed” or “test failed”) is in output.xml. In other words, I can’t directly use RF keywords to do error handling tasks. So, for my case, not sure if I can use your suggested Return codes (rc) method. Regardless, could you provide me with a detailed example of using rc.
The COMMAND LINE OUTPUT when I run “python3 -m robot -d ….” won’t show the DUT “test passed” or “test failed” info (will be displayed on output.xml, though) because that info is derived from running a shell script via the UART port (connected to a hardware DUT).
Regarding “Another option that might be easier is use an xml library to read output.xml and get the test status from there”, my understanding is that I should have additional RF keywords for the RF test case (under testsuites) to read output.xml and capture “test passed” or “test failed” info. My question is – if the “test failed” info is captured, how can it overwrite/affect the overall PASS/FAIL test results displayed on RF COMMAND LINE OUTPUT, log.html and report.html, respectively.
BTW, would you kindly provide an example to use xml library to read output.xml?
python3 -m robot -d still returns the return code and stdout the same as the robot command so you can still test the same way
From the builtin Library set XML should do what you need for reading output.xml
Here’s an example of testing the return code:
CAL_Greg_1.robot
*** Settings ***
Library Process
*** Variables ***
${TestFile} ${CURDIR}${/}CAL_Greg_2.robot
${TestOutput} ${CURDIR}${/}CAL_Greg_2
@{robotcmd} robot -d ${TestOutput}_r ${TestFile}
@{pythocmd} python3 -m robot -d ${TestOutput}_p ${TestFile}
*** Test Cases ***
Test Run Robot
Log ${robotcmd}
${result}= Run Process @{robotcmd}
Log ${result.rc}
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers 0 ${result.rc}
Should Not Contain ${result.stdout} | FAIL |
Test Run Python3
Log ${pythocmd}
${result}= Run Process @{pythocmd}
Log ${result.rc}
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers 0 ${result.rc}
Should Not Contain ${result.stdout} | FAIL |
CAL_Greg_2.robot
*** Settings ***
*** Variables ***
*** Test Cases ***
Test Always Pass 1
Pass Execution Always Pass 1
Test Always Pass 2
Pass Execution Always Pass 2
Test Always Pass 3
Pass Execution Always Pass 3
Test Might Fail
${random} = Evaluate random.randint(0, 10)
IF ${random} > 5
Pass Execution Test Might Fail
ELSE
Fail Test Might Fail
END
For some reason, Test Run Python3 always fails with python3: No module named robot and RC = 1, I don’ t know why, maybe you need to use the full path to your python executable or set an environment variable. but this should give you enough of an idea.
Hi Dave,
Thank you so much again for your quick reply!
I’m sure I will try to understand/follow your suggestions/solutions later.
Now I have a related question (before I saw your reply, I had got the link below).
python - Syntax for using the Run Process keyword in RobotFramework - Stack Overflow
where there is an argument “Test123” . What is that? Is it the redirected output file (from stdout) after running MyPythonScript.MyFunction(). … It seems that the provided answer doesn’t work for my case. Any suggestions?
Thank you!!!
Greg
It appears to be just a text string thats passed to MyPythonScript.MyFunction(), without knowing what MyFunction() does who knows? it really doesn’t matter as we are not using that function.
That article did help me to figure out how to chanege CAL_Greg_1.robot to make the python version run, here’s the updated version:
CAL_Greg_1.robot
*** Settings ***
Library Process
*** Variables ***
${TestFile} ${CURDIR}${/}CAL_Greg_2.robot
${TestOutput} ${CURDIR}${/}CAL_Greg_2
@{robotcmd} robot -d ${TestOutput}_r ${TestFile}
@{pythocmd} python3 -m robot -d ${TestOutput}_p ${TestFile}
*** Test Cases ***
Test Run Robot
Log ${robotcmd}
${result}= Run Process @{robotcmd}
Log ${result.rc}
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers 0 ${result.rc}
Should Not Contain ${result.stdout} | FAIL |
Test Run Python3
Log ${pythocmd}
${result}= Run Process @{pythocmd}
Log ${result.rc}
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers 0 ${result.rc}
Should Not Contain ${result.stdout} | FAIL |
Test Run Python3 ML
${result}= Run Process
... python3
... -m
... robot
... -d
... ${TestOutput}_p
... ${TestFile}
Log ${result.rc}
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers 0 ${result.rc}
Should Not Contain ${result.stdout} | FAIL |
Thanks for your quick response again!!
Yes, the article is the right one I referred to.
I used your above suggestions (Test Run Python3 and Test Run Python3 ML, respectively) to run the tests. I can see the return code (rc) values. However, they aren’t what I want as I stated on 12/05/22 reply – the rc values don’t reflect the DUT test results (“test passed” or “test failed”). What I tried to do –
Run a python test like “python3 /home/Greg/PycharmProjects/RF/Libraries/Scripts /greg.py > /home/Greg/PycharmProjects/RF/Results/greg_test” under *** Keywords *** (or, *** Test Cases ***) section.
${result}= Run Process
… python3
… -c ------------------>>>>>> (I also tried w/o “-c”)
Use a python function to capture the “test passed” or “test failed” info from “greg_test” file of the 1) above.
Use RF builtIn keywords such as
Should Be True ${result.rc} == 0 Error while running testcase
Should Be True ${test_result} == 0 ${msg}
to process my DUT test status (the “test passed” or “test failed”) which will be automatically reflected to the RF COMMAND LINE OUTPUT, log.html and report.html. In other words, if the “test failed” is captured, the RF COMMAND LINE OUTPUT, log.html and report.html will show FAIL (red color).
Right now my question is – the above ${result}= Run Process …… syntax is wrong (I think) because I can’t find out the redirected output file “greg_test”. Note that if I intended to make a “greg_test” file directly via “python3 /home/Greg/PycharmProjects/RF/Libraries/Scripts /greg.py > /home/Greg/PycharmProjects/RF/Results/greg_test” (instead of running the above ${result}= Run Process …… ). I can see the expected info on RF COMMAND LINE OUTPUT, log.html and report.html, respectively.
Would you kindly tell me the right syntax on how to run “python3 /home/Greg/PycharmProjects/RF/Libraries/Scripts /greg.py > /home/Greg/PycharmProjects/RF/Results/greg_test” via “${result}= Run Process …“. I also checked Process (robotframework.org). Regardless, I couldn’t find out the redirected output file “greg_test”.
Thank you so much again for your time and help!!
Greg
re: -c you seem to be missing the command after it, because -c is used when you have a simple (usually single line) piece of python cod you want to execute e…g python3 -c "print('7 + 3 =',7+3)". In your case you are executing a py file so you shouldn’t need -c
You might however need -u since you are redirecting the stdout to a file?
I would suggest you start simple:
${result}= Run Process
… python3
… /home/Greg/PycharmProjects/RF/Libraries/Scripts/greg.py
Log ${result.rc}
Log ${result.stdout}
Log ${result.stderr}
Using this method what you would have sent to the file /home/Greg/PycharmProjects/RF/Results/greg_test will be in ${result.stdout} making it easier to check it’s contents for expected strings or confirm it doesn’t contain undesirable strings (e.g. Should Not Contain or Should Contain and similar keywords)
I’m also wondering if it’s the > character that’s giving you trouble, you might need to escape it :\>
Hi Dave,
Thank you so much for your continued support!!
I used -u option but still could NOT see an expected redirected file.
I used your suggestion as below, it works! !
${result}= Run Process
… python3
… /home/Greg/PycharmProjects/RF/Libraries/Scripts/greg.py
Log ${result.rc}
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers 0 ${result.rc}
Should Contain ${result.stdout} passed
The only issue is that there are too many test results contents on RF COMMAND LINE OUTPUT. So, I still want to figure out how to redirect stdout to a file under “${result}= Run Process ……” . I would greatly appreciate it if you could help me again on this matter!!!
Greg
Actually the answer to that is in the documentation for Run Process
you simply add stdout=<path where you want the file>, you can also add stderr=STDOUT to redirect standard error to standard out and combine them to appear as they would on the command line.
You could also send the standard error to it’s own seperate file if you want. stderr=<path where you want the file>
${result}= Run Process
… python3
… /home/Greg/PycharmProjects/RF/Libraries/Scripts/greg.py
… stdout=/home/Greg/PycharmProjects/RF/Libraries/Scripts/greg.txt
… stderr=STDOUT
Log ${result.rc}
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers 0 ${result.rc}
Should Contain ${result.stdout} passed
What’s great about this is the output is in the text file for reading and still also available in the variable for testing the result.
Don’t forget about the Built-in variables (e.g. ${CURDIR}) that you can use in place of fixed paths (useful if you share this code with team members via a source control system)
Hi Dave,
Thank you so much again for your time and continued support!!
Actually, adding
“… stdout=/home/Greg/PycharmProjects/RF/Libraries/Scripts/greg.txt” can’t solve my initial “issue” – “….there are too many test results contents on RF COMMAND LINE OUTPUT”. I also removed “… stderr=STDOUT”, which still can’t solve the “issue”. However, when I went back to Robot Framework User Guide (Ver. 6.0.1), section 3.1.3 Test results/Command line output (see below)
Second test | FAIL | Error message is displayed here
I realized that when the test case fails, supposedly there will have error messages (one line or many lines, depending on your specific test case) displayed on the Command line output. Right? If you have the same understanding as mine, I think I can close this ticket.
Again, I really appreciate your time and kind help!!
Greg
