FlaUI, Zoomba and AutoIT are all Windows only, so if you are testing a windows only app, these are good choices.
FlaUI may not work with every app, but will work with many, it’s advantage is you can use xpath’s to directly identify the object (button, field, etc) that you want to interact with, making your tests more reliable
AutoIT is higher level and can’t directly identify objects in the window, but gives you control of the keyboard and mouse, so you may need to use shortcut keys and tab navigation, AutoIT does however have the ability to give you window control, identify window’s location, state (maximised, minimised, etc) and can activate a window for you
I’ve not use Zoomba, can’t comment on this one.
SikuliLibrary and ImageHorizonLibrary both work on image matching, so you will need many screen shots to locate items on the screen to interact with them, this is harder than using the windows libraries above, but these both work on MacOS and Linux and maybe other OS’s as well.
The idea of using Appium to automate desktop apps is an interesting one, I’ve never tried it, but looking at the Appium Drivers list there appears to be official drivers for Windows and MacOS and a community driver for Linux, I’ll be looking into this more, it might make my testing easier as I need to test on Windows, Mac and Linux for the same app.
The really great part about robot framework is you don’t have to choose this library or that library, you combine the libraries to cover any shortfall’s in functionality.
For case of not using Appium based libraries, keep in mind for cross-platform execution constraints like AutoIt doesn’t run on Linux, have to RDP/VNC over to remote SUT to test based off image recognition like Sikuli, etc.
You can make use of remote library interface and run the remote library (e.g. AutoIt, FlaUI) on the SUT, and run RF tests remotely elsewhere, calling the remote library to execute the desired keywords, etc. Going this way, you don’t necessarily need RDP/VNC other than to monitor the test run or to keep the remote desktop session (the GUI) open and unlocked for automated execution (since some GUI test tools fail to run when desktop is locked, etc.)
Appium abstracts that out with their own remote interfacing via Webdriver’s JSONWireProtocol, that works for any framework, and with the Appium libraries for RF, you can also make use of that instead of RF’s remote library interfacing.
Thanks for the hint. I’d prefer Appium for sure.
It’s not really part of the initial question but relevant w.r.t. suitability of librarie I guess: From a deployment point of view I’d like to run robot in a Jenkins CI/CD pipeline (in some way of creating a reproducible Python environment in a Jenkins pipeline “setup” stage using a robot project versioned in a BitBucket repo). This allows to define and maintain tests suits/test cases + reports/logs in a central place and in a controlled manner.
The desktop application needs to be run on a virtual windows machine or on a real windows machine (in my case a virtual machine would be ok at frist, later on I need a USB connection for accessing another SUT later on). The app under test resides on the windows machine. Have I understood correctly that to interface with the app UI it’s sufficient to have Appium + Appium Driver for Windows deployed in the robot project in the Jenkins runtime environment? Am I correct that I don’t need robotframework-xvfb in the virtual windows machine setup for creating a virtual display (in comparison to when using non-Webdriver libs from above)?
Your intent for how you plan to run RF in CI pipeline is fine. On the SUT deployment, depending on underlying library used, you may need to run a “server” component to receive the remote commands to execute from RF.
With Appium and any of the RF Appium-based libraries, you would need to run the Windows WebDriver component of the server (e.g. WinAppDriver, GitHub - FlaUI/FlaUI.WebDriver, etc.).
With other non-Appium libraries, you would need to run a remote server and run the desired library (AutoIt, Sikuli, FlaUI, etc.) as a remote library served through the remote (library) server. Because these tools won’t run against a remote SUT and some like AutoIt can’t run in a *nix environment.
How you maintain the server component on the SUT, whether is manually, or via jenkins CI jobs to start/stop/reboot the server software, etc. is up to you. CI job probably is jenkins talking to an Windows jenkins agent on the SUT.
RE: robotframework-xvfb, I believe you should not need that when using Appium option since that is meant for headless web browsers to execute on, and you’re doing desktop automation with Appium, not web (browser) automation. There is no “headless” option for desktop automation, at least not in the present moment, would be nice to have in future. Everything that executes is through the desktop GUI session, so you don’t need an XVFB redirect of desktop, which I believe doesn’t (natively) work for Windows anyhow, that’s for *nix OSes.
Just a minor note, Appium and non-Appium approach will work for RF solution, up to your preference. The biggest incentives for going Appium are the following, should they apply to you:
combine test codebase and test orchestration with web browser or mobile app based tests (e.g. you can create a single test that performs and validates some actions on the desktop side as well as mobile/web side, like do action A on desktop verify on mobile/web, do action B on mobile/web, verify result on desktop). This will be more seamless when done through Appium. RF provides a pretty consolidated way to handle this as well, especially if you pick the right library that abstracts away the interfacing to deskop, mobile, web tooling (otherwise, you have to wrap that up yourself). Appium just provides it out of the box based on how you can use it, whether used with RF or separately.
ability to go with other automation frameworks, so as not to be tied down to RF. Porting Appium related tests to another platform minimizes the amount of rework to do in terms of tool interface, even though you may have to rework the test language.
implement automation using industry standard tool and API/protocol (Appium and WebDriver API / JSONWireProtocol). In this sense, you probably have more people familiar with Appium and the WebDriver protocol than you are to have people familiar with RF, the various GUI automation libraries for RF, and the remote library interface for RF. So this is something to consider when you’re looking to expand the team or for long term maintenance of the test codebase.