Automate non-windows desktop applications

Robot Framework
1

I need to automate with RF some desktop apps/tools I can’t do it

2

Someone could help me?

3

I use SikuliLibrary, for any operating system. But there are other image base libraries.

1 Like
4

I have used WhiteLibrary with little to no problems, but nothing heavy in terms of the requirement.

Worth noting (would apply to all I believe) to inspect the windows:

5

to inspect the windows i have the accesibility insights for windows

1 Like
7

I’m not able to download the SikuliX IDE, could you send me the .exe?

8

It is .jar file. Search better.

9

Yes, i have it but when i execute it, The sikulix program starts but i can not see any window

10

My experience with Desktop Automation (on Windows) is:

Use Image Based Libraries (like Sikuli or ImageHorizon Library) only as a last resort if nothing else works.

Before that, figure out what technology your application is using.

Is it maybe Java or Java Swing?
Try out the
GitHub - robotframework/remoteswinglibrary: RemoteSwingLibrary or
JavaAccessBridge — RPA Framework documentation

Is it a Win32, Winform, WPF Application?
Try out
GitHub - GDATASoftwareAG/robotframework-flaui: Windows user interface automation library for Robot-Framework. FlaUILibrary is a wrapper for the FlaUI automation library. or
Windows — RPA Framework documentation

If nothing else works, try out
GitHub - eficode/robotframework-imagehorizonlibrary: Cross-platform Robot Framework library for GUI automation based on image recognition or
GitHub - rainmanwy/robotframework-SikuliLibrary: Sikuli Robot Framework Library provide keywords for Robot Framework to test UI through Sikuli.

11

@_daryl

Just an FYI - I believe flaui Library replaces While Library as the creator of flaui Library and While Library is the same person, he mentioned he stopped maintenance of While when he started flaui (there was a post in the old google groups about it)

Dave.

1 Like
12

Hi @damies13, Thank you
I had realised it was no longer maintained, but hadn’t noticed that flaui was it’s replacement… makes sense why I never see much talk on white library :joy: I will definitely take a look.
Much appreciated.

1 Like
13

Hi Carlos,

When using sikuli Library, you won’t get the sikulix gui at all. Sikuli Library runs sikulix in the background and calls it’s API’s to tell it what it should do simulating a sikulix script.

So for a really simple example (assuming your using windows?), take a screen shot of you start button as "start_button.png"and your Programs menu item as “programs_menu.png” and put them in the same folder as this robot file, then you should be able to use this example:

*** Settings ***
Library    SikuliLibrary

*** Variables ***

*** Test Cases ***
Simple Sikulix Test
    Add Image Path    ${CURDIR}
    Click    start_button.png
    Wait For Image    programs_menu.png
    Click    programs_menu.png

From there you should be able to get it working.

Another alternative to Sikuli Library is ImageHorizon Library, but unfortunately there is a bug in the current version of ImageHorizon Library that means it doesn’t handle screen scaling well, I submitted a fix for that a while back but there hasn’t been a new version of ImageHorizon Library yet, so you’ll probably have an easier start with Sikuli Library, unless you are sure all the.machines your are testing on will have screen scaling turned off.

Dave.

1 Like
14

Thanks damies13, yesterday I was able to make a script that it works correctly. Now i have another little problem because when I use two monitors, the script with SikuliLibrary only works in the main one but if if the application opens on a monitor other than the main one, I have an error because robotframework does not find the image to click it.

15

Hi Carlos,

Yeah that’s one of the limitations with SikuliLibrary, If I remember correctly ImageHorizon treats all the monitors as one great big monitor, so if the secondary monitor is to left or above the primary you get negative co-ordinates (which also can be confusing).

Every Library has it’s pro’s and cons.

Are you using windows or another OS? If your using Windows you might be able to use the Autoit library to bring the window to the primary monitor so that SikuliLibrary can see it.

If you’re using another OS, it’ll be helpful to know which one, so I can make OS appropriate suggestions.

Dave.

16

Hi Dave,

I use Windows as OS so i can try to test the Autoit Library only to bring the app to the primary monitor. I think that it would be easy to use it, right?

Thanks for all the information.

Carlos

17

SikuliLibrary allows to select which monitor is in focus. You should control on which monitor the app is working, when tests run.

18

Hi,

Which is the Keyword for that? Because in the documentation i have not found anything to work between monitors or windows

Thanks!!

19

Hi Carlos,

@HelioGuilherme66’s comment made me go look, it looks like there is a some useful keywords there.

Take a screenshot of the first screen in your app and call it something like “main.jpg”, then you could do this in a keyword called something like Switch To Screen With AppName:

As this is using SikuliLibrary keywords it’s probably a better solution than my AutoITLibrary suggestion as you don’t need to introduce other Libraries.

Dave.

2 Likes
20

Does Sikuli work with python 3.11?

21

Hi @vdavis345,

I would expect so as the python module just starts Sikuli in a java runtime and then makes the java calls, so most of the code for this library is actually java.

I know it worked in python 3.10 last time I used it.

Dave.