Hi Jorge,
My Proxmox environment uses NoVNC so I inspected the html on that to have a look at what your up against.
Unfortunately on my side there’s no elements inside that canvas, the canvas just appears as an image and a hidden textarea, you interact by clicking the locations on the image and when you type, you’re typing into a textarea that then seems to send that keystroke info.
If your environment is the same, then you’ll have to choose between 3 options:
- run robot framework on the remote machine (inside the environment that NoVNC connects to)
- image based automation:
- ImageHorizonLibrary
- SikuliLibrary
As for tips for making image based detection more stable, use the smallest image that uniquely identifies what you want to interact with or reference, start with this post I made a while ago: ImageHorizonLibrary - #9 by damies13
I use ImageHorizonLibrary extensively for testing one of my applications on Windows + MacOS + Linux so have some experience in this topic, yes you will want to control and ensure resolution, scaling and colour depth don’t change, after that they can be fairly stable.
For text fields, use the labels, identify the co-ordinates of the label then click at an offset (usually below or to the right of the label), for buttons, use screenshots of the text portion of the button’s face, not the full button, so you avoid the width/height markers of the button in case the button changes size when the app is windowed vs maximised. if you do need to include the sides of the button, keep the number of button sides to 1 max 2 sides, then apply the label / button approach to the other element.
If you’re having trouble getting an image to recognise reliably share a screenshot of as much as you can the screenshot you’re using and the screen your trying to match it to and I’ll offer suggestions on what I would do.
Hope this helps,
Dave.