Library for automated visual regression testing

Libraries Browser
1

Hi there,

Is there a specific library for automates visual regression testing with robotframework of your responsive web UI by comparing DOM screenshots over time?

I want to use chromium as a browser and that is the baseline and want to compare (buttons, fields, etc) it with the web view of webkit, Safari or FireFox.

Thanks for the respons.

2

First part is easy, Browser like most Robot Framework libraries has a Take Screenshot keyword

I’m not aware of anything that will provide an image compare out of the box, but you can do that in python, so you could create a python keyword that compares two screenshots (images), I found this example, how-to-compare-two-images that uses the python numpy module, they had 2 different sized images, but if you set the size of your browser window to always be the same then you can skip the reshape step and just compare the images, i’m not sure what the number returned is, probably the number of different pixels, so if you divide by the total pixel count of one image that would potentially give you a percentage different store this in a variable ${pct_diff} then to pass /fail your test use the builtin keywords:

${result} =	Evaluate 	${pct_diff} < 10 	#set you tolerance level here
Should Be True	${result}

Hope that helps,

Dave.

3

There’s a library called pixelmatch that uses a cut down version of the full Python Image Library (PIL) and I found that to be very effective for performing image comparisons, returns both a numerical value and a difference image
pixelmatch · PyPI

1 Like
4

Not the most elegant solution, but you’re welcome to use it if it helps - if you can improve it, please let me know :smiley:

img.py file
from robot.api.deco import keyword

from PIL import Image

from pixelmatch.contrib.PIL import pixelmatch

@keyword

def compare_images(img1_str: str, img2_str: str, diff_str: str)-> int:

ERR_DIR = "B:/Documents/Errors/"

SAM_DIR = "B:/Documents/Screenshots/"

REF_DIR = "B:/Documents/Ref_Images/"

img1 = Image.open(REF_DIR + img1_str)

img2 = Image.open(SAM_DIR + img2_str)

diff = Image.new("RGBA", img1.size)

mismatch = pixelmatch(img1, img2, diff, threshold= 0.1, includeAA=True, diff_mask=True)

diff.save(ERR_DIR + diff_str)

return mismatch

In robot file
Library img.py
…
${diff} = Compare Images ${file2} ${file} ${file}
${diff} is a numerical value for the number of pixels that don’t match
${file2} is the reference image
${file} is the screenshot (uses the same name to create the diff image as well)

Hope that helps.

5

Hi @JoostW you probably solved this issue.

However for those who are searching for some solution I could recommend 2 options we built: