Which Dashboard to use for Robot Framework?

Extensions
1

Hello All,

some beginner questions :

1.)
I’ve done some simple automation tests with robot framework.

But now I’m interested in topics such as reporting, alarming and presentation of the test results.

During my research I came across the robotmk plugin, but this seems to support only checkmk.

Are there also good plugins / open source software available for connecting Icinga and / or Grafana dashboard ?

Regarding Icinga I haven’t found anything yet.

And regarding Grafana I found an interface / plugin, but the development has been idle / stopped for over 2 years.

So checkmk is the primary tool for representating the tests with robot framework?

2.)

We need to have tests mainly against a Citrix farm.

I got the information that the library “ImageHorizon” is best suited for this.
Is that true ?

Thanks and Regards,
Raphael

2

Hi Raphael,

I can’t help with all your questions but I might be able to help with some

  • checkmk - I’ve never heard of it, sorry I can help with that.
  • Icinga - I’ve never heard of it, sorry I can help with that.
  • Grafana - I am familiar with, this is just a web gui for displaying database data in a pretty format, usually a time series database like influxdb or Prometheus, getting data into those time series databases is usually done via a rest api (perhaps this is where checkmk comes in?) but i guess it’s up to you how you get the results into the database and what level of detail you want to send.
  • No updates to the plugin in 2 years may not be a bad thing, it depends if there are any outstanding issues. I have projects i’ve not updated for over a year because there is nothing that needs changing. if the plugin works use it.
  • ImageHorizon or SikuliLibrary are probably your best bet for automating the Citrix client at the moment, i don’t know of any robot framework libraries or even a python module for the Citrix client.
  • If you can find a python module for automating the Citrix client, it’s possible to turn that into a robot framework library with the same capabilities, i’ll even help you with that as it would be useful for me too.
  • Another option that might work for you, run a command to launch the citrix session, then run robot framework as an application within the citrix session to test the application itself.

Dave.

4

Hi Dave,

thanks for your answer.

Grafana - I am familiar with, this is just a web gui for displaying database data in a pretty format, usually a time series database like influxdb or Prometheus, getting data into those time series databases is usually done via a rest api (perhaps this is where checkmk comes in?) but i guess it’s up to you how you get the results into the database and what level of detail you want to send.

Do you use Robot Framework in combination with Grafana ?
And if “yes” - how do you get the RF results into the Grafana database ?

I tried to use this : Robot Framework Grafana Dashboard dashboard for Grafana | Grafana Labs

but I get SQL errors when I use the 2-years old library to convert the result file into the SQL database.

Thank you !
Raphael
And

5

No I don’t, I used it to monitor production application logs for statistical data to base our performance testing on , but that was at a previous employer 2+ years ago.

So as it’s been quite a while, I could only give you some general guidance.

I followed the link you provided to the dashboard and found this page describing the setup

The basic setup seems to be:

  • setup a mysql server
    • create a table robot_results
    • add a user grafanaReader
  • Configure a graphana datasource to connect to the mysql server with grafanaReader
  • install python modules:
    • mysqlclient
    • dbbot-sqlalchemy
  • run your robot test
  • after running your robot test, run dbbot passing the details of the mysql server, grafanaReader user and the output.xml from your robot test
  • then the dashboard should show your test results.

The widgets on that dashboard look like standard grafana graphs widgets, so I presume that dbbot is just parsing the output.xml and writing to the database table robot_results.

I would image if you followed the setup steps they provided exactly you would probably get it working provided the structure of the output.xml hasn’t changed too much.

By the same token, there is no need to do it the way they did if you don’t want to, you could send the data you want to the database you already have and replicate that dashboard with the standard graphana widgets.

There is no “Standard” way to setup the reporting of robot framework test results in graphana, as from what i’ve seen it’s not a commonly used solution. robot framework is flexible and can be tweaked to report however you want, likewise graphana is flexible and can graph data from a variety of data sources. so the combination of these two flexible solutions means there are probably dozens of ways you could configure it, it’s a matter of what works best for you/your organisation.

Did you already have a graphana server that is being used for reporting and you wanted to add robot framework results? or was this something you were setting up from scratch? I’m happy to try and help you get it working but I’ll need to know what you already have in place, to avoid setting up new servers unnecessarily.

Dave.

6

Hi Dave,

I got the attached error message when I executed the dbbot command.
I guess - as you wrote before - that the output.xml has changed too much.

I send email to the developers but no answer yet.

C:\raphael\code\python\results>python -m dbbot.run -b mysql://grafanaReader:PASSWORD@localhost/robot_results output.xml
Traceback (most recent call last):
File “C:\Python310\lib\site-packages\sqlalchemy\engine\base.py”, line 1819, in _execute_context
self.dialect.do_execute(
File “C:\Python310\lib\site-packages\sqlalchemy\engine\default.py”, line 732, in do_execute
cursor.execute(statement, parameters)
File “C:\Python310\lib\site-packages\MySQLdb\cursors.py”, line 206, in execute
res = self._query(query)
File “C:\Python310\lib\site-packages\MySQLdb\cursors.py”, line 319, in _query
db.query(q)
File “C:\Python310\lib\site-packages\MySQLdb\connections.py”, line 254, in query
_mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1071, ‘Specified key was too long; max key length is 3072 bytes’)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “C:\Python310\lib\runpy.py”, line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File “C:\Python310\lib\runpy.py”, line 86, in _run_code
exec(code, run_globals)
File “C:\Python310\lib\site-packages\dbbot\run.py”, line 53, in
DbBot().run()
File “C:\Python310\lib\site-packages\dbbot\run.py”, line 31, in init
self._db = DatabaseWriter(database_url, verbose_stream)
File “C:\Python310\lib\site-packages\dbbot\reader\database_writer.py”, line 28, in init
self._init_schema()
File “C:\Python310\lib\site-packages\dbbot\reader\database_writer.py”, line 45, in _init_schema
self._metadata.create_all(bind=self._engine)
File “C:\Python310\lib\site-packages\sqlalchemy\sql\schema.py”, line 4889, in create_all
bind._run_ddl_visitor(
File “C:\Python310\lib\site-packages\sqlalchemy\engine\base.py”, line 3147, in _run_ddl_visitor
conn._run_ddl_visitor(visitorcallable, element, **kwargs)
File “C:\Python310\lib\site-packages\sqlalchemy\engine\base.py”, line 2130, in _run_ddl_visitor
visitorcallable(self.dialect, self, **kwargs).traverse_single(element)
File “C:\Python310\lib\site-packages\sqlalchemy\sql\visitors.py”, line 524, in traverse_single
return meth(obj, **kw)
File “C:\Python310\lib\site-packages\sqlalchemy\sql\ddl.py”, line 851, in visit_metadata
self.traverse_single(
File “C:\Python310\lib\site-packages\sqlalchemy\sql\visitors.py”, line 524, in traverse_single
return meth(obj, **kw)
File “C:\Python310\lib\site-packages\sqlalchemy\sql\ddl.py”, line 895, in visit_table
self.connection.execute(
File “C:\Python310\lib\site-packages\sqlalchemy\engine\base.py”, line 1306, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File “C:\Python310\lib\site-packages\sqlalchemy\sql\ddl.py”, line 80, in _execute_on_connection
return connection._execute_ddl(
File “C:\Python310\lib\site-packages\sqlalchemy\engine\base.py”, line 1398, in _execute_ddl
ret = self._execute_context(
File “C:\Python310\lib\site-packages\sqlalchemy\engine\base.py”, line 1862, in _execute_context
self.handle_dbapi_exception(
File “C:\Python310\lib\site-packages\sqlalchemy\engine\base.py”, line 2043, in handle_dbapi_exception
util.raise(
File “C:\Python310\lib\site-packages\sqlalchemy\util\compat.py”, line 207, in raise
raise exception
File “C:\Python310\lib\site-packages\sqlalchemy\engine\base.py”, line 1819, in _execute_context
self.dialect.do_execute(
File “C:\Python310\lib\site-packages\sqlalchemy\engine\default.py”, line 732, in do_execute
cursor.execute(statement, parameters)
File “C:\Python310\lib\site-packages\MySQLdb\cursors.py”, line 206, in execute
res = self._query(query)
File “C:\Python310\lib\site-packages\MySQLdb\cursors.py”, line 319, in _query
db.query(q)
File “C:\Python310\lib\site-packages\MySQLdb\connections.py”, line 254, in query
_mysql.connection.query(self, query)
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1071, ‘Specified key was too long; max key length is 3072 bytes’)
[SQL:
CREATE TABLE suites (
id INTEGER NOT NULL AUTO_INCREMENT,
suite_id INTEGER,
xml_id VARCHAR(64) NOT NULL,
name VARCHAR(256) NOT NULL,
source VARCHAR(1024),
doc TEXT,
PRIMARY KEY (id),
CONSTRAINT unique_suites UNIQUE (name, source),
FOREIGN KEY(suite_id) REFERENCES suites (id)
)

]
(Background on this error at: Error Messages — SQLAlchemy 1.4 Documentation)

7

From that error I would suspect that mysql has changed between the version they used and the version your using.

Or possibly python or even the db module in python has changed somehow.

I suspect if you use a mysql client o connect to your mysql database and try running that create table query you would get the same (1071, ‘Specified key was too long; max key length is 3072 bytes’) error.

Dave.

8

Hello Dave, hello Damies13,

sorry, I haven’t bothered with this topic for the last few weeks,
but now I have to take care of it again.

Sorry if I repeat my question …

What options are available to get the results from the tests
into a GUI / monitoring dashboard ?

Unfortunately, I didn’t get an answer from the developer of this site :

I strongly assume that the Python library for importing the data into Grafana
is no longer compatible with current versions of Robot Framework.

Do you know of other projects / opensource software that is able to graphically display the results of Robot Framework? Or can convert or export these into tools like Grafana, Cacti / Icinga2 ?

Thanks and regards,
Raphael

9

Hi Raphael,

Based on your experience i’d say that project is probably not being supported anymore.

I would suggest using the standard Grafana and the Prometheus database, then just use the prometheus python module to write to the prometheus database and then just use the standard Grafana widgets to display your data how you want.

Sometimes it’s easier to build your own solution than use someone else’s old solution that might not work anymore. And between Grafana and Prometheus you should have everything you need to build a nice dashboard without having to muck around with database schema’s.

Dave.

1 Like
10

Hello.

You can take a look here:

It’s use Robot Framework and InfluxDB plugin for Jenkins to push all testing data to Grafana.

1 Like