Hi,
Is it possible to capture the network logs from chrome browser using robot framework?
What is the best practice?
Will appreciate your assistance .
You can record har files with Robot Framework Browser. See the New Context keyword.
1 Like
Hi Jani,
Thanks for the suggestion.
I looked into it and it does look very simple implementation . Is there another library / way to capture the network logs and maybe convert it to json for better msnipulstion to retrieve the data?
Thanks again .
HAR format is already JSON, so you can use whatever library/package to process that further. You could use Python’s built-in json package, or something like RPA.JSON, for example.
Hi Jani,
Thanks again,
Is it possible to capture specific domain or retrieve data from specific call using robot in har file using robot framework?
Moshe
Hi! Yes. As already mentioned, the HAR file format is JSON, and you can filter that for the information you need.
Here’s an example that gets all the called URLs:
tasks.robot:
*** Settings ***
Library Browser
Library RPA.JSON
*** Variables ***
${HAR_FILE_PATH}= ${OUTPUT_DIR}${/}har.har
*** Tasks ***
Record HAR file
Configure HAR
New Page https://developer.mozilla.org/
Parse HAR
*** Keywords ***
Configure HAR
${har_config}=
... Create Dictionary
... path=${HAR_FILE_PATH}
... omitContent=True
New Context recordHar=${har_config}
Parse HAR
${json}= Load Json From File ${HAR_FILE_PATH}
${urls}= Get Values From Json ${json} $..url
Save Json To File ${urls} ${OUTPUT_DIR}${/}urls.json
The recorded HAR file (har.har) - just a snippet since it is very long:
{
"log": {
"version": "1.2",
"creator": {
"name": "Playwright",
"version": "1.20.1"
},
"browser": {
"name": "chromium",
"version": "101.0.4929.0"
},
"pages": [
{
"startedDateTime": "2022-04-26T06:32:08.494Z",
"id": "page@6098b3b1b408cf628e5ba3ee76bd5c3a",
"title": "MDN Web Docs",
"pageTimings": {
"onContentLoad": 541,
"onLoad": 824
}
}
],
"entries": [
{
"_requestref": "request@da07673de4c688cf86773ce3482e6df2",
"_frameref": "frame@d899d6d52c23551ce4c08380fda5d06b",
"_monotonicTime": 1128865079.342,
"startedDateTime": "2022-04-26T06:32:08.500Z",
"time": 252.972,
"request": {
"method": "GET",
"url": "https://developer.mozilla.org/",
"httpVersion": "HTTP/2.0",
"cookies": [],
"headers": [
{
"name": ":authority",
"value": "developer.mozilla.org"
},
{
"name": ":method",
"value": "GET"
},
{
"name": ":path",
"value": "/"
},
{
"name": ":scheme",
"value": "https"
},
{
"name": "accept",
"value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
},
{
"name": "accept-encoding",
"value": "gzip, deflate, br"
},
{
"name": "sec-fetch-dest",
"value": "document"
},
{
"name": "sec-fetch-mode",
"value": "navigate"
},
{
"name": "sec-fetch-site",
"value": "none"
},
{
"name": "sec-fetch-user",
"value": "?1"
},
{
"name": "upgrade-insecure-requests",
"value": "1"
},
{
"name": "user-agent",
"value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/101.0.4929.0 Safari/537.36"
}
],
"queryString": [],
"headersSize": 538,
"bodySize": 0
},
"response": {
"status": 302,
"statusText": "",
"httpVersion": "HTTP/2.0",
"cookies": [],
"headers": [
{
"name": "cache-control",
"value": "no-store"
},
{
"name": "content-length",
"value": "0"
},
{
"name": "date",
"value": "Tue, 26 Apr 2022 06:32:08 GMT"
},
{
"name": "location",
"value": "/en-US/"
},
{
"name": "server",
"value": "CloudFront"
},
{
"name": "via",
"value": "1.1 4aba10f1ef4bda5f7c1b86a07ca64264.cloudfront.net (CloudFront)"
},
{
"name": "x-amz-cf-id",
"value": "v0Jniif8i0uAlBhkapx1lIUxbwVlJmkR_chYmG9s4a0QE8mUCEJl2w=="
},
{
"name": "x-amz-cf-pop",
"value": "HEL51-P1"
},
{
"name": "x-cache",
"value": "Miss from cloudfront"
}
],
"content": {
"size": -1,
"mimeType": "x-unknown",
"compression": 0
},
"headersSize": 334,
"bodySize": 0,
"redirectURL": "https://developer.mozilla.org/en-US/",
"_transferSize": 334
},
"cache": {
"beforeRequest": null,
"afterRequest": null
},
"timings": {
"dns": 90.418,
"connect": 58.401,
"ssl": 40.94,
"send": 0,
"wait": 61.566,
"receive": 1.647
},
"pageref": "page@6098b3b1b408cf628e5ba3ee76bd5c3a",
"serverIPAddress": "108.156.22.40",
"_serverPort": 443,
"_securityDetails": {
"protocol": "TLS 1.3",
"subjectName": "developer.mozilla.org",
"issuer": "Amazon",
"validFrom": 1633392000,
"validTo": 1667433599
}
},
{
"_requestref": "request@5e9297339b23574d50a26cfcefc72b35",
"_frameref": "frame@d899d6d52c23551ce4c08380fda5d06b",
"_monotonicTime": 1128865293.382,
"startedDateTime": "2022-04-26T06:32:08.714Z",
"time": 25.923,
"request": {
"method": "GET",
"url": "https://developer.mozilla.org/en-US/",
"httpVersion": "HTTP/2.0",
"cookies": [],
"headers": [
{
"name": ":authority",
"value": "developer.mozilla.org"
},
{
"name": ":method",
"value": "GET"
},
{
"name": ":path",
"value": "/en-US/"
},
{
"name": ":scheme",
"value": "https"
},
{
"name": "accept",
"value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
},
{
"name": "accept-encoding",
"value": "gzip, deflate, br"
},
{
"name": "sec-fetch-dest",
"value": "document"
},
{
"name": "sec-fetch-mode",
"value": "navigate"
},
{
"name": "sec-fetch-site",
"value": "none"
},
{
"name": "sec-fetch-user",
"value": "?1"
},
{
"name": "upgrade-insecure-requests",
"value": "1"
},
{
"name": "user-agent",
"value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/101.0.4929.0 Safari/537.36"
}
],
"queryString": [],
"headersSize": 550,
"bodySize": 0
},
"response": {
"status": 200,
"statusText": "",
"httpVersion": "HTTP/2.0",
"cookies": [],
"headers": [
{
"name": "age",
"value": "19240"
},
{
"name": "cache-control",
"value": "max-age=86400, public"
},
{
"name": "content-encoding",
"value": "gzip"
},
{
"name": "content-security-policy",
"value": "default-src 'self'; script-src 'report-sample' 'self' *.speedcurve.com 'sha256-q7cJjDqNO2e1L5UltvJ1LhvnYN7yJXgGO7b6h9xkL1o=' www.google-analytics.com/analytics.js 'sha256-JEt9Nmc3BP88wxuTZm9aKNu87vEgGmKW1zzy/vb1KPs=' polyfill.io/v3/polyfill.min.js assets.codepen.io production-assets.codepen.io 'sha256-x6Tv+AdV5e6dcolO0TEo+3BG4H2nG2ACjyG8mz6QCes='; script-src-elem 'report-sample' 'self' *.speedcurve.com 'sha256-q7cJjDqNO2e1L5UltvJ1LhvnYN7yJXgGO7b6h9xkL1o=' www.google-analytics.com/analytics.js 'sha256-JEt9Nmc3BP88wxuTZm9aKNu87vEgGmKW1zzy/vb1KPs=' polyfill.io/v3/polyfill.min.js assets.codepen.io production-assets.codepen.io 'sha256-x6Tv+AdV5e6dcolO0TEo+3BG4H2nG2ACjyG8mz6QCes='; style-src 'report-sample' 'self' 'unsafe-inline'; object-src 'none'; base-uri 'self'; connect-src 'self' updates.developer.allizom.org updates.developer.mozilla.org www.google-analytics.com stats.g.doubleclick.net; font-src 'self'; frame-src 'self' interactive-examples.mdn.mozilla.net interactive-examples.prod.mdn.mozilla.net interactive-examples.stage.mdn.mozilla.net mdn.github.io yari-demos.prod.mdn.mozit.cloud mdn.mozillademos.org yari-demos.stage.mdn.mozit.cloud jsfiddle.net www.youtube-nocookie.com codepen.io; img-src 'self' *.githubusercontent.com *.googleusercontent.com *.gravatar.com mozillausercontent.com firefoxusercontent.com profile.stage.mozaws.net profile.accounts.firefox.com lux.speedcurve.com mdn.mozillademos.org media.prod.mdn.mozit.cloud media.stage.mdn.mozit.cloud interactive-examples.mdn.mozilla.net interactive-examples.prod.mdn.mozilla.net interactive-examples.stage.mdn.mozilla.net wikipedia.org www.google-analytics.com www.gstatic.com; manifest-src 'self'; media-src 'self' archive.org videos.cdn.mozilla.net; child-src 'self'; worker-src 'self';"
},
{
"name": "content-type",
"value": "text/html; charset=utf-8"
},
{
"name": "date",
"value": "Tue, 26 Apr 2022 01:11:28 GMT"
},
{
"name": "etag",
"value": "W/\"42ba7227a513c5710d41655ea48ee383\""
},
{
"name": "last-modified",
"value": "Tue, 26 Apr 2022 01:07:59 GMT"
},
{
"name": "server",
"value": "AmazonS3"
},
{
"name": "strict-transport-security",
"value": "max-age=63072000"
},
{
"name": "vary",
"value": "Accept-Encoding"
},
{
"name": "via",
"value": "1.1 4aba10f1ef4bda5f7c1b86a07ca64264.cloudfront.net (CloudFront)"
},
{
"name": "x-amz-cf-id",
"value": "qcSXQjRuX7iZySAzlBseXVLXjRUOy9oH2XPybDnNx2o4cD6vO1ODXg=="
},
{
"name": "x-amz-cf-pop",
"value": "HEL51-P1"
},
{
"name": "x-cache",
"value": "Hit from cloudfront"
},
{
"name": "x-content-type-options",
"value": "nosniff"
},
{
"name": "x-frame-options",
"value": "DENY"
},
{
"name": "x-xss-protection",
"value": "1; mode=block"
}
],
"content": {
"size": 79221,
"mimeType": "text/html; charset=utf-8",
"compression": 62570
},
"headersSize": 2423,
"bodySize": 16651,
"redirectURL": "",
"_transferSize": 16651
},
"cache": {
"beforeRequest": null,
"afterRequest": null
},
"timings": {
"dns": -1,
"connect": -1,
"ssl": -1,
"send": 0,
"wait": 25.923,
"receive": -1
},
"pageref": "page@6098b3b1b408cf628e5ba3ee76bd5c3a",
"serverIPAddress": "108.156.22.40",
"_serverPort": 443,
"_securityDetails": {
"protocol": "TLS 1.3",
"subjectName": "developer.mozilla.org",
"issuer": "Amazon",
"validFrom": 1633392000,
"validTo": 1667433599
}
},
...
The filtered urls.json:
[
"https://developer.mozilla.org/",
"https://developer.mozilla.org/en-US/",
"https://developer.mozilla.org/static/media/ZillaSlab-Bold.subset.0beac26b.woff2",
"https://developer.mozilla.org/static/css/main.55b68e13.chunk.css",
"https://cdn.speedcurve.com/js/lux.js?id=108906238",
"https://developer.mozilla.org/static/js/ga.js",
"https://developer.mozilla.org/static/js/runtime-main.e858fdfd.js",
"https://developer.mozilla.org/static/js/4.a756dea3.chunk.js",
"https://developer.mozilla.org/static/js/main.e0441009.chunk.js",
"https://developer.mozilla.org/static/media/triangle.e8097649.svg",
"https://developer.mozilla.org/static/media/search.43d0e3bf.svg",
"https://developer.mozilla.org/static/media/home-masthead-background.4306d252.svg",
"https://developer.mozilla.org/static/media/arrow.e1de8ede.svg",
"https://developer.mozilla.org/static/media/twitter.cc5b37fe.svg",
"https://developer.mozilla.org/static/media/github-mark-small.348586b8.svg",
"https://developer.mozilla.org/static/media/instagram.445373fc.svg",
"https://developer.mozilla.org/static/media/ZillaSlab-Regular.subset.ce3a756d.woff2",
"https://www.google-analytics.com/analytics.js",
"https://www.google-analytics.com/j/collect?v=1&_v=j96&aip=1&a=1903151242&t=pageview&_s=1&dl=https%3A%2F%2Fdeveloper.mozilla.org%2Fen-US%2F&ul=en-us&de=UTF-8&dt=MDN%20Web%20Docs&sd=30-bit&sr=1280x720&vp=1280x720&je=0&_u=YEBAAAABAAAAAC~&jid=342012907&gjid=421563640&cid=549396694.1643871150&tid=UA-36116321-5&_gid=346451150.1643871150&_r=1&_slc=1&z=1690852481",
"https://stats.g.doubleclick.net/j/collect?t=dc&aip=1&_r=3&v=1&_v=j96&tid=UA-36116321-5&cid=549396694.1643871150&jid=342012907&gjid=421563640&_gid=346451150.1643871150&_u=YEBAAAAAAAAAAC~&z=553170267"
]
3 Likes
Thanks Jani for the elaborate explanation.
appreciate your assistance.
Hi @janipalsamaki,
I’ve tried your code, there is one point I don’t understand why the result always fail if there is no har.har file or in the other word it fail at the first time.
However, har.har file is already created.
Poppy
@oearhp
Were you able to solve this issue? I am running into the same problem…
Thanks
You Need to Close the Context before Parsing.
“Make sure to await context to close for the file to be saved.”
1 Like
Is there a way to get the recorded har during runtime?
Har is recorded during run time, but if your question that is it possible to start har recording dynamically after opening new context, that is not supported by Browser library. But it is supported by Playwright: Page | Playwright If you would like to have such feature, please raise an issue about it and even better if you can create PR for the implementation.