Hi,
Is it possible to capture the network logs from chrome browser using robot framework?
What is the best practice?
Will appreciate your assistance .
Hi Jani,
Thanks for the suggestion.
I looked into it and it does look very simple implementation . Is there another library / way to capture the network logs and maybe convert it to json for better msnipulstion to retrieve the data?
Thanks again .
HAR format is already JSON, so you can use whatever library/package to process that further. You could use Python’s built-in json package, or something like RPA.JSON, for example.
Hi Jani,
Thanks again,
Is it possible to capture specific domain or retrieve data from specific call using robot in har file using robot framework?
Moshe
Hi! Yes. As already mentioned, the HAR file format is JSON, and you can filter that for the information you need.
Here’s an example that gets all the called URLs:
tasks.robot:
*** Settings ***
Library Browser
Library RPA.JSON
*** Variables ***
${HAR_FILE_PATH}= ${OUTPUT_DIR}${/}har.har
*** Tasks ***
Record HAR file
Configure HAR
New Page https://developer.mozilla.org/
Parse HAR
*** Keywords ***
Configure HAR
${har_config}=
... Create Dictionary
... path=${HAR_FILE_PATH}
... omitContent=True
New Context recordHar=${har_config}
Parse HAR
${json}= Load Json From File ${HAR_FILE_PATH}
${urls}= Get Values From Json ${json} $..url
Save Json To File ${urls} ${OUTPUT_DIR}${/}urls.json
The recorded HAR file (har.har) - just a snippet since it is very long:
{
"log": {
"version": "1.2",
"creator": {
"name": "Playwright",
"version": "1.20.1"
},
"browser": {
"name": "chromium",
"version": "101.0.4929.0"
},
"pages": [
{
"startedDateTime": "2022-04-26T06:32:08.494Z",
"id": "page@6098b3b1b408cf628e5ba3ee76bd5c3a",
"title": "MDN Web Docs",
"pageTimings": {
"onContentLoad": 541,
"onLoad": 824
}
}
],
"entries": [
{
"_requestref": "request@da07673de4c688cf86773ce3482e6df2",
"_frameref": "frame@d899d6d52c23551ce4c08380fda5d06b",
"_monotonicTime": 1128865079.342,
"startedDateTime": "2022-04-26T06:32:08.500Z",
"time": 252.972,
"request": {
"method": "GET",
"url": "https://developer.mozilla.org/",
"httpVersion": "HTTP/2.0",
"cookies": [],
"headers": [
{
"name": ":authority",
"value": "developer.mozilla.org"
},
{
"name": ":method",
"value": "GET"
},
{
"name": ":path",
"value": "/"
},
{
"name": ":scheme",
"value": "https"
},
{
"name": "accept",
"value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
},
{
"name": "accept-encoding",
"value": "gzip, deflate, br"
},
{
"name": "sec-fetch-dest",
"value": "document"
},
{
"name": "sec-fetch-mode",
"value": "navigate"
},
{
"name": "sec-fetch-site",
"value": "none"
},
{
"name": "sec-fetch-user",
"value": "?1"
},
{
"name": "upgrade-insecure-requests",
"value": "1"
},
{
"name": "user-agent",
"value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/101.0.4929.0 Safari/537.36"
}
],
"queryString": [],
"headersSize": 538,
"bodySize": 0
},
"response": {
"status": 302,
"statusText": "",
"httpVersion": "HTTP/2.0",
"cookies": [],
"headers": [
{
"name": "cache-control",
"value": "no-store"
},
{
"name": "content-length",
"value": "0"
},
{
"name": "date",
"value": "Tue, 26 Apr 2022 06:32:08 GMT"
},
{
"name": "location",
"value": "/en-US/"
},
{
"name": "server",
"value": "CloudFront"
},
{
"name": "via",
"value": "1.1 4aba10f1ef4bda5f7c1b86a07ca64264.cloudfront.net (CloudFront)"
},
{
"name": "x-amz-cf-id",
"value": "v0Jniif8i0uAlBhkapx1lIUxbwVlJmkR_chYmG9s4a0QE8mUCEJl2w=="
},
{
"name": "x-amz-cf-pop",
"value": "HEL51-P1"
},
{
"name": "x-cache",
"value": "Miss from cloudfront"
}
],
"content": {
"size": -1,
"mimeType": "x-unknown",
"compression": 0
},
"headersSize": 334,
"bodySize": 0,
"redirectURL": "https://developer.mozilla.org/en-US/",
"_transferSize": 334
},
"cache": {
"beforeRequest": null,
"afterRequest": null
},
"timings": {
"dns": 90.418,
"connect": 58.401,
"ssl": 40.94,
"send": 0,
"wait": 61.566,
"receive": 1.647
},
"pageref": "page@6098b3b1b408cf628e5ba3ee76bd5c3a",
"serverIPAddress": "108.156.22.40",
"_serverPort": 443,
"_securityDetails": {
"protocol": "TLS 1.3",
"subjectName": "developer.mozilla.org",
"issuer": "Amazon",
"validFrom": 1633392000,
"validTo": 1667433599
}
},
{
"_requestref": "request@5e9297339b23574d50a26cfcefc72b35",
"_frameref": "frame@d899d6d52c23551ce4c08380fda5d06b",
"_monotonicTime": 1128865293.382,
"startedDateTime": "2022-04-26T06:32:08.714Z",
"time": 25.923,
"request": {
"method": "GET",
"url": "https://developer.mozilla.org/en-US/",
"httpVersion": "HTTP/2.0",
"cookies": [],
"headers": [
{
"name": ":authority",
"value": "developer.mozilla.org"
},
{
"name": ":method",
"value": "GET"
},
{
"name": ":path",
"value": "/en-US/"
},
{
"name": ":scheme",
"value": "https"
},
{
"name": "accept",
"value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
},
{
"name": "accept-encoding",
"value": "gzip, deflate, br"
},
{
"name": "sec-fetch-dest",
"value": "document"
},
{
"name": "sec-fetch-mode",
"value": "navigate"
},
{
"name": "sec-fetch-site",
"value": "none"
},
{
"name": "sec-fetch-user",
"value": "?1"
},
{
"name": "upgrade-insecure-requests",
"value": "1"
},
{
"name": "user-agent",
"value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/101.0.4929.0 Safari/537.36"
}
],
"queryString": [],
"headersSize": 550,
"bodySize": 0
},
"response": {
"status": 200,
"statusText": "",
"httpVersion": "HTTP/2.0",
"cookies": [],
"headers": [
{
"name": "age",
"value": "19240"
},
{
"name": "cache-control",
"value": "max-age=86400, public"
},
{
"name": "content-encoding",
"value": "gzip"
},
{
"name": "content-security-policy",
"value": "default-src 'self'; script-src 'report-sample' 'self' *.speedcurve.com 'sha256-q7cJjDqNO2e1L5UltvJ1LhvnYN7yJXgGO7b6h9xkL1o=' www.google-analytics.com/analytics.js 'sha256-JEt9Nmc3BP88wxuTZm9aKNu87vEgGmKW1zzy/vb1KPs=' polyfill.io/v3/polyfill.min.js assets.codepen.io production-assets.codepen.io 'sha256-x6Tv+AdV5e6dcolO0TEo+3BG4H2nG2ACjyG8mz6QCes='; script-src-elem 'report-sample' 'self' *.speedcurve.com 'sha256-q7cJjDqNO2e1L5UltvJ1LhvnYN7yJXgGO7b6h9xkL1o=' www.google-analytics.com/analytics.js 'sha256-JEt9Nmc3BP88wxuTZm9aKNu87vEgGmKW1zzy/vb1KPs=' polyfill.io/v3/polyfill.min.js assets.codepen.io production-assets.codepen.io 'sha256-x6Tv+AdV5e6dcolO0TEo+3BG4H2nG2ACjyG8mz6QCes='; style-src 'report-sample' 'self' 'unsafe-inline'; object-src 'none'; base-uri 'self'; connect-src 'self' updates.developer.allizom.org updates.developer.mozilla.org www.google-analytics.com stats.g.doubleclick.net; font-src 'self'; frame-src 'self' interactive-examples.mdn.mozilla.net interactive-examples.prod.mdn.mozilla.net interactive-examples.stage.mdn.mozilla.net mdn.github.io yari-demos.prod.mdn.mozit.cloud mdn.mozillademos.org yari-demos.stage.mdn.mozit.cloud jsfiddle.net www.youtube-nocookie.com codepen.io; img-src 'self' *.githubusercontent.com *.googleusercontent.com *.gravatar.com mozillausercontent.com firefoxusercontent.com profile.stage.mozaws.net profile.accounts.firefox.com lux.speedcurve.com mdn.mozillademos.org media.prod.mdn.mozit.cloud media.stage.mdn.mozit.cloud interactive-examples.mdn.mozilla.net interactive-examples.prod.mdn.mozilla.net interactive-examples.stage.mdn.mozilla.net wikipedia.org www.google-analytics.com www.gstatic.com; manifest-src 'self'; media-src 'self' archive.org videos.cdn.mozilla.net; child-src 'self'; worker-src 'self';"
},
{
"name": "content-type",
"value": "text/html; charset=utf-8"
},
{
"name": "date",
"value": "Tue, 26 Apr 2022 01:11:28 GMT"
},
{
"name": "etag",
"value": "W/\"42ba7227a513c5710d41655ea48ee383\""
},
{
"name": "last-modified",
"value": "Tue, 26 Apr 2022 01:07:59 GMT"
},
{
"name": "server",
"value": "AmazonS3"
},
{
"name": "strict-transport-security",
"value": "max-age=63072000"
},
{
"name": "vary",
"value": "Accept-Encoding"
},
{
"name": "via",
"value": "1.1 4aba10f1ef4bda5f7c1b86a07ca64264.cloudfront.net (CloudFront)"
},
{
"name": "x-amz-cf-id",
"value": "qcSXQjRuX7iZySAzlBseXVLXjRUOy9oH2XPybDnNx2o4cD6vO1ODXg=="
},
{
"name": "x-amz-cf-pop",
"value": "HEL51-P1"
},
{
"name": "x-cache",
"value": "Hit from cloudfront"
},
{
"name": "x-content-type-options",
"value": "nosniff"
},
{
"name": "x-frame-options",
"value": "DENY"
},
{
"name": "x-xss-protection",
"value": "1; mode=block"
}
],
"content": {
"size": 79221,
"mimeType": "text/html; charset=utf-8",
"compression": 62570
},
"headersSize": 2423,
"bodySize": 16651,
"redirectURL": "",
"_transferSize": 16651
},
"cache": {
"beforeRequest": null,
"afterRequest": null
},
"timings": {
"dns": -1,
"connect": -1,
"ssl": -1,
"send": 0,
"wait": 25.923,
"receive": -1
},
"pageref": "page@6098b3b1b408cf628e5ba3ee76bd5c3a",
"serverIPAddress": "108.156.22.40",
"_serverPort": 443,
"_securityDetails": {
"protocol": "TLS 1.3",
"subjectName": "developer.mozilla.org",
"issuer": "Amazon",
"validFrom": 1633392000,
"validTo": 1667433599
}
},
...
The filtered urls.json:
[
"https://developer.mozilla.org/",
"https://developer.mozilla.org/en-US/",
"https://developer.mozilla.org/static/media/ZillaSlab-Bold.subset.0beac26b.woff2",
"https://developer.mozilla.org/static/css/main.55b68e13.chunk.css",
"https://cdn.speedcurve.com/js/lux.js?id=108906238",
"https://developer.mozilla.org/static/js/ga.js",
"https://developer.mozilla.org/static/js/runtime-main.e858fdfd.js",
"https://developer.mozilla.org/static/js/4.a756dea3.chunk.js",
"https://developer.mozilla.org/static/js/main.e0441009.chunk.js",
"https://developer.mozilla.org/static/media/triangle.e8097649.svg",
"https://developer.mozilla.org/static/media/search.43d0e3bf.svg",
"https://developer.mozilla.org/static/media/home-masthead-background.4306d252.svg",
"https://developer.mozilla.org/static/media/arrow.e1de8ede.svg",
"https://developer.mozilla.org/static/media/twitter.cc5b37fe.svg",
"https://developer.mozilla.org/static/media/github-mark-small.348586b8.svg",
"https://developer.mozilla.org/static/media/instagram.445373fc.svg",
"https://developer.mozilla.org/static/media/ZillaSlab-Regular.subset.ce3a756d.woff2",
"https://www.google-analytics.com/analytics.js",
"https://www.google-analytics.com/j/collect?v=1&_v=j96&aip=1&a=1903151242&t=pageview&_s=1&dl=https%3A%2F%2Fdeveloper.mozilla.org%2Fen-US%2F&ul=en-us&de=UTF-8&dt=MDN%20Web%20Docs&sd=30-bit&sr=1280x720&vp=1280x720&je=0&_u=YEBAAAABAAAAAC~&jid=342012907&gjid=421563640&cid=549396694.1643871150&tid=UA-36116321-5&_gid=346451150.1643871150&_r=1&_slc=1&z=1690852481",
"https://stats.g.doubleclick.net/j/collect?t=dc&aip=1&_r=3&v=1&_v=j96&tid=UA-36116321-5&cid=549396694.1643871150&jid=342012907&gjid=421563640&_gid=346451150.1643871150&_u=YEBAAAAAAAAAAC~&z=553170267"
]
3 Likes
Thanks Jani for the elaborate explanation.
appreciate your assistance.
Hi @janipalsamaki,
I’ve tried your code, there is one point I don’t understand why the result always fail if there is no har.har file or in the other word it fail at the first time.
However, har.har file is already created.
Poppy