Capture network logs from chrome browser

Hi,

Is it possible to capture the network logs from chrome browser using robot framework?
What is the best practice?

Will appreciate your assistance .

You can record har files with Robot Framework Browser. See the New Context keyword.

1 Like

Hi Jani,

Thanks for the suggestion.

I looked into it and it does look very simple implementation . Is there another library / way to capture the network logs and maybe convert it to json for better msnipulstion to retrieve the data?

Thanks again .

HAR format is already JSON, so you can use whatever library/package to process that further. You could use Python’s built-in json package, or something like RPA.JSON, for example.

Hi Jani,
Thanks again,
Is it possible to capture specific domain or retrieve data from specific call using robot in har file using robot framework?

Moshe

Hi! Yes. As already mentioned, the HAR file format is JSON, and you can filter that for the information you need.

Here’s an example that gets all the called URLs:

tasks.robot:

*** Settings ***
Library           Browser
Library           RPA.JSON

*** Variables ***
${HAR_FILE_PATH}=    ${OUTPUT_DIR}${/}har.har

*** Tasks ***
Record HAR file
    Configure HAR
    New Page    https://developer.mozilla.org/
    Parse HAR

*** Keywords ***
Configure HAR
    ${har_config}=
    ...    Create Dictionary
    ...    path=${HAR_FILE_PATH}
    ...    omitContent=True
    New Context    recordHar=${har_config}

Parse HAR
    ${json}=    Load Json From File    ${HAR_FILE_PATH}
    ${urls}=    Get Values From Json    ${json}    $..url
    Save Json To File    ${urls}    ${OUTPUT_DIR}${/}urls.json

The recorded HAR file (har.har) - just a snippet since it is very long:

{
  "log": {
    "version": "1.2",
    "creator": {
      "name": "Playwright",
      "version": "1.20.1"
    },
    "browser": {
      "name": "chromium",
      "version": "101.0.4929.0"
    },
    "pages": [
      {
        "startedDateTime": "2022-04-26T06:32:08.494Z",
        "id": "page@6098b3b1b408cf628e5ba3ee76bd5c3a",
        "title": "MDN Web Docs",
        "pageTimings": {
          "onContentLoad": 541,
          "onLoad": 824
        }
      }
    ],
    "entries": [
      {
        "_requestref": "request@da07673de4c688cf86773ce3482e6df2",
        "_frameref": "frame@d899d6d52c23551ce4c08380fda5d06b",
        "_monotonicTime": 1128865079.342,
        "startedDateTime": "2022-04-26T06:32:08.500Z",
        "time": 252.972,
        "request": {
          "method": "GET",
          "url": "https://developer.mozilla.org/",
          "httpVersion": "HTTP/2.0",
          "cookies": [],
          "headers": [
            {
              "name": ":authority",
              "value": "developer.mozilla.org"
            },
            {
              "name": ":method",
              "value": "GET"
            },
            {
              "name": ":path",
              "value": "/"
            },
            {
              "name": ":scheme",
              "value": "https"
            },
            {
              "name": "accept",
              "value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
            },
            {
              "name": "accept-encoding",
              "value": "gzip, deflate, br"
            },
            {
              "name": "sec-fetch-dest",
              "value": "document"
            },
            {
              "name": "sec-fetch-mode",
              "value": "navigate"
            },
            {
              "name": "sec-fetch-site",
              "value": "none"
            },
            {
              "name": "sec-fetch-user",
              "value": "?1"
            },
            {
              "name": "upgrade-insecure-requests",
              "value": "1"
            },
            {
              "name": "user-agent",
              "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/101.0.4929.0 Safari/537.36"
            }
          ],
          "queryString": [],
          "headersSize": 538,
          "bodySize": 0
        },
        "response": {
          "status": 302,
          "statusText": "",
          "httpVersion": "HTTP/2.0",
          "cookies": [],
          "headers": [
            {
              "name": "cache-control",
              "value": "no-store"
            },
            {
              "name": "content-length",
              "value": "0"
            },
            {
              "name": "date",
              "value": "Tue, 26 Apr 2022 06:32:08 GMT"
            },
            {
              "name": "location",
              "value": "/en-US/"
            },
            {
              "name": "server",
              "value": "CloudFront"
            },
            {
              "name": "via",
              "value": "1.1 4aba10f1ef4bda5f7c1b86a07ca64264.cloudfront.net (CloudFront)"
            },
            {
              "name": "x-amz-cf-id",
              "value": "v0Jniif8i0uAlBhkapx1lIUxbwVlJmkR_chYmG9s4a0QE8mUCEJl2w=="
            },
            {
              "name": "x-amz-cf-pop",
              "value": "HEL51-P1"
            },
            {
              "name": "x-cache",
              "value": "Miss from cloudfront"
            }
          ],
          "content": {
            "size": -1,
            "mimeType": "x-unknown",
            "compression": 0
          },
          "headersSize": 334,
          "bodySize": 0,
          "redirectURL": "https://developer.mozilla.org/en-US/",
          "_transferSize": 334
        },
        "cache": {
          "beforeRequest": null,
          "afterRequest": null
        },
        "timings": {
          "dns": 90.418,
          "connect": 58.401,
          "ssl": 40.94,
          "send": 0,
          "wait": 61.566,
          "receive": 1.647
        },
        "pageref": "page@6098b3b1b408cf628e5ba3ee76bd5c3a",
        "serverIPAddress": "108.156.22.40",
        "_serverPort": 443,
        "_securityDetails": {
          "protocol": "TLS 1.3",
          "subjectName": "developer.mozilla.org",
          "issuer": "Amazon",
          "validFrom": 1633392000,
          "validTo": 1667433599
        }
      },
      {
        "_requestref": "request@5e9297339b23574d50a26cfcefc72b35",
        "_frameref": "frame@d899d6d52c23551ce4c08380fda5d06b",
        "_monotonicTime": 1128865293.382,
        "startedDateTime": "2022-04-26T06:32:08.714Z",
        "time": 25.923,
        "request": {
          "method": "GET",
          "url": "https://developer.mozilla.org/en-US/",
          "httpVersion": "HTTP/2.0",
          "cookies": [],
          "headers": [
            {
              "name": ":authority",
              "value": "developer.mozilla.org"
            },
            {
              "name": ":method",
              "value": "GET"
            },
            {
              "name": ":path",
              "value": "/en-US/"
            },
            {
              "name": ":scheme",
              "value": "https"
            },
            {
              "name": "accept",
              "value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
            },
            {
              "name": "accept-encoding",
              "value": "gzip, deflate, br"
            },
            {
              "name": "sec-fetch-dest",
              "value": "document"
            },
            {
              "name": "sec-fetch-mode",
              "value": "navigate"
            },
            {
              "name": "sec-fetch-site",
              "value": "none"
            },
            {
              "name": "sec-fetch-user",
              "value": "?1"
            },
            {
              "name": "upgrade-insecure-requests",
              "value": "1"
            },
            {
              "name": "user-agent",
              "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/101.0.4929.0 Safari/537.36"
            }
          ],
          "queryString": [],
          "headersSize": 550,
          "bodySize": 0
        },
        "response": {
          "status": 200,
          "statusText": "",
          "httpVersion": "HTTP/2.0",
          "cookies": [],
          "headers": [
            {
              "name": "age",
              "value": "19240"
            },
            {
              "name": "cache-control",
              "value": "max-age=86400, public"
            },
            {
              "name": "content-encoding",
              "value": "gzip"
            },
            {
              "name": "content-security-policy",
              "value": "default-src 'self'; script-src 'report-sample' 'self' *.speedcurve.com 'sha256-q7cJjDqNO2e1L5UltvJ1LhvnYN7yJXgGO7b6h9xkL1o=' www.google-analytics.com/analytics.js 'sha256-JEt9Nmc3BP88wxuTZm9aKNu87vEgGmKW1zzy/vb1KPs=' polyfill.io/v3/polyfill.min.js assets.codepen.io production-assets.codepen.io 'sha256-x6Tv+AdV5e6dcolO0TEo+3BG4H2nG2ACjyG8mz6QCes='; script-src-elem 'report-sample' 'self' *.speedcurve.com 'sha256-q7cJjDqNO2e1L5UltvJ1LhvnYN7yJXgGO7b6h9xkL1o=' www.google-analytics.com/analytics.js 'sha256-JEt9Nmc3BP88wxuTZm9aKNu87vEgGmKW1zzy/vb1KPs=' polyfill.io/v3/polyfill.min.js assets.codepen.io production-assets.codepen.io 'sha256-x6Tv+AdV5e6dcolO0TEo+3BG4H2nG2ACjyG8mz6QCes='; style-src 'report-sample' 'self' 'unsafe-inline'; object-src 'none'; base-uri 'self'; connect-src 'self' updates.developer.allizom.org updates.developer.mozilla.org www.google-analytics.com stats.g.doubleclick.net; font-src 'self'; frame-src 'self' interactive-examples.mdn.mozilla.net interactive-examples.prod.mdn.mozilla.net interactive-examples.stage.mdn.mozilla.net mdn.github.io yari-demos.prod.mdn.mozit.cloud mdn.mozillademos.org yari-demos.stage.mdn.mozit.cloud jsfiddle.net www.youtube-nocookie.com codepen.io; img-src 'self' *.githubusercontent.com *.googleusercontent.com *.gravatar.com mozillausercontent.com firefoxusercontent.com profile.stage.mozaws.net profile.accounts.firefox.com lux.speedcurve.com mdn.mozillademos.org media.prod.mdn.mozit.cloud media.stage.mdn.mozit.cloud interactive-examples.mdn.mozilla.net interactive-examples.prod.mdn.mozilla.net interactive-examples.stage.mdn.mozilla.net wikipedia.org www.google-analytics.com www.gstatic.com; manifest-src 'self'; media-src 'self' archive.org videos.cdn.mozilla.net; child-src 'self'; worker-src 'self';"
            },
            {
              "name": "content-type",
              "value": "text/html; charset=utf-8"
            },
            {
              "name": "date",
              "value": "Tue, 26 Apr 2022 01:11:28 GMT"
            },
            {
              "name": "etag",
              "value": "W/\"42ba7227a513c5710d41655ea48ee383\""
            },
            {
              "name": "last-modified",
              "value": "Tue, 26 Apr 2022 01:07:59 GMT"
            },
            {
              "name": "server",
              "value": "AmazonS3"
            },
            {
              "name": "strict-transport-security",
              "value": "max-age=63072000"
            },
            {
              "name": "vary",
              "value": "Accept-Encoding"
            },
            {
              "name": "via",
              "value": "1.1 4aba10f1ef4bda5f7c1b86a07ca64264.cloudfront.net (CloudFront)"
            },
            {
              "name": "x-amz-cf-id",
              "value": "qcSXQjRuX7iZySAzlBseXVLXjRUOy9oH2XPybDnNx2o4cD6vO1ODXg=="
            },
            {
              "name": "x-amz-cf-pop",
              "value": "HEL51-P1"
            },
            {
              "name": "x-cache",
              "value": "Hit from cloudfront"
            },
            {
              "name": "x-content-type-options",
              "value": "nosniff"
            },
            {
              "name": "x-frame-options",
              "value": "DENY"
            },
            {
              "name": "x-xss-protection",
              "value": "1; mode=block"
            }
          ],
          "content": {
            "size": 79221,
            "mimeType": "text/html; charset=utf-8",
            "compression": 62570
          },
          "headersSize": 2423,
          "bodySize": 16651,
          "redirectURL": "",
          "_transferSize": 16651
        },
        "cache": {
          "beforeRequest": null,
          "afterRequest": null
        },
        "timings": {
          "dns": -1,
          "connect": -1,
          "ssl": -1,
          "send": 0,
          "wait": 25.923,
          "receive": -1
        },
        "pageref": "page@6098b3b1b408cf628e5ba3ee76bd5c3a",
        "serverIPAddress": "108.156.22.40",
        "_serverPort": 443,
        "_securityDetails": {
          "protocol": "TLS 1.3",
          "subjectName": "developer.mozilla.org",
          "issuer": "Amazon",
          "validFrom": 1633392000,
          "validTo": 1667433599
        }
      },
      ...

The filtered urls.json:

[
  "https://developer.mozilla.org/",
  "https://developer.mozilla.org/en-US/",
  "https://developer.mozilla.org/static/media/ZillaSlab-Bold.subset.0beac26b.woff2",
  "https://developer.mozilla.org/static/css/main.55b68e13.chunk.css",
  "https://cdn.speedcurve.com/js/lux.js?id=108906238",
  "https://developer.mozilla.org/static/js/ga.js",
  "https://developer.mozilla.org/static/js/runtime-main.e858fdfd.js",
  "https://developer.mozilla.org/static/js/4.a756dea3.chunk.js",
  "https://developer.mozilla.org/static/js/main.e0441009.chunk.js",
  "https://developer.mozilla.org/static/media/triangle.e8097649.svg",
  "https://developer.mozilla.org/static/media/search.43d0e3bf.svg",
  "https://developer.mozilla.org/static/media/home-masthead-background.4306d252.svg",
  "https://developer.mozilla.org/static/media/arrow.e1de8ede.svg",
  "https://developer.mozilla.org/static/media/twitter.cc5b37fe.svg",
  "https://developer.mozilla.org/static/media/github-mark-small.348586b8.svg",
  "https://developer.mozilla.org/static/media/instagram.445373fc.svg",
  "https://developer.mozilla.org/static/media/ZillaSlab-Regular.subset.ce3a756d.woff2",
  "https://www.google-analytics.com/analytics.js",
  "https://www.google-analytics.com/j/collect?v=1&_v=j96&aip=1&a=1903151242&t=pageview&_s=1&dl=https%3A%2F%2Fdeveloper.mozilla.org%2Fen-US%2F&ul=en-us&de=UTF-8&dt=MDN%20Web%20Docs&sd=30-bit&sr=1280x720&vp=1280x720&je=0&_u=YEBAAAABAAAAAC~&jid=342012907&gjid=421563640&cid=549396694.1643871150&tid=UA-36116321-5&_gid=346451150.1643871150&_r=1&_slc=1&z=1690852481",
  "https://stats.g.doubleclick.net/j/collect?t=dc&aip=1&_r=3&v=1&_v=j96&tid=UA-36116321-5&cid=549396694.1643871150&jid=342012907&gjid=421563640&_gid=346451150.1643871150&_u=YEBAAAAAAAAAAC~&z=553170267"
]

1 Like

Thanks Jani for the elaborate explanation.

appreciate your assistance.