Skip to content

BiDi Data Collection: How Vibium Reads Request Bodies

This guide was updated after the 26.11.16 release, so parts of it MAY need a nightly build. See What’s in Nightly.

WebDriver BiDi network events tell you about requests — the URL, method, headers, status code — but they don’t include the actual body bytes. This is by design: cloning every request and response body would be expensive, and most automation doesn’t need it.

When you do need body data (reading POST payloads, capturing response content), BiDi provides an opt-in system: data collectors.

When a network.beforeRequestSent event fires, you get:

{
"method": "network.beforeRequestSent",
"params": {
"request": {
"request": "req-123",
"url": "https://api.example.com/users",
"method": "POST",
"headers": [{"name": "Content-Type", "value": {"type": "string", "value": "application/json"}}],
"bodySize": 42
}
}
}

You can see that 42 bytes were sent, but you can’t see what those bytes are. bodySize tells you a body exists — getting the actual content requires a different mechanism.

Data collection is a three-step protocol:

Vibium Browser
│ │
│ 1. addDataCollector │
│ "watch request bodies" │
│──────────────────────────────►│
│ │
│ ┌───────────────────┤
│ │ 2. Clones request │
│ │ bodies into │
│ │ buffer as they │
│ │ happen │
│ └───────────────────┤
│ │
│ 3. getData(request: "req-1") │
│──────────────────────────────►│
│ │
│ body bytes │
│◄──────────────────────────────│
│ │

Tell the browser what to collect and how much to buffer:

{
"method": "network.addDataCollector",
"params": {
"dataTypes": ["request"],
"maxEncodedDataSize": 10485760
}
}
  • dataTypes — what to collect: "request", "response", or both
  • maxEncodedDataSize — max bytes per individual item (not total). Requests larger than this are silently skipped.

The browser returns a collector ID:

{
"result": {
"collector": "collector-abc-123"
}
}

From this point on, the browser clones the body of every matching network request into a buffer. The cloning happens at the network layer, before the page’s JavaScript sees the data.

The collector runs in the background. You don’t need to do anything special — just let requests happen naturally (page navigation, fetch calls, form submissions, etc.).

When you want the body for a specific request, ask for it by request ID:

{
"method": "network.getData",
"params": {
"dataType": "request",
"request": "req-123"
}
}

The browser returns the raw bytes:

{
"result": {
"bytes": {
"type": "string",
"value": "{\"name\":\"Alice\",\"email\":\"alice@example.com\"}"
}
}
}

If the request hasn’t completed yet, network.getData blocks until the body is available. If the body was too large for the collector’s maxEncodedDataSize, or no collector was active when the request happened, you get a no such network data error.

When you’re done collecting, tear down the collector to free resources:

{
"method": "network.removeDataCollector",
"params": {
"collector": "collector-abc-123"
}
}

Already-collected data remains available until the browser discards it (subject to a global memory limit).

Body cloning has real costs:

  • Memory — The browser must buffer a copy of every matching body. A page that loads 50 images and 20 API calls could consume significant memory.
  • Performance — Cloning the network stream adds overhead to every request.
  • Privacy — Request bodies can contain credentials, tokens, PII. Not collecting them by default is safer.

The opt-in model means you only pay these costs when you need them. Most automation (clicking buttons, reading text, taking screenshots) never touches request bodies.

When you call page.route() or page.onRequest(), Vibium automatically sets up a data collector for request bodies behind the scenes. This makes request.postData() work:

JavaScript
await page.route('**/api/**', async (route) => {
const body = await route.request.postData();
console.log('API received:', body); // '{"name":"Alice"}'
route.continue();
});
Python
def log_body(route):
body = route.request["post_data"]
print("API received:", body) # '{"name":"Alice"}'
route.continue_()
page.route("**/api/**", log_body)
Java
page.route("**/api/**", route -> {
String body = route.request().postData();
System.out.println("API received: " + body); // {"name":"Alice"}
route.doContinue();
});

In the Python async API the handler is async def and reads await route.request.post_data().

Without a data collector, postData() would always return null — the BiDi event simply doesn’t include the body.

The collector is set up lazily (only when you register a route or request listener) and torn down when you call page.removeAllListeners('request'). If the browser doesn’t support network.addDataCollector, postData() gracefully returns null.

The addDataCollector command accepts optional contexts and userContexts parameters to limit collection to specific pages or browser contexts. Vibium currently registers a session-wide collector (no context filter) so that postData() works regardless of which page made the request.

The most common case: your test clicks a button, and you want to verify what data was sent to the server.

JavaScript
page.onRequest(async (req) => {
if (req.method() === 'POST' && req.url().includes('/api/checkout')) {
const body = JSON.parse(await req.postData());
assert(body.items.length === 3);
assert(body.total === 59.97);
}
});
await page.find({ role: 'button', text: 'Place Order' }).click();
Python
import json
def check_order(req):
if req.method() == "POST" and "/api/checkout" in req.url():
body = json.loads(req.post_data())
assert len(body["items"]) == 3
assert body["total"] == 59.97
page.on_request(check_order)
page.find(role="button", text="Place Order").click()
Java
page.onRequest(req -> {
if (req.method().equals("POST") && req.url().contains("/api/checkout")) {
String body = req.postData(); // {"items":[1,2,3],"total":59.97}
}
});
page.find(new SelectorOptions().role("button").text("Place Order")).click();

Intercept requests, inspect the body, and return different mock responses based on what was sent:

JavaScript
await page.route('**/api/search', async (route) => {
const body = JSON.parse(await route.request.postData());
if (body.query === 'empty') {
route.fulfill({ status: 200, body: JSON.stringify({ results: [] }) });
} else {
route.fulfill({ status: 200, body: JSON.stringify({ results: ['item1'] }) });
}
});
Python
import json
def mock_search(route):
body = json.loads(route.request["post_data"])
if body["query"] == "empty":
route.fulfill(status=200, body=json.dumps({"results": []}))
else:
route.fulfill(status=200, body=json.dumps({"results": ["item1"]}))
page.route("**/api/search", mock_search)
Java
page.route("**/api/search", route -> {
String body = route.request().postData(); // {"query":"empty"}
String results = body.contains("\"empty\"") ? "[]" : "[\"item1\"]";
route.fulfill(new FulfillOptions().status(200).body("{\"results\":" + results + "}"));
});

Verify that form data is properly encoded before it hits the server:

JavaScript
const reqPromise = page.capture.request('**/submit');
await page.find({ css: '#name' }).fill('Jane Doe');
await page.find({ css: '#email' }).fill('jane@example.com');
await page.find({ role: 'button', text: 'Submit' }).click();
const req = await reqPromise;
const body = await req.postData();
// body === 'name=Jane+Doe&email=jane%40example.com'
Python
def submit():
page.find(css="#name").fill("Jane Doe")
page.find(css="#email").fill("jane@example.com")
page.find(role="button", text="Submit").click()
req = page.capture.request("**/submit", submit)
body = req["post_data"]
# body == 'name=Jane+Doe&email=jane%40example.com'
Java
Request req = page.capture().request("**/submit", () -> {
page.find(new SelectorOptions().css("#name")).fill("Jane Doe");
page.find(new SelectorOptions().css("#email")).fill("jane@example.com");
page.find(new SelectorOptions().role("button").text("Submit")).click();
});
String body = req.postData();
// body is "name=Jane+Doe&email=jane%40example.com"

Response Body Collection: response.body() and response.json()

Section titled “Response Body Collection: response.body() and response.json()”

When you call page.onResponse() or page.waitForResponse(), Vibium sets up a data collector that includes the "response" data type. This makes response.body() and response.json() work:

JavaScript
page.onResponse(async (resp) => {
if (resp.url().includes('/api/users')) {
const data = await resp.json();
console.log('Got users:', data); // [{name: "Alice"}, ...]
}
});
Python
def log_users(resp):
if "/api/users" in resp.url():
data = resp.json()
print("Got users:", data) # [{'name': 'Alice'}, ...]
page.on_response(log_users)
Java
page.onResponse(resp -> {
if (resp.url().contains("/api/users")) {
Object data = resp.json();
System.out.println("Got users: " + data); // [{name=Alice}, ...]
}
});

Or with waitForResponse:

JavaScript
const resp = await page.capture.response('**/api/users');
const body = await resp.body(); // raw string
const data = await resp.json(); // parsed JSON
Python
import json
resp = page.capture.response("**/api/users", lambda: page.find(css="#users").click())
body = resp["body"] # raw string
data = json.loads(body) # parsed JSON
Java
Response resp = page.capture().response("**/api/users",
() -> page.find(new SelectorOptions().css("#users")).click());
String body = resp.body(); // raw string
Object data = resp.json(); // parsed JSON

Under the hood, body() calls network.getData with "dataType": "response" and the request ID from the network.responseCompleted event. Base64-encoded responses (binary content) are automatically decoded. If no data collector was active when the response arrived, body() returns null.

Collecting both request and response bodies gives you a complete record of all network traffic — useful for debugging, generating HAR files, or replay testing:

{
"method": "network.addDataCollector",
"params": {
"dataTypes": ["request", "response"],
"maxEncodedDataSize": 5242880
}
}

Combined with network.beforeRequestSent and network.responseCompleted events (which provide URLs, headers, timing, etc.), you have everything needed to reconstruct the full HTTP conversation.

  • maxEncodedDataSize is per-item, not total. Setting it to 10 MB means each individual request body can be up to 10 MB, not 10 MB across all requests.
  • The browser has a global memory cap (max total collected size) for all collected data. When it’s exceeded, older data gets evicted. The exact limit is browser-dependent.
  • Collector must be active when the request happens. You can’t retroactively collect data for requests that already completed before the collector was registered.
  • Binary bodies are base64-encoded in the bytes field (with "type": "base64"). Text bodies use "type": "string".
  • The disown flag on network.getData lets you release a specific collector’s claim on the data, freeing memory earlier.