BiDi Data Collection: How Vibium Reads Request Bodies
This guide was updated after the 26.11.16 release, so parts of it MAY need a nightly build. See What’s in Nightly.
WebDriver BiDi network events tell you about requests — the URL, method, headers, status code — but they don’t include the actual body bytes. This is by design: cloning every request and response body would be expensive, and most automation doesn’t need it.
When you do need body data (reading POST payloads, capturing response content), BiDi provides an opt-in system: data collectors.
The Problem
Section titled “The Problem”When a network.beforeRequestSent event fires, you get:
{ "method": "network.beforeRequestSent", "params": { "request": { "request": "req-123", "url": "https://api.example.com/users", "method": "POST", "headers": [{"name": "Content-Type", "value": {"type": "string", "value": "application/json"}}], "bodySize": 42 } }}You can see that 42 bytes were sent, but you can’t see what those bytes are. bodySize tells you a body exists — getting the actual content requires a different mechanism.
How Data Collectors Work
Section titled “How Data Collectors Work”Data collection is a three-step protocol:
Vibium Browser │ │ │ 1. addDataCollector │ │ "watch request bodies" │ │──────────────────────────────►│ │ │ │ ┌───────────────────┤ │ │ 2. Clones request │ │ │ bodies into │ │ │ buffer as they │ │ │ happen │ │ └───────────────────┤ │ │ │ 3. getData(request: "req-1") │ │──────────────────────────────►│ │ │ │ body bytes │ │◄──────────────────────────────│ │ │Step 1: Register a collector
Section titled “Step 1: Register a collector”Tell the browser what to collect and how much to buffer:
{ "method": "network.addDataCollector", "params": { "dataTypes": ["request"], "maxEncodedDataSize": 10485760 }}dataTypes— what to collect:"request","response", or bothmaxEncodedDataSize— max bytes per individual item (not total). Requests larger than this are silently skipped.
The browser returns a collector ID:
{ "result": { "collector": "collector-abc-123" }}From this point on, the browser clones the body of every matching network request into a buffer. The cloning happens at the network layer, before the page’s JavaScript sees the data.
Step 2: Wait for a request to happen
Section titled “Step 2: Wait for a request to happen”The collector runs in the background. You don’t need to do anything special — just let requests happen naturally (page navigation, fetch calls, form submissions, etc.).
Step 3: Retrieve the collected data
Section titled “Step 3: Retrieve the collected data”When you want the body for a specific request, ask for it by request ID:
{ "method": "network.getData", "params": { "dataType": "request", "request": "req-123" }}The browser returns the raw bytes:
{ "result": { "bytes": { "type": "string", "value": "{\"name\":\"Alice\",\"email\":\"alice@example.com\"}" } }}If the request hasn’t completed yet, network.getData blocks until the body is available. If the body was too large for the collector’s maxEncodedDataSize, or no collector was active when the request happened, you get a no such network data error.
Cleanup: Remove the collector
Section titled “Cleanup: Remove the collector”When you’re done collecting, tear down the collector to free resources:
{ "method": "network.removeDataCollector", "params": { "collector": "collector-abc-123" }}Already-collected data remains available until the browser discards it (subject to a global memory limit).
Why Opt-In?
Section titled “Why Opt-In?”Body cloning has real costs:
- Memory — The browser must buffer a copy of every matching body. A page that loads 50 images and 20 API calls could consume significant memory.
- Performance — Cloning the network stream adds overhead to every request.
- Privacy — Request bodies can contain credentials, tokens, PII. Not collecting them by default is safer.
The opt-in model means you only pay these costs when you need them. Most automation (clicking buttons, reading text, taking screenshots) never touches request bodies.
How Vibium Uses This
Section titled “How Vibium Uses This”request.postData()
Section titled “request.postData()”When you call page.route() or page.onRequest(), Vibium automatically sets up a data collector for request bodies behind the scenes. This makes request.postData() work:
await page.route('**/api/**', async (route) => { const body = await route.request.postData(); console.log('API received:', body); // '{"name":"Alice"}' route.continue();});def log_body(route): body = route.request["post_data"] print("API received:", body) # '{"name":"Alice"}' route.continue_()
page.route("**/api/**", log_body)page.route("**/api/**", route -> { String body = route.request().postData(); System.out.println("API received: " + body); // {"name":"Alice"} route.doContinue();});In the Python async API the handler is async def and reads await route.request.post_data().
Without a data collector, postData() would always return null — the BiDi event simply doesn’t include the body.
The collector is set up lazily (only when you register a route or request listener) and torn down when you call page.removeAllListeners('request'). If the browser doesn’t support network.addDataCollector, postData() gracefully returns null.
Scoping
Section titled “Scoping”The addDataCollector command accepts optional contexts and userContexts parameters to limit collection to specific pages or browser contexts. Vibium currently registers a session-wide collector (no context filter) so that postData() works regardless of which page made the request.
Common Use Cases
Section titled “Common Use Cases”Verifying API Payloads
Section titled “Verifying API Payloads”The most common case: your test clicks a button, and you want to verify what data was sent to the server.
page.onRequest(async (req) => { if (req.method() === 'POST' && req.url().includes('/api/checkout')) { const body = JSON.parse(await req.postData()); assert(body.items.length === 3); assert(body.total === 59.97); }});
await page.find({ role: 'button', text: 'Place Order' }).click();import json
def check_order(req): if req.method() == "POST" and "/api/checkout" in req.url(): body = json.loads(req.post_data()) assert len(body["items"]) == 3 assert body["total"] == 59.97
page.on_request(check_order)
page.find(role="button", text="Place Order").click()page.onRequest(req -> { if (req.method().equals("POST") && req.url().contains("/api/checkout")) { String body = req.postData(); // {"items":[1,2,3],"total":59.97} }});
page.find(new SelectorOptions().role("button").text("Place Order")).click();Request Mocking with Body Inspection
Section titled “Request Mocking with Body Inspection”Intercept requests, inspect the body, and return different mock responses based on what was sent:
await page.route('**/api/search', async (route) => { const body = JSON.parse(await route.request.postData());
if (body.query === 'empty') { route.fulfill({ status: 200, body: JSON.stringify({ results: [] }) }); } else { route.fulfill({ status: 200, body: JSON.stringify({ results: ['item1'] }) }); }});import json
def mock_search(route): body = json.loads(route.request["post_data"]) if body["query"] == "empty": route.fulfill(status=200, body=json.dumps({"results": []})) else: route.fulfill(status=200, body=json.dumps({"results": ["item1"]}))
page.route("**/api/search", mock_search)page.route("**/api/search", route -> { String body = route.request().postData(); // {"query":"empty"} String results = body.contains("\"empty\"") ? "[]" : "[\"item1\"]"; route.fulfill(new FulfillOptions().status(200).body("{\"results\":" + results + "}"));});Form Submission Validation
Section titled “Form Submission Validation”Verify that form data is properly encoded before it hits the server:
const reqPromise = page.capture.request('**/submit');
await page.find({ css: '#name' }).fill('Jane Doe');await page.find({ css: '#email' }).fill('jane@example.com');await page.find({ role: 'button', text: 'Submit' }).click();
const req = await reqPromise;const body = await req.postData();// body === 'name=Jane+Doe&email=jane%40example.com'def submit(): page.find(css="#name").fill("Jane Doe") page.find(css="#email").fill("jane@example.com") page.find(role="button", text="Submit").click()
req = page.capture.request("**/submit", submit)body = req["post_data"]# body == 'name=Jane+Doe&email=jane%40example.com'Request req = page.capture().request("**/submit", () -> { page.find(new SelectorOptions().css("#name")).fill("Jane Doe"); page.find(new SelectorOptions().css("#email")).fill("jane@example.com"); page.find(new SelectorOptions().role("button").text("Submit")).click();});String body = req.postData();// body is "name=Jane+Doe&email=jane%40example.com"Response Body Collection: response.body() and response.json()
Section titled “Response Body Collection: response.body() and response.json()”When you call page.onResponse() or page.waitForResponse(), Vibium sets up a data collector that includes the "response" data type. This makes response.body() and response.json() work:
page.onResponse(async (resp) => { if (resp.url().includes('/api/users')) { const data = await resp.json(); console.log('Got users:', data); // [{name: "Alice"}, ...] }});def log_users(resp): if "/api/users" in resp.url(): data = resp.json() print("Got users:", data) # [{'name': 'Alice'}, ...]
page.on_response(log_users)page.onResponse(resp -> { if (resp.url().contains("/api/users")) { Object data = resp.json(); System.out.println("Got users: " + data); // [{name=Alice}, ...] }});Or with waitForResponse:
const resp = await page.capture.response('**/api/users');const body = await resp.body(); // raw stringconst data = await resp.json(); // parsed JSONimport json
resp = page.capture.response("**/api/users", lambda: page.find(css="#users").click())body = resp["body"] # raw stringdata = json.loads(body) # parsed JSONResponse resp = page.capture().response("**/api/users", () -> page.find(new SelectorOptions().css("#users")).click());String body = resp.body(); // raw stringObject data = resp.json(); // parsed JSONUnder the hood, body() calls network.getData with "dataType": "response" and the request ID from the network.responseCompleted event. Base64-encoded responses (binary content) are automatically decoded. If no data collector was active when the response arrived, body() returns null.
Network Tracing
Section titled “Network Tracing”Collecting both request and response bodies gives you a complete record of all network traffic — useful for debugging, generating HAR files, or replay testing:
{ "method": "network.addDataCollector", "params": { "dataTypes": ["request", "response"], "maxEncodedDataSize": 5242880 }}Combined with network.beforeRequestSent and network.responseCompleted events (which provide URLs, headers, timing, etc.), you have everything needed to reconstruct the full HTTP conversation.
Limits and Edge Cases
Section titled “Limits and Edge Cases”maxEncodedDataSizeis per-item, not total. Setting it to 10 MB means each individual request body can be up to 10 MB, not 10 MB across all requests.- The browser has a global memory cap (
max total collected size) for all collected data. When it’s exceeded, older data gets evicted. The exact limit is browser-dependent. - Collector must be active when the request happens. You can’t retroactively collect data for requests that already completed before the collector was registered.
- Binary bodies are base64-encoded in the
bytesfield (with"type": "base64"). Text bodies use"type": "string". - The
disownflag onnetwork.getDatalets you release a specific collector’s claim on the data, freeing memory earlier.