Skip to content

Vibium Find: Natural Language by Default

find locates an element by what it is, not only by how the page is built. A bare string is a description. Exact locators are explicit. A reference image is matched where it appears on screen. Every match becomes an @eN ref that click, tap, highlight, and the SDK element methods accept.

Find by intent. Interact by reference.

A bare string used to be a CSS selector. It is now a description.

Python
# Before: positional CSS
vibe.find("h1")
vibe.find(".submit")
vibe.find("#login")
# After: explicit CSS
vibe.find(css="h1")
vibe.find(css=".submit")
vibe.find(css="#login")
# New: description by default
vibe.find("the main heading")
vibe.find("the submit button")
JavaScript
// Before // After
vibe.find('h1') vibe.find({ css: 'h1' })
vibe.find('.submit') vibe.find({ css: '.submit' })
vibe.findAll('.row') vibe.findAll({ css: '.row' })
vibe.find('the submit button')
Java
// Before // After
page.find("h1") page.find(new SelectorOptions().css("h1"))
page.findAll(".row") page.findAll(new SelectorOptions().css(".row"))
page.find("the submit button")
CLI
# Before # After
vibium find "h1" vibium find css h1
vibium find ".submit" vibium find css .submit
vibium find "#login" vibium find css '#login'
vibium find "the main heading"

The migration is a spelling change. The meaning is not the same: a description is resolved against the page’s names and roles, and when they are not enough, against a screenshot.

A bare string that reads like a selector (h1, .submit, #login, nav > a) is refused at once, before any wait and before any model call, with the css form for your surface. It is never reinterpreted as CSS, and it never costs a model request. An element genuinely named “h1” is reachable with the explicit option: vibium find description "h1", find({ description: 'h1' }), find(description="h1"). The MCP tool’s selector field and the SDK selector key keep meaning CSS.

Only find changed. click "<selector>", fill, wait, count, and the other selector-taking commands still take CSS.

FormCLIJavaScriptPythonJava
Descriptionfind "the Add to Cart button"find('the Add to Cart button')find("the Add to Cart button")find("the Add to Cart button")
CSSfind css "button.primary"find({ css: 'button.primary' })find(css="button.primary")new SelectorOptions().css("button.primary")
Tagfind tag h1find({ tag: 'h1' })find(tag="h1").tag("h1")
Role + textfind role button --name "Add to Cart"find({ role: 'button', text: 'Add to Cart' })find(role="button", text="Add to Cart").role("button").text("Add to Cart")
Accessible namefind role link --name "More"find({ role: 'link', name: 'More' })find(role="link", name="More").role("link").name("More")
Imagefind image thumbnail.pngfind({ image: 'thumbnail.png' })find(image="thumbnail.png").image(Path.of("thumbnail.png"))

text, label, placeholder, testid, xpath, alt, and title keep their meanings and still compose with role and css. name filters by the accessible name a person reads the element by (text, label, alt, or aria-label); it is what find role ... --name has always meant. A description and an image each stand alone; combining either with another locator is an error.

findAll, find_all, and --all take the same forms and return every match. JSON output from the CLI is always an array, empty when nothing matched:

Terminal window
$ vibium find image thumbnail.png --all --json
{"ok":true,"result":[{"ref":"@e5","role":"image","tag":"img","box":{"x":20,"y":112,"width":96,"height":96},"source":"image",...}]}
  1. The page’s names and roles are read: controls, headings, images, and labelled fields. An exact name wins; a single element whose name contains the phrase, or the lone element of a named role, is accepted. Words the tree cannot judge, such as colors, positions, and “icon”, skip this step.
  2. If that is not enough and an AI provider is configured, the screenshot goes to a vision model, which answers with bounding boxes. Boxes are checked against the image, snapped to the element under them when one covers them, and registered as refs.
  3. With no provider, the miss is reported with what would have happened next, and with the css form when the description reads like a selector.

The model call is one bounded turn. It receives one screenshot and the request, reports boxes, and never acts.

  1. The viewport is captured and the reference is matched with SIFT features and a geometric check, locally, in the binary. Every verified occurrence is returned, deduplicated, with its box. Nothing leaves the machine.
  2. When nothing verifies and a provider is configured, the screenshot and the reference go to the model.
  3. Without a provider, no match is an empty list (or a miss for the single form), never a configuration error.

Only what is on screen is searched. Scroll first when the target is below the fold. A flat or tiny reference that yields too few features is reported as such; use a larger or more detailed crop.

--strategy (strategy= in the SDKs) chooses the method:

ValueDescription locatorImage locatorcss, tag, role, …
auto (default)tree first, then model if configuredlocal first, then model if configuredexact
cverrorlocal only; never calls a modelerror
vlmmodel only; needs a providermodel only; needs a providererror
treepage tree onlyerrorexact

auto can send a screenshot to the configured provider: for a description, when the tree cannot answer; for an image, when local matching finds nothing. The reference image goes with it. Each call is billed by the provider like a vibium check turn. cv, css, tag, and the semantic locators never call a model. --details shows which method answered and whether a model was called.

Description and image fallback use the same settings as vibium run and vibium check: VIBIUM_AI_PROVIDER, VIBIUM_AI_MODEL, VIBIUM_AI_BASE_URL, VIBIUM_AI_REASONING_EFFORT, the provider’s API key, and ~/.config/vibium/ai.env. The model must accept images. Override them for one call with --provider, --model, --ai-base-url, and --reasoning-effort on the CLI, provider/model/aiBaseURL/ reasoningEffort in the SDK options, or the same fields on the MCP tools. There is no separate find model setting.

find and map share one table per page. The same element keeps its ref across calls; new matches take the next number. A match with no element of its own, such as a region inside a canvas or a phone screen, is driven by coordinates: click, dblclick, hover, tap, highlight, and bounds work; fill, type, and the other DOM commands refuse it. Before a coordinate click the page is checked: a navigation makes the ref stale, and a scroll or resize re-detects an image match or, for a model match, reports the ref stale. Run find again.

--details adds the method (css, semantic, tree, sift, vlm), whether a model was called, the provider and model, elapsed time, and the raw scores. A SIFT inlier count, a detector score, and a model’s confidence are different quantities. None is a probability.