By Seth Earley, Founder & CEO, Earley Information Science
────────────────────────────────────
Published: October 21, 2026
Last Updated: November 3, 2026 | Version 1.0
────────────────────────────────────
Who This Is For: Digital experience leaders, e-commerce and site search managers, enterprise search owners, Chief Data Officers, and AI/ML leaders responsible for retrieval quality. Also valuable for finance and operations executives who fund search programs and need evidence of their results.
Prerequisites: Basic familiarity with site search or enterprise search analytics.
────────────────────────────────────
Ask the owner of almost any search program a simple question: is search better today than it was a year ago? In most organizations the honest answer is "we think so." Synonyms were added, relevance weights were adjusted, a vendor was replaced. Activity is easy to list. Evidence is not, because nobody recorded an objective starting point before the work began.
That gap does not show up in the index. It shows up in the budget meeting, where the people holding the funds ask for proof and receive anecdotes. The program loses that argument for a defensible reason: it never gave them a number they could trust or repeat.
Independent Measurement Is What Gets Funded
A technical manufacturer cut the time its engineers spent searching for information by several hours a week, across thousands of engineers. Nobody argued with the result. The figures came from time tracking and resource management systems that existed before the project did. Because the yardstick was independent of the initiative, the outcome, millions of dollars a year in recovered capacity, was believed, funded, and sustained.
Compare that with the common pattern, in which the search team builds its own instruments, and those instruments tend to show gains rather than test for them. The remedy is unglamorous. Before any optimization begins, build an instrumented baseline: a measurement layer that establishes where search stands, monitors it continuously, and reports in terms an executive can act on. In our practice this is a distinct engagement with its own deliverables, and treating it as a service rather than a byproduct changes how the entire program behaves afterward. You cannot optimize what you cannot see, and most organizations have never measured the contribution of their individual search elements.
Why the Stakes Are Higher Now
Search is no longer only a way for shoppers and employees to find things. It is the retrieval layer that every AI program depends on. There are more variables, more implementations, and more teams and vendors involved, which makes results from different projects hard to compare like for like.
Independent research shows how common the underlying failure already is. Baymard Institute's usability benchmark of the 50 top-grossing US e-commerce sites found that 31% of product-finding tasks using site search ended in failure, and that 70% of the search engines tested could not handle a basic product-type synonym, leaving shoppers to guess a site's internal vocabulary. Those are averages across benchmarked sites, not a verdict on any single program. They do show that the failure is typical, and that most of it goes unmeasured until a budget review forces the question.
One Executive Number, Six Components
Executives will not follow six search metrics and should not be asked to. A composite search performance index combines ranking quality, zero-result rate, search conversion, latency, catalog coverage, and agent share of traffic into a single score. The composite anchors the executive conversation. The components tell the team where to work, and when the score moves, the cause is visible and traceable.
Agent share is the newest component and is fast becoming the most important, because it is the denominator against which every other number must be read. A metric computed from human behavior means something different when 5% of traffic is machine than when 40% is.
What Ranking Quality Can and Cannot Tell You
Compute ranking metrics such as Normalized Discounted Cumulative Gain (NDCG) from behavioral evidence: clicks, add-to-cart events, and purchases. Avoid a hand-picked test set that the search team assembled and the search team passes. Then segment by query tier. Head, torso, and tail queries fail in different ways and respond to different remedies, and a program that watches only its top 100 queries is watching the part of demand least likely to be broken.
Know the limit of the metric. Ranking quality grades the order of what came back. It says nothing about whether the right products were in the result set. A product missing an attribute, or classified inconsistently, never appears, so it never registers as a miss. That blind spot is why ranking quality has to be read beside zero-result tracking and catalog coverage.
Date the baseline as well, and record what share of traffic was human on the day it was taken. Behavioral ground truth loses value as agent traffic grows, because agents click nothing. A program that does not track the mix will keep reporting a number long after it stops describing demand.
Where Averages Hide the Damage
Every query that returns nothing is a failed customer interaction, and the aggregate rate is where those failures conceal themselves. An overall zero-result rate of 8% can sit on top of a 25% failure rate in one high-value category. The average looks tolerable while a business line starves.
Baseline by category and weight by revenue. The categories where high failure meets high revenue are the ones to fix first. Baymard's own audits add a reminder of why this matters: close to half of e-commerce sites still offer shoppers no way to recover from a search that returns nothing, which turns a hidden category failure into an abandoned session.
Alerts That Anticipate
Threshold alerts on the core metrics give a team foresight rather than hindsight. Relevance drift is subtle. A catalog import here and a mapping change there are each small enough to pass unnoticed, yet the cost compounds as performance erodes. Alerting converts that slow decay into a correction made the same week, not a surprise at the end of the quarter.
The Discipline of Measuring Without Fixing
The counterintuitive feature of a baseline engagement is its narrow scope. It does not remediate. It answers one question, how healthy is our search, and stops there. It is the diagnostic, not the treatment.
That restraint is what makes the numbers credible. The question is never really which party measures and which party improves. It is whether the baseline was locked before anyone started fixing anything. A baseline computed after remediation has begun, or revised once results are known, cannot be trusted no matter who produced it. Once it is locked, any skeptic can check it, improvement targets are set against evidence rather than aspiration, and the optimization work is judged against a clear and defensible starting point. The sequence is baseline, then remediation, then baseline again.
The market is reaching the same conclusion from another direction. Adobe now publishes machine-readability benchmarks for retail sites, scoring what AI systems can and cannot see on a page before anyone debates strategy. The practice is no longer fringe, and organizations that skip the baseline now trail their own tooling vendors.
A Four-Week Path
This is a month of work, not a quarter. Most enterprises already hold query logs, click and conversion events, and catalog metadata in their analytics and commerce platforms, so the task is extraction and access rather than new instrumentation. A baseline also needs less observation time than most programs assume. Two to four weeks smooths out day-of-week effects and produces a reliable read. High-traffic categories often stabilize sooner, while long-tail queries need the full window.
In the first week, connect the data sources: query logs, click and conversion events, catalog and index metadata, and latency timings. Flag AI referral and agent traffic in the logs from day one so the human and machine split is visible in the first snapshot. This is plumbing only, with no dashboards yet.
In weeks two and three, establish the baseline. Measure ranking quality by query tier, zero-result rate by category weighted by revenue, search conversion segmented by buyer type, and latency percentiles, reporting human and machine traffic separately. Head queries settle quickly. Tail queries need the longer window before their numbers deserve trust. Record the snapshot, because every future claim will be measured against it.
In week four, switch on the composite score, the alert thresholds, and the first executive report: a single page covering the composite score and its direction, the three weakest categories, and what each one is costing the business.
Then begin optimizing, with every sprint accountable to a number that predates it. Organizations that invert the order get a year of activity and no evidence. In the era of machine buyers, evidence is the one search deliverable a budget meeting will accept.
Frequently Asked Questions
What is an instrumented search baseline?
An instrumented search baseline is a measurement layer, built before any optimization begins, that establishes where search performance stands, monitors it continuously, and reports it as a composite score executives can track. It draws on query logs, click and conversion events, and catalog metadata that most enterprises already hold in existing analytics and commerce platforms, so the work is extraction and access rather than new instrumentation.
Why should search be measured before it is tuned?
Without a baseline recorded before any changes, a team cannot show whether new synonyms, weighting adjustments, or a vendor replacement improved anything, because instruments built by the project itself tend to display gains rather than test them. Locking the baseline first gives executives a number that predates the work, which is the kind of evidence a budget review will accept and fund.
Why does the independence of the measurement source matter to executives?
A figure drawn from systems that existed before an initiative cannot be accused of flattering it. In one technical manufacturing enterprise, the reduction in engineer search time came from existing time tracking and resource management systems, so the result, millions of dollars a year in recovered capacity, was believed, funded, and sustained. A yardstick the project invented for itself invites the argument that the gain was built into the measure.
Why can a search program score well on ranking quality and still lose sales?
Ranking quality grades only the order of the results that came back. A product missing a required attribute, or classified inconsistently, never enters the result set, so it never registers as a ranking miss and no abandoned session points to it. Reading ranking quality beside zero-result tracking by category and catalog coverage exposes the products that are eligible in inventory but invisible to search.
Why should a baseline record the human to machine traffic mix on the day it is taken?
Behavioral metrics such as click-based ranking quality describe human shoppers, and AI agents click nothing. A ranking score computed when 5% of traffic is machine means something different from the same score at 40%. Recording the mix dates the baseline, shows how much of demand its behavioral evidence still represents, and lets human and machine performance be reported separately as agent share grows.
Read the original version of this article on VKTR.
Frequently Asked Questions
What is an instrumented search baseline?
An instrumented search baseline is a measurement layer, built before any optimization begins, that establishes where search performance stands, monitors it continuously, and reports it as a composite score executives can track. It draws on query logs, click and conversion events, and catalog metadata that most enterprises already hold in existing analytics and commerce platforms, so the work is extraction and access rather than new instrumentation.
Why should search be measured before it is tuned?
Without a baseline recorded before any changes, a team cannot show whether new synonyms, weighting adjustments, or a vendor replacement improved anything, because instruments built by the project itself tend to display gains rather than test them. Locking the baseline first gives executives a number that predates the work, which is the kind of evidence a budget review will accept and fund.
Why does the independence of the measurement source matter to executives?
A figure drawn from systems that existed before an initiative cannot be accused of flattering it. In one technical manufacturing enterprise, the reduction in engineer search time came from existing time tracking and resource management systems, so the result, millions of dollars a year in recovered capacity, was believed, funded, and sustained. A yardstick the project invented for itself invites the argument that the gain was built into the measure.
Why can a search program score well on ranking quality and still lose sales?
Ranking quality grades only the order of the results that came back. A product missing a required attribute, or classified inconsistently, never enters the result set, so it never registers as a ranking miss and no abandoned session points to it. Reading ranking quality beside zero-result tracking by category and catalog coverage exposes the products that are eligible in inventory but invisible to search.
Why should a baseline record the human to machine traffic mix on the day it is taken?
Behavioral metrics such as click-based ranking quality describe human shoppers, and AI agents click nothing. A ranking score computed when 5% of traffic is machine means something different from the same score at 40%. Recording the mix dates the baseline, shows how much of demand its behavioral evidence still represents, and lets human and machine performance be reported separately as agent share grows.
Frequently Asked Questions
What is an instrumented search baseline?
An instrumented search baseline is a measurement layer, built before any optimization begins, that establishes where search performance stands, monitors it continuously, and reports it as a composite score executives can track. It draws on query logs, click and conversion events, and catalog metadata that most enterprises already hold in existing analytics and commerce platforms, so the work is extraction and access rather than new instrumentation.
Why should search be measured before it is tuned?
Without a baseline recorded before any changes, a team cannot show whether new synonyms, weighting adjustments, or a vendor replacement improved anything, because instruments built by the project itself tend to display gains rather than test them. Locking the baseline first gives executives a number that predates the work, which is the kind of evidence a budget review will accept and fund.
Why does the independence of the measurement source matter to executives?
A figure drawn from systems that existed before an initiative cannot be accused of flattering it. In one technical manufacturing enterprise, the reduction in engineer search time came from existing time tracking and resource management systems, so the result, millions of dollars a year in recovered capacity, was believed, funded, and sustained. A yardstick the project invented for itself invites the argument that the gain was built into the measure.
Why can a search program score well on ranking quality and still lose sales?
Ranking quality grades only the order of the results that came back. A product missing a required attribute, or classified inconsistently, never enters the result set, so it never registers as a ranking miss and no abandoned session points to it. Reading ranking quality beside zero-result tracking by category and catalog coverage exposes the products that are eligible in inventory but invisible to search.
Why should a baseline record the human to machine traffic mix on the day it is taken?
Behavioral metrics such as click-based ranking quality describe human shoppers, and AI agents click nothing. A ranking score computed when 5% of traffic is machine means something different from the same score at 40%. Recording the mix dates the baseline, shows how much of demand its behavioral evidence still represents, and lets human and machine performance be reported separately as agent share grows.