Back to Blog

Why Analysts Spend Half Their Time Waiting for Queries

Why Analysts Spend Half Their Time Waiting for Queries

Query execution time gets most of the attention in analytics tooling. Warehouses are benchmarked against each other on how fast they run complex aggregations. Engineers spend time on query optimization, partition pruning, and materialization strategies to shave seconds off scan times. This is real work with real value.

But the slowest part of the analytics workflow for most analysts is not query execution. It is everything that happens before the query runs: figuring out which tables contain the relevant data, finding the right join keys, translating a business question into a SQL expression that actually captures the intent, and then debugging why the number looks wrong on the first attempt.

This pre-query phase is largely invisible in tooling metrics. Query execution time is easy to measure. Time spent reading documentation, checking schema, talking to colleagues about what a column means, and re-writing a query that returned a plausible-but-wrong result is not. The result is an optimization focus that addresses the visible cost while the invisible cost remains untouched.

Where the time actually goes

A realistic breakdown of an analyst's time on a moderately complex ad-hoc question looks roughly like this: a quarter of the time is spent understanding the question well enough to formulate it as a query. Another quarter is spent on schema exploration: finding the right tables, checking column definitions, validating that join keys are what they appear to be. Another quarter is spent on the first draft of the query and debugging why the initial result is not right. The remaining quarter is actual analysis: looking at the numbers, forming an interpretation, and communicating the finding.

These are rough proportions, not precise measurements, and they vary significantly by analyst experience and by how well-documented a schema is. But the general pattern is consistent: query execution is a small fraction of the total time. The formulation and debugging phases dominate.

Schema exploration as the hidden bottleneck

Schema exploration is the least discussed phase of the analytics workflow. Most data teams do not have complete, up-to-date documentation for every table. Column names are often abbreviated or inherited from legacy systems. A column called ord_tp might mean order type, or it might mean order template, or it might be a field that was relevant three years ago and is now always null. created_at might be the creation timestamp or it might be populated from an API call with a different time zone than the analyst assumes.

Experienced analysts build up a mental model of their warehouse schema over time, but this model is incomplete, partially stale, and non-transferable. When a new analyst joins or when the question touches a part of the schema that the analyst does not regularly use, the schema exploration phase expands. The analyst has to read code, ask colleagues, or run exploratory queries to understand the data before they can write the actual query.

This is the phase where a well-maintained data catalog provides the most value, but data catalogs require ongoing investment to stay current, and the investment often trails the rate at which the underlying schema evolves. The result is that the catalog is authoritative for the well-documented tables and unreliable for the tables that were added recently, modified in a migration, or inherited from an acquisition.

The iteration problem in query writing

SQL queries for business questions are rarely written correctly on the first attempt. The first version runs and produces a number. The analyst then has to validate whether the number is correct. Validation requires either comparing against a known reference, checking the row count to see if it looks plausible, or tracing the query logic manually to confirm it captures the intended business definition.

Common failure modes include: the date filter is off by one due to timezone handling; the aggregation produces duplicate rows because the join cardinality is not what the analyst assumed; the filter on a status field uses a value that is no longer the current valid value; or the metric definition in the query does not match the business definition because the analyst used a shorthand that is slightly wrong.

Each of these failures requires a debugging cycle: identify where the result diverges from expectation, trace it back to the query logic, modify the query, and re-run. For a three-table join with multiple filters and a calculated metric, this cycle can repeat three or four times before the query is right. And each cycle includes a query execution wait, which compounds the time cost even when the warehouse is fast.

The underestimated cost of context switching

The timeline of an ad-hoc analysis is rarely a straight line from question to answer. The analyst gets the request, starts work, hits a blocker (cannot find the right table, does not know how a column is defined), asks a colleague, and waits for a response. The colleague responds the next morning. The analyst context-switches back to the question, re-reads the request, picks up where they left off, runs a test query, discovers another issue.

In practice, a question that involves two hours of focused work may span two to three days of calendar time because of context switching and waiting for input. The warehouse speed is irrelevant to the calendar time; what determines calendar time is the interruption pattern.

This is why reducing query turnaround for business users matters beyond just the time metric. When a business user can get an answer in minutes rather than submitting to a two-day process, decisions happen faster, questions get asked more frequently, and the feedback loop between data and action tightens. The value is not just in analyst hours saved; it is in the organizational metabolism change that comes from faster access to data.

The question formulation asymmetry

There is an asymmetry in how business users and analysts think about questions. A business user's mental model of a question is outcome-oriented: "how many customers churned last month, and which segments had the highest rate?" An analyst's mental model is implementation-oriented: "join the subscriptions table to the churn events table on customer_id, filter for events in the last calendar month, group by segment, calculate the ratio."

The translation between these two representations is not automatic. The business user's question is underspecified: "churned" might mean subscription cancelled, or it might mean no login activity for 30 days, and the business user and analyst might have different definitions in mind. "Segment" might refer to the pricing tier, the acquisition channel, the company size bucket, or some other dimension.

Most analytics workflows do not have a structured translation step between the two representations. The business user submits a question in their natural language. The analyst interprets it, makes assumptions about definitions, and translates it into SQL. If the assumptions are wrong, the answer is wrong, but the wrong answer may not be obvious. The business user sees a number that looks plausible and uses it.

This is the specific failure mode we built TableAI to address from the other direction: instead of asking the business user to learn SQL or waiting for an analyst to mediate every question, the query layer handles the translation. When the translation is ambiguous, the system makes the interpretation visible rather than silent. The goal is to close the gap between the question someone has and the answer they can get, without requiring either the business user to write SQL or the analyst to handle every routine translation.

What the time cost actually implies

The proportion of analyst time spent waiting, searching, and debugging rather than analyzing is not a minor inefficiency. It is the reason that most organizations with data teams feel perpetually understaffed on analytics. Adding more analysts increases raw capacity, but if half of each analyst's time is consumed by query formulation overhead rather than analysis, hiring does not scale as efficiently as it should.

The leverage point is the formulation phase, not the execution phase. Tooling that reduces the time spent getting a question into a runnable form has a larger impact on analytics team throughput than tooling that speeds up query execution, because formulation time is a larger fraction of total time for most analysts on most questions. That is the framing we start from when thinking about where to build.