How to measure AI agent provisioning time
Measure AI agent provisioning from the create action to a complete response. Define the test, count failures, and report timings with reproducible evidence.
To measure AI agent provisioning time, define the starting action, choose a useful finish and retain a record of every attempt. This guide provides a reproducible test protocol for that measurement. It does not report cloudrun benchmark results or promise a particular deployment time.
For this benchmark, "ready" means a newly created agent returns a complete response through the normal user interface. A machine identifier, a green infrastructure status or an open chat panel is an intermediate event. Record those events because they explain delays, but continue timing until the user can do something useful.
What should a provisioning timer include?
Start the clock immediately before an authenticated user submits the create action. Stop it when the first complete response appears in the same client. Account setup and billing can be prerequisites, but the reported result must say so. A returning user's deployment time does not describe a new visitor's entire onboarding experience. Our Hermes hosting guide covers the operating steps beyond that first response.
Use a plain prompt with no external tools, such as "Reply with the word ready." Fix the prompt and model before running the experiment. If the model returns an unexpected response, record the outcome and inspect it instead of silently rerunning the attempt.
| Event | What it tells the reader | Counts toward the main duration? |
|---|---|---|
| User submits create | The customer-facing process starts | Yes, this is the start |
| Provider accepts machine request | Infrastructure work has begun | Yes |
| Runtime answers a health request | The service can receive requests | Yes |
| Chat becomes available | The interface can submit a prompt | Yes |
| First output appears | The model has started replying | Yes, record separately |
| Complete response appears | The agent has completed the agreed test | Yes, this is the finish |
Do not stop the main timer at the first output token. Readers should be able to distinguish time to first output from time to a completed answer, especially when the response streams slowly or fails partway through.
Write down the environment before testing
Record the application release, runtime version, image or snapshot identifier, infrastructure provider, region, machine size and model identifier. Include the client's location and browser version. State whether the service uses a prepared image, a warm machine pool or a fresh installation for each attempt.
Those are different test conditions. A prepared image can be a legitimate part of the product, but readers should know that the installation happened before the clock started. When comparing providers, keep the differences between local, VPS and managed hosting separate from the timing result. Likewise, a warm pool result should not be described as a cold machine launch.
Use a dedicated test workspace with the same entitlement and resource limits a customer would receive. Keep real client credentials and data out of the benchmark. Decide the number of attempts and the stop conditions before starting, and budget for the resources the experiment creates.
Keep one timer responsible for the result
Measure the customer-facing interval from one client process. Server events help explain where the time went, but subtracting timestamps from different machines can introduce clock differences into the result.
For a short browser test that stays awake and in one execution context, performance.now() supplies a monotonic duration clock. Unlike a wall-clock timestamp, it is not adjusted when the system clock changes. Keep a separate UTC start time for the evidence record. MDN documents the behavior and sleep limitations of performance.now().
If navigation replaces the page, the instrumentation needs to retain a comparable timing context. Do not subtract two raw performance.now() values from unrelated page contexts. The W3C High Resolution Time specification explains time origins and the limits of comparisons across execution contexts.
An unedited screen recording can support the result, but keep machine-readable event timings too. The recording shows the interface a user saw; the event record makes the calculation inspectable. Neither a promotional animation nor a prerecorded response is suitable evidence.
Record slow attempts and failures
Use a fixed initial sample, such as 30 attempts spread across several sessions. Thirty is a proposed sample size for this experiment, not a claim of statistical certainty. Run a separately labelled concurrent batch if you also want to measure launching several agents at once.
Every attempt needs an outcome, including provider errors, timeouts and incomplete responses. If an attempt times out, retain the timeout duration and failure reason. Do not remove it from the count because it lacks a successful completion time.
Report the total attempts, successful attempts, failures, median successful duration, 95th percentile successful duration and longest successful duration. Label duration statistics as conditional on success. If testing a threshold such as 60 seconds, report the proportion of all attempts that met it. Count failures in the denominator.
For this proposed sample, use the nearest-rank percentile method and state it in the results. With 30 successful observations, the 95th percentile is the 29th sorted observation. That makes the calculation reproducible, while the small sample still limits what readers can infer about rare delays.
Report the result the measurements support
If only the fastest attempt completes within a minute, publish that as the fastest observed attempt. If the median is within a minute, report the median and the spread. Neither result supports an unconditional promise that every deployment finishes in 60 seconds.
Put the measured result beside the sample size, date and tested configuration. A reader should not need to reach the footnotes to discover that billing was already complete, the machines were warm or several attempts failed.
State whether a timing claim describes the fastest attempt, the median, or a percentile. Avoid turning a result from one configuration into a guarantee for every customer. A precise result is more useful than a round number that hides the conditions.
Evaluate setup time on your own workload
Use the same start and finish conditions when evaluating cloudrun. Record time to the first successful task as well as time to a created project, and retain failed attempts in the results.
Review the developer setup path and create a test workspace when you are ready to run the evaluation. This guide supplies the method; it does not report a measured cloudrun result.
What can this benchmark establish?
Does a fast first response prove the agent will stay online?
No. Provisioning speed measures the initial path to a usable agent. Sustained availability, restart behavior, scheduled tasks and recovery require separate tests. This experiment should not make claims about those behaviors.
Does the timer include connecting apps or importing an existing agent?
This proposed test ends after a plain response from a new agent. Connecting apps, migrating files and verifying scheduled work are separate activities. Measuring a complete migration requires timings and checks for that workflow too.
Can a screenshot support a 60-second claim?
A screenshot can show one state at one moment. A reproducible timing claim needs a recorded sequence, explicit start and finish events, the test configuration and the complete set of attempts. Retain those records so another operator can inspect the calculation and repeat the test.