On this page
Rate limiting is a resource-allocation policy, not just a counter attached to an endpoint. An API can accept a modest number of requests while still exhausting its database, worker pool, or AI budget. The important question is how much work each caller can cause, including work that continues after the original request has finished.
This guide develops a limit design for a service that creates document summaries and exports. It uses illustrative policies rather than recommended universal thresholds. Your actual limits should come from legitimate usage, dependency capacity, business requirements, and controlled testing. A copied number with no workload context is difficult to defend or maintain.
Identify the resource you are protecting
The OWASP API4:2023 resource-consumption guidance highlights risks involving execution time, memory, upload size, operation counts, pagination, and third-party spending. This is a useful reminder that request frequency is only one dimension of consumption. A control should correspond to the resource that can actually be exhausted.
For the example service, list CPU time, model tokens, export storage, concurrent jobs, and database reads separately. Identify which are shared globally and which can be attributed to a tenant. Then select a measurement and enforcement point for each. A public edge can count requests, while a worker may be better placed to enforce the remaining time or document budget of a long-running job.
Choose an identity that supports fairness
An IP address can be useful as one signal, but it may group unrelated users behind the same network or change as a client moves. Authenticated tenant, user, integration, and workload identities can support different allocation policies. Choose the identity according to the boundary you want to protect, and use layered controls where the risks differ.
For summary generation, give each tenant a capacity allowance and each user a smaller concurrency allowance. Maintain a separate service-wide ceiling so many tenants cannot collectively overload the provider. Do not let a caller choose its own trusted quota identity in a request body. Derive the identity from verified context, and define how anonymous requests are handled before authentication has established a more useful principal.
Distinguish bursts, sustained use, and concurrency
A burst allowance answers whether a client can make several requests close together. A sustained limit answers how much use is permitted over time. A concurrency limit answers how many operations may remain active at once. These controls can produce very different behavior even when their numbers look similar.
A document export may be infrequent but long-running. Limiting starts per minute does not prevent too many exports from overlapping. In the example design, a job occupies a concurrency slot until it completes, fails, or reaches a bounded lease expiration. Use representative workloads to choose the combination. Document the experience of a legitimate user who submits a normal batch, so the policy does not protect capacity by making the intended product unusable.
Budget expensive requests before accepting work
Estimate the work a request can cause before committing to it. For a summary job, that estimate might use document size, permitted output length, and a bounded number of retrieval or model steps. Reserve an allowance, execute within it, and reconcile actual usage afterward. Keep estimates and measured consumption separate so operators understand the difference.
A reservation system needs a failure plan. If a worker crashes, its reservation should not remain active forever. Use explicit job states and bounded leases, then reconcile abandoned work according to the capabilities of your provider. Avoid promising exact cost control when downstream consumption can continue after cancellation. Define what the application can stop, what it can measure later, and what exposure remains during those intervals.
Control amplification inside a single request
Look for parameters that multiply work: array length, nested selections, export ranges, document count, model attempts, and tool recursion. A request-count limit will not notice that one accepted call asks the backend to perform hundreds of operations. Bound those multipliers where they are interpreted, not only where the request first arrives.
For the example summary service, cap the authorized document set, restrict output length, and limit retries across the whole task. A retry at one layer should not quietly reset the budget at another. Propagate a trusted task identifier and remaining allowance through the workflow. When integrating tools, decide whether a proposed call fits the remaining budget before execution. Do not let the model itself redefine the maximum number of steps.
Design client behavior for rejection and recovery
A rejected request should give the client enough information to respond sensibly without exposing internal capacity details. Distinguish a temporary throttle from a permanently oversized request or an exhausted longer-term allowance. Document supported retry guidance and ensure official client examples use bounded backoff rather than immediate, endless retries.
For asynchronous operations, idempotency can help prevent repeated submissions from creating duplicate work when the client is uncertain whether the first attempt succeeded. Its implementation must define the key scope, retention window, and behavior when the same key is reused with different input. Do not claim that an idempotency header alone prevents duplicates. Test the state transitions and concurrent submissions that determine the actual result in your service.
Account for distributed state and outages
A limiter spread across multiple regions may not observe every request at the same instant. Decide which limits require strongly coordinated state and which tolerate some temporary overshoot. Keep the tradeoff visible in the design. A hard spending promise is difficult to support if several workers can independently reserve the same remaining allowance.
Plan for the quota store becoming slow or unavailable. High-cost operations may need to stop when their budget cannot be verified. Low-cost public reads may use a bounded local fallback, depending on the risk. Document the fallback ceiling and how it interacts with global protection. An outage policy that silently removes all controls is not a fallback budget. Test recovery too, including stale leases and duplicated completion events.
Evaluate fairness and false rejection
Measure successful legitimate tasks, rejection reasons, active work, queue age, and resource consumption by the identities relevant to your policy. A falling request count can mean better control, broken clients, or frustrated users. Pair the operational dashboard with representative task tests and a review of affected tenants.
Run scenarios for a normal burst, a large but valid export, a slow provider, a worker crash, and many clients retrying after an outage. Use synthetic data and operate only in environments you control or are authorized to test. Record the configuration and stop conditions before the test. The objective is to discover policy weaknesses without turning a capacity exercise into an uncontrolled production incident.
Keep overrides narrow and reviewable
Some customers or internal jobs may need a temporary higher allowance. Make overrides explicit by identity, operation, duration, and owner. Keep the global safety ceiling independent where possible. Review whether the exception changes provider capacity or simply reallocates an existing budget. An unlimited override is difficult to reason about, especially when the workflow includes model-generated tool calls or automatic retries that may expand after a future release.
Conclusion: protect the whole unit of work
Effective resource controls connect request admission with the work that follows. Use the right identities, distinguish frequency from concurrency, bound amplification, and define how reservations behave during failure. Measure the policy against legitimate tasks rather than treating a high rejection count as success.
Explore the AI shield API guide to place budgets inside an inference pipeline. Use the API firewall shield overview to separate early traffic controls from deeper application limits. The strongest policy is one that protects shared capacity while remaining understandable to the people and clients who depend on it.
