SDK & Client-Side Throttling

A well-behaved client never discovers a rate limit by being rejected by it. It knows the ceiling it was given, paces itself below it, and treats a 429 as evidence of a bug in its own scheduling rather than as normal operation — the discipline that turns the reactive patterns in the Frontend Resilience & UX Handling reference into a proactive one. Client-side throttling is the mirror image of server-side enforcement: the same token bucket arithmetic, but running in a browser tab or an SDK process, deciding when to send rather than whether to admit.

The payoff is concrete. A server-side limit rejects the 101st request and the client retries it, so the work is done twice and the user waits for both attempts. A client-side limit delays the 101st request by 600 ms and it succeeds first time. Same throughput ceiling, one round trip instead of two, and no error path in the user interface. The cost is that the client must hold state that stays correct across tabs, retries, and process restarts.

Mechanism: three cooperating limits

A client SDK that behaves well under load enforces three separate things, and conflating them is the usual source of bugs.

Rate — requests per unit time, mirroring the server’s advertised limit. Implemented as a local token bucket that refills at the documented rate, sized slightly below the server’s number so that clock differences and in-flight requests do not push you over. Ninety per cent of the advertised limit is a sensible default.

Concurrency — how many requests may be in flight at once, which is a separate resource from rate. A browser will happily open six connections per origin over HTTP/1.1 and effectively unlimited streams over HTTP/2; a Node process with a connection pool of 50 can saturate a server without ever exceeding a per-second rate. Bound it explicitly with a semaphore.

Adaptation — what the client does when the server disagrees with its model. A 429 means the local model is wrong: the limit changed, another process shares the key, or a retry storm is under way. The correct response is to shrink the local rate immediately, honour Retry-After, and probe back upward slowly.

The combination of multiplicative decrease and additive increase is the same control law that keeps TCP congestion stable, and for the same reason: it converges quickly when the ceiling drops and probes gently when headroom returns.

Configuration reference

maxQueueWaitMs is the option people omit and regret. Without a time bound, a queue that grows during a slow period releases requests minutes later, against a user interface that has moved on — the request succeeds, the response is discarded, and the quota is spent for nothing.

Implementation walkthrough

A compact scheduler that enforces all three controls and adapts on rejection:

// throttle.ts — client-side pacing: rate + concurrency + adaptive shrink.
interface Options {
  requestsPerSecond: number;
  burst?: number;
  maxConcurrent?: number;
  maxQueue?: number;
  maxQueueWaitMs?: number;
}

export class ClientThrottle {
  private tokens: number;
  private lastRefill = Date.now();
  private inFlight = 0;
  private queue: Array<{ run: () => void; enqueuedAt: number }> = [];
  private baseRate: number;
  private rate: number;
  private pausedUntil = 0;

  constructor(private opts: Options) {
    this.baseRate = opts.requestsPerSecond;
    this.rate = opts.requestsPerSecond;
    this.tokens = opts.burst ?? opts.requestsPerSecond;
  }

  private refill() {
    const now = Date.now();
    const capacity = this.opts.burst ?? this.baseRate;
    this.tokens = Math.min(capacity, this.tokens + ((now - this.lastRefill) / 1000) * this.rate);
    this.lastRefill = now;
  }

  /** Wrap any request function; resolves when it has been sent and answered. */
  async schedule<T>(fn: () => Promise<T>): Promise<T> {
    const maxQueue = this.opts.maxQueue ?? (this.opts.maxConcurrent ?? 6) * 2;
    if (this.queue.length >= maxQueue) {
      throw new Error("client_queue_full");     // fail fast, do not grow without bound
    }
    await new Promise<void>((resolve) => {
      this.queue.push({ run: resolve, enqueuedAt: Date.now() });
      this.pump();
    });
    this.inFlight += 1;
    try {
      return await fn();
    } finally {
      this.inFlight -= 1;
      this.pump();
    }
  }

  /** Called by the transport when the server rejects: the local model was wrong. */
  onRejected(retryAfterSeconds?: number) {
    this.rate = Math.max(this.baseRate * 0.1, this.rate * 0.5);   // multiplicative decrease
    if (retryAfterSeconds && retryAfterSeconds > 0) {
      this.pausedUntil = Date.now() + retryAfterSeconds * 1000;   // obey the server
      this.tokens = 0;
    }
  }

  /** Called after a clean window: probe back toward the advertised rate. */
  onCleanWindow() {
    this.rate = Math.min(this.baseRate, this.rate * 1.1);          // gentle increase
  }

  private pump() {
    const maxConcurrent = this.opts.maxConcurrent ?? 6;
    const waitBound = this.opts.maxQueueWaitMs ?? 30_000;
    const now = Date.now();
    if (now < this.pausedUntil) {
      setTimeout(() => this.pump(), this.pausedUntil - now);
      return;
    }
    this.refill();
    while (this.queue.length && this.tokens >= 1 && this.inFlight < maxConcurrent) {
      const item = this.queue.shift()!;
      if (now - item.enqueuedAt > waitBound) continue;             // drop stale work
      this.tokens -= 1;
      item.run();
    }
    if (this.queue.length) {
      const waitMs = Math.max(20, ((1 - this.tokens) / this.rate) * 1000);
      setTimeout(() => this.pump(), waitMs);
    }
  }
}

Wiring it into a transport keeps the adaptation loop closed — every 429 teaches the scheduler, every clean window lets it recover:

// transport.ts — one place where the throttle learns from the server.
const throttle = new ClientThrottle({ requestsPerSecond: 90, burst: 20, maxConcurrent: 6 });
let cleanSince = Date.now();

export async function apiFetch(input: RequestInfo, init?: RequestInit): Promise<Response> {
  return throttle.schedule(async () => {
    const res = await fetch(input, init);
    if (res.status === 429) {
      const retryAfter = Number(res.headers.get("Retry-After") ?? "1");
      throttle.onRejected(Number.isFinite(retryAfter) ? retryAfter : 1);
      cleanSince = Date.now();
      return res;                                   // the retry layer above decides what to do
    }
    // Trust the server's own accounting when it publishes it.
    const remaining = Number(res.headers.get("X-RateLimit-Remaining") ?? NaN);
    if (Number.isFinite(remaining) && remaining < 5) throttle.onRejected();
    if (Date.now() - cleanSince > 10_000) { throttle.onCleanWindow(); cleanSince = Date.now(); }
    return res;
  });
}

Reading X-RateLimit-Remaining is what turns a blind pacer into an informed one: the server is publishing exactly how much headroom is left, and a client that slows down at remaining < 5 almost never sees a rejection at all. That header contract is defined in rate-limit response headers, and consuming it is the cheapest resilience win available to an SDK.

Distributed considerations: many tabs, many processes

A browser client is not one client. Five open tabs mean five schedulers, each believing it owns the full budget, and the server sees five times the intended rate — the same multiplication problem that per-instance proxy counters cause at the edge, moved to the user’s laptop.

Three coordination options, in descending order of fidelity. A SharedWorker or service worker owns one scheduler for the origin and every tab posts requests to it; this is exact but unavailable in some embedded contexts. A BroadcastChannel leader election picks one tab to hold the bucket and re-elects when it closes; nearly exact, with a brief window during handover. A divided budget — each tab takes rate / observedTabs — needs no messaging and degrades gracefully, at the cost of unused headroom when tabs are idle. The persistence details, including surviving a reload, are covered in persisting rate-limit state across tabs.

Server-side SDKs face the same question with a different answer: the coordinating primitive is usually the same Redis the server team already runs, or an explicit per-process share of a documented budget agreed in configuration.

Mobile SDKs add one more wrinkle: the operating system suspends your process, and the scheduler’s notion of elapsed time comes back wrong or the queued work resumes all at once. Refill from a monotonic clock where the platform offers one, clamp negative elapsed time to zero, and re-check the queue’s staleness bound on resume so a batch queued before a two-hour suspension is discarded rather than fired in a single burst the moment the app foregrounds.

Failure modes and mitigations

• Local limit above the server’s. Pacing at exactly the advertised rate guarantees occasional rejection, because network jitter bunches arrivals. Target 90% and let the adaptive loop find the rest.
• Unbounded queue. A scheduler with no maxQueue converts a slow server into an out-of-memory crash, and every queued request eventually arrives too late to matter. Bound both depth and wait time.
• Retrying inside the scheduler and outside it. If the transport retries and a wrapper also retries, one user action becomes nine requests. Retry in exactly one layer — see retry queue implementation.
• Non-idempotent retries. Replaying a POST that already succeeded but whose response was lost creates duplicate resources. Attach an idempotency key to every mutating request before it enters the queue.
• Adaptation with no recovery. Halving the rate on rejection without probing back leaves a long-running process crawling at 10% for the rest of the day. Always pair decrease with increase.
• Ignoring Retry-After. A client that keeps its own schedule after being told to wait 30 seconds will be rejected 30 seconds’ worth of times, and some servers escalate to a longer block.
• One global bucket for many credentials. In a multi-tenant SDK, key the bucket by credential; otherwise one tenant’s batch job throttles every other tenant sharing the process.

Instrumenting the client so throttling is visible

A client-side limiter that works silently is indistinguishable from a slow API — users report “the app feels laggy” and nobody connects it to a scheduler doing exactly what it was told. Emit four numbers from the SDK and the picture becomes obvious.

In a browser, ship those counters to your existing front-end telemetry with the same sampling as performance metrics; in a server SDK, expose them as process metrics so operators can correlate a slow batch job with its own self-imposed pacing. A single dashboard panel showing queue wait next to rejection count answers the two questions support actually receives: “is the API slow?” and “are we being limited?”

Choosing defaults when the server publishes nothing

Plenty of APIs document no limit at all, or document one and enforce another. Defaults then have to be chosen defensively, because the first symptom of getting them wrong is your SDK taking out a third party’s service — and the second is an IP-level block that affects every user of your library.

Start at a deliberately low ceiling (say 5 requests per second, 2 concurrent), instrument, and raise it only with evidence. Prefer per-host buckets so a client talking to several services cannot spend one host’s budget on another. Treat any 429, 503, or Retry-After as authoritative, and treat repeated connection resets as an unstated limit — some gateways drop connections rather than answering with a status code. When the API belongs to a partner, ask for the numbers and put them in configuration rather than in code, so raising them later is a config change rather than a release.

Contract with the server

Client throttling only works when the server publishes numbers to throttle against: a documented steady rate, a documented burst, X-RateLimit-* on every response, and Retry-After on every rejection. Where those exist, the SDK should read them at runtime rather than hardcode them — a limit raised for an enterprise customer should take effect without an SDK release. Where they do not exist, the SDK’s defaults become the contract, and they should be conservative enough that the first customer to hit them does not take out the API for the rest.

Surfacing pacing in the interface

Self-pacing changes what the user sees, and pretending otherwise produces the worst outcome: a button that looks broken because its request is sitting in a queue. Three interface rules keep a throttled client honest.

Distinguish queued from in-flight. A request waiting for a token has not started; a spinner that claims otherwise makes a 2-second wait feel like a stall. Show queue position or a simple “waiting to send” state for anything queued longer than about 400 ms, and switch to the normal loading state when it actually leaves.

Make bulk operations show throughput, not a fake percentage. For a batch of 500 uploads paced at 5 per second, a progress bar plus “about 90 seconds remaining, 5 per second” is accurate and calming; a bar that jumps to 40% and freezes for a minute is neither. The estimate comes free from the scheduler — items remaining divided by the current rate.

Never let pacing hide a real failure. If the queue is draining slowly because the local rate was shrunk by repeated 429s, say so once, plainly, in the same language the exponential backoff UX guide recommends for reactive waits. A user who knows they are being rate limited will wait; a user watching an unexplained slow bar will reload the page and double the load.

The design goal is that a self-pacing client feels slower but predictable rather than fast then broken. That trade is almost always worth making, because unpredictability is what generates support tickets, and because a request that never gets rejected also never needs an error state, a retry, or a recovery path in the interface.

In this section

• Client-Side Token Bucket for SDKs — the pacing primitive, sized against the server’s published limit.
• Request Concurrency Limiting in Browsers — semaphores, HTTP/2 streams, and why rate alone is not enough.
• Idempotency Keys for Safe Retries — making a replayed request harmless.

Finally, ship the pacing behaviour on by default. An SDK whose throttling must be enabled explicitly protects nobody, because the integrations most likely to cause trouble are exactly the ones that never read the configuration section.

The three deep dives below cover the pieces in order: the bucket that decides when to send, the semaphore that decides how many may be in flight, and the idempotency key that makes a retry safe in the first place.

An SDK that paces well is invisible; one that does not becomes the reason a customer’s integration is described as unreliable, regardless of which side actually rejected the request.

Ship the defaults conservatively, read the server’s published numbers at runtime, and let the adaptation loop find the rest. That combination behaves well against APIs that document their limits and safely against those that do not.

Related

• Frontend Resilience & UX Handling — the parent reference for reacting to limits in the interface.
• Exponential Backoff UX — what to do once a rejection has happened anyway.
• Client-Side Rate Limit State — where the shared budget is stored between tabs and reloads.
• Retry-After Parsing — turning the server’s pause instruction into a schedule.
• Token Bucket Implementation — the server-side twin of the client bucket.