Best Practices: Handling for Web Service Throttling#

Understand the Throttling Policy#

We have several tiers of subscriptions as outlined in our throttling advisory; you may need to check your agreement or check in with your sales representative for the most up-to-date rates.

  • Requests can be:

    • GET: Processes one record at a time.

    • POST: Processes up to 100 records in a single request.

  • 429 Responses (Too Many Requests):

    • A 429 response occurs when the request rate exceeds the throttling limit.

    • Throttling limits are observed as “records per second” - a single batch of 100 records is aggregated as 100 records.

    • Clients must implement a mechanism to handle 429s to avoid continuous failures if throughput is too high.

Optimize Your Request Strategy#

  • Single-threaded with batching (recommended):

    • Use a single worker thread to process requests.

    • Batch requests with 100 records per POST for optimal processing (where supported - please refer to documentation for the specific service).

    • If using a single-threaded process, you should generally not run into 429 errors, even if batching 100 records per request.

    • This approach minimizes the likelihood of hitting 429 errors as the server prefers batch processing over individual record requests.

  • Multithreaded (optional for enterprise subscriptions):

    • If higher throughput is needed, consider upgrading to an enterprise plan to unlock higher limits to support multithreaded workloads.

    • Ensure the request rate across threads stays within the throttling limits. Requests are throttled across all requests using your subscription.

  • Manage throughput in queues.

    • If your request volume is difficult to predict due to traffic from multiple applications or servers, you may need to centralize traffic to a queue for optimal performance or assign a certain throughput cap to each server.

Handle 429 Responses Gracefully#

Regardless of testing, real-world scenarios and load shapes are often hard to control and even harder to predict. Applications should always explicitly handle 429 errors and implement appropriate back-off strategies for high-throughput events.

  • Monitor for 429 Errors:

    • If a 429 error occurs, pause processing on the affected thread for a second to wait for the throttle state to clear on our servers.

  • Retry Logic:

    • Implement a back-off/self-throttling strategy:

      • Wait a second before retrying the failed request.

      • If requests are constantly being retried, consider scaling down the number of threads in use or implementing a queue with a controlled dequeue rate.

Test Your Throughput#

Testing your application/process can provide useful insight into how you might optimize batch processing and how often you might observe 429 errors. Riding the upper limit of your throttling limit may result in regular 429 errors, but this can be mitigated by scaling down your application or discussing a higher subscription tier with your sales representative.

  • Conduct internal tests to evaluate your RPS performance:

    • Average RPS: Helps identify if you’re nearing the subscription limit.

      • Consider short-term aggregates in one-to-five second buckets. A sudden spike in requests could cause issues even if the rate over a longer period is low.

    • Max RPS: Indicates if you can scale up or down threads and batch sizes.

      • A low average RPS with a high maximum RPS may indicate that you can scale up but need to control short-term bursts or handle occasional 429 errors.

Key Best Practices#

  • Use a single-threaded approach to avoid unnecessary complexity unless your subscription supports heavy multithreaded workloads.

  • Use a multi-threaded approach if your license supports high RPS workloads.

  • Batch your records for efficiency (up to 100 per POST, depending on the service).

    • By design, most users should not run into 429 errors when batching and using a single-thread.

  • Implement robust HTTP 429-handling logic to ensure resilience in throttling scenarios.

Summary#

This approach ensures that clients can process data efficiently while respecting the throttling limits. Handling 429 responses gracefully and optimizing your batching and threading strategies are key to maintaining uninterrupted and performant service.