Welcome to the new Evident.io Support Portal. Our goal was to make it easier to navigate, so please let us know if you have any questions or concerns. We have migrated the ticket content from the old portal, but you may need to re-register to log in and view your tickets.

AWS rate-limit exceeded exceptions

Problem

AWS API rate limit exceeded, causing rate-limit exceptions to be thrown by both ESP and other tools using AWS API.

Symptoms

  1. Reports contain error alerts with the following message:
    "message" => "The signature is being rate-limited by AWS.  Please stagger API usage so the signature is not running concurrently with other processes on the same service or request an API rate limit increase from AWS for the impacted service."
  2. Other tools reports RequestLimitExceeded exceptions when making AWS API calls.

Cause

ESP can potentially generate a large number of AWS API calls during a scan cycle.  If AWS's rate limit is reached, these API requests may result in rate-limit exceeded exceptions for both ESP and other tools.

Resolution

READ FIRST: Evident.io is aware that ESP generates a large number of AWS API calls, and it is a problem that's being taken very seriously.  Engineering is working towards a long-term solution, but in the meantime, please consider the recommendations below to alleviate this issue:

  1. Increase the scan interval for specific services to lower the overall number of API calls.  You can configure the scan interval from ESP Web, go to Control Panel -> External Accounts -> Scan Interval.  Increasing the scan interval will reduce the overall number of API calls, but this will not reduce the number of API calls made per scan cycle.
  2. Disable signatures that you do not need.  You can disable signatures by going to Control Panel -> Disabled Signatures (https://esp.evident.io/control_panel/disabled_signatures).  ESP will not run any disabled signatures; however, do note that existing alerts may take up to 27 hours to end.
  3. Reduce the number of AWS assets.  For certain services, the number of assets (e.g. EC2 instances, EBS Volumes) is proportional to the number of API calls that ESP makes.  Cleaning up unused assets can lower the number of API calls made per scan cycle.
  4. Stagger your other API-heavy workloads so they run at different times from your ESP scans.
  5. Contact AWS to increase the rate limit for you account.

It is recommended to first identify the AWS service(s) that is exceeding the rate limit, and then perform tuning to target those services.  For example, if the EC2 rate limit is being reached, disabling signature the AWS:EC2-001 is a good first step towards reducing API calls for that service.

  • 66
  • 12-Dec-2017
  • 44 Views