What happens when you have too many EC2 Images / AMI?

Last Updated on July 19, 2021

I have always been wary of EC2 Images and EBS Snapshots since they are cheap and are usually the last place where people would look to lower their AWS bill.

When I saw one of my clients that has a high cost on EBS Snapshots, I checked if there are EBS Snapshots that are unattached to EC2 Images.

To my surprise, the Python script that I was using threw a Read timeout error.

When I checked where the error was triggered, it turns out that it is the describe_images part of boto3 EC2.

Read timeout error during describe_images.

I quickly checked other places where the EC2 Images / AMIs are listed and saw that the listing of the AMI during EC2 Instance launch and in the AWS Console was also not working.

Error listing AMI / EC2 Images during EC2 Instance creation
Error listing AMI / EC2 Images in the AWS EC2 Console

When I investigated the issue, it turns out describe_images does not support pagination. So calling the AWS API with EC2 DescribeImages will return all the EC2 Images that are currently accessible in the account.

I was able to create a workaround script to list all of the EC2 Images and it turns out it was already more than 200,000 EC2 Images. Most of them were around 3 years old already.

Now imagine there was an issue with an EC2 Instance and we had to launch it from a backup EC2 Image. We will not be able to do so because Describe Images was not working.

In the end, we decided to delete most of the EC2 Images and it went back to normal.

We also deleted the unattached EBS Snapshots and costs.

Leave a Reply

Your email address will not be published. Required fields are marked *