Last Updated on April 3, 2021

When working to programmatically check or configure your AWS infrastructure in Python, we need to use Boto3. Most of the time we will need to check the output of the Boto3 Client by printing it to the terminal. Unfortunately, printing directly the boto3 response is not really easy to understand.

The best way to work with the boto3 response is to convert it to a JSON string before printing it to the terminal.

---

• Printing Boto3 Client response
• Converting Boto3 Response to JSON String
• Fixing TypeError: Object of type datetime is not JSON serializable
• Advantage of printing JSON Strings in Lambda functions and CloudWatch Logs

Printing Boto3 Client response

To see how we can output boto3 client’s response to the terminal we will initially use the describe_regions function of the boto3 EC2 client.

See the Python script below.

import boto3

client = boto3.client('ec2')

response = client.describe_regions(RegionNames=['us-east-1'])

print(response)

Running the Python script will result in the long single line output below.

{'Regions': [{'Endpoint': 'ec2.us-east-1.amazonaws.com', 'RegionName': 'us-east-1', 'OptInStatus': 'opt-in-not-required'}], 'ResponseMetadata': {'RequestId': '9437271e-6132-468f-b19d-535f9d7bda09', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '9437271e-6132-468f-b19d-535f9d7bda09', 'cache-control': 'no-cache, no-store', 'strict-transport-security': 'max-age=31536000; includeSubDomains', 'content-type': 'text/xml;charset=UTF-8', 'content-length': '449', 'date': 'Sat, 03 Apr 2021 08:30:15 GMT', 'server': 'AmazonEC2'}, 'RetryAttempts': 0}}

In my screen it will look like this since it automatically wraps the output.

With the output above, it is hard to understand or search for the specific part of the response that you are looking for.

Note: The response object of the boto3 client is a dictionary (dict) type. The examples below will work even if you change response object to any dictionary object.

---

Converting Boto3 Response to JSON String

To easier understand the returned response of the boto3 client it is best to convert it to a JSON string before printing it. To change the response to a JSON string use the function json.dumps.

import boto3
import json

client = boto3.client('ec2')

response = client.describe_regions(RegionNames=['us-east-1'])

json_string = json.dumps(response, indent=2)

print(json_string)

This will result in an output that is easier to understand format. The output when I ran the Python script above can be seen below.

{
  "Regions": [
    {
      "Endpoint": "ec2.us-east-1.amazonaws.com",
      "RegionName": "us-east-1",
      "OptInStatus": "opt-in-not-required"
    }
  ],
  "ResponseMetadata": {
    "RequestId": "f6a33cc5-58c7-4948-8f44-2769ede5f166",
    "HTTPStatusCode": 200,
    "HTTPHeaders": {
      "x-amzn-requestid": "f6a33cc5-58c7-4948-8f44-2769ede5f166",
      "cache-control": "no-cache, no-store",
      "strict-transport-security": "max-age=31536000; includeSubDomains",
      "content-type": "text/xml;charset=UTF-8",
      "content-length": "449",
      "date": "Mon, 29 Mar 2021 12:11:52 GMT",
      "server": "AmazonEC2"
    },
    "RetryAttempts": 0
  }
}

In my screen it will look like this.

That is a much easier to understand format.

The difference when converting the boto3 client response to JSON string before printing it are the following.

• The single quotes (') are now double quotes ("). This is because json.dumps converts Python objects to JSON strings.
• The output is now putting each key-value pair of the response dictionary in a newline. This is because of the indent parameter in json.dumps. If you remove the indent parameter the output will be printed in a single line again.

We can also shorten the code by directly putting json.dumps inside the print function.

import boto3
import json

client = boto3.client('ec2')

response = client.describe_regions(RegionNames=['us-east-1'])

print(json.dumps(response, indent=2))

The output will be the same as the one above that is easier to understand.

Note: The indent parameter gets a positive integer to know how many spaces it will indent to pretty-print the JSON string. You can increase it to indent=4 if you find indent=2 not noticeable.

---

Fixing TypeError: Object of type datetime is not JSON serializable

In the example below, we are trying to print the response of EC2 client’s describe_instances function in JSON format. This will result in a Object of type datetime is not JSON serializable error.

import boto3
import json

client = boto3.client('ec2')

response = client.describe_instances()

print(json.dumps(response, indent=2))

Error Output

Traceback (most recent call last):
  File "boto3_ec2_json_bad.py", line 8, in <module>
    json_string = json.dumps(response, indent=2)
  File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\__init__.py", line 234, in dumps
    return cls(
  File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\encoder.py", line 201, in encode
    chunks = list(chunks)
  File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\encoder.py", line 325, in _iterencode_list
    yield from chunks
  File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\encoder.py", line 325, in _iterencode_list
    yield from chunks
  File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\encoder.py", line 438, in _iterencode
    o = _default(o)
  File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type datetime is not JSON serializable

The reason for this error is that there are values in the describe_instances response that is in the type datetime, like the LaunchTime key of each EC2 Instances.

There are a lot of datetime objects not only in the EC2 client, but also for other services like DynamoDB, S3, Lambda and many more.

To fix the TypeError: Object of type datetime is not JSON serializable, we will just add the default=str parameter when calling json.dumps function.

Below is the corrected Python script.

import boto3
import json

client = boto3.client('ec2')

response = client.describe_instances()

print(json.dumps(response, indent=2, default=str))

Output

{
  "Reservations": [
    {
      "Groups": [],
      "Instances": [
        {
          "AmiLaunchIndex": 0,
          "ImageId": "ami-016aa01b240468f0c",
          "InstanceId": "i-05b2515c1f1d70195",
          "InstanceType": "t4g.nano",
          "LaunchTime": "2021-03-29 11:46:29+00:00",
          "Monitoring": {
            "State": "disabled"
          },
          "Placement": {
            "AvailabilityZone": "ap-southeast-1a",
            "GroupName": "",
            "Tenancy": "default"
          },
          "PrivateDnsName": "ip-172-31-42-147.ap-southeast-1.compute.internal",
          "PrivateIpAddress": "172.31.42.147",
          "ProductCodes": [],
          "PublicDnsName": "",
          "State": {
            "Code": 80,
            "Name": "stopped"
          },
          "StateTransitionReason": "User initiated (2021-03-29 11:48:01 GMT)",
          "SubnetId": "subnet-3f271e76",
          "VpcId": "vpc-8f3d19e8",
          "Architecture": "arm64",
          "BlockDeviceMappings": [
            {
              "DeviceName": "/dev/xvda",
              "Ebs": {
                "AttachTime": "2021-03-29 11:34:20+00:00",
                "DeleteOnTermination": true,
                "Status": "attached",
                "VolumeId": "vol-00938286e87553800"
              }
            }
          ],
          "ClientToken": "",
          "EbsOptimized": true,
          "EnaSupport": true,
          "Hypervisor": "xen",
          "IamInstanceProfile": {
            "Arn": "arn:aws:iam::758357942546:instance-profile/ec2_ssm_role",
            "Id": "AIPA2X6MXHWNVRQZPJB5C"
          },
          "NetworkInterfaces": [
            {
              "Attachment": {
                "AttachTime": "2021-03-29 11:34:20+00:00",
                "AttachmentId": "eni-attach-09421875a6b55f2b9",
                "DeleteOnTermination": true,
                "DeviceIndex": 0,
                "Status": "attached"
              },
              "Description": "",
              "Groups": [
                {
                  "GroupName": "TestSecurityGroup",
                  "GroupId": "sg-0779dca17c74a411e"
                }
              ],
              "Ipv6Addresses": [],
              "MacAddress": "06:f1:78:c4:ef:c0",
              "NetworkInterfaceId": "eni-0cefe51088fc1536a",
              "OwnerId": "758357942546",
              "PrivateDnsName": "ip-172-31-42-147.ap-southeast-1.compute.internal",
              "PrivateIpAddress": "172.31.42.147",
              "PrivateIpAddresses": [
                {
                  "Primary": true,
                  "PrivateDnsName": "ip-172-31-42-147.ap-southeast-1.compute.internal",
                  "PrivateIpAddress": "172.31.42.147"
                }
              ],
              "SourceDestCheck": true,
              "Status": "in-use",
              "SubnetId": "subnet-3f271e76",
              "VpcId": "vpc-8f3d19e8",
              "InterfaceType": "interface"
            }
          ],
          "RootDeviceName": "/dev/xvda",
          "RootDeviceType": "ebs",
          "SecurityGroups": [
            {
              "GroupName": "OpenVPNAcces",
              "GroupId": "sg-0779dca17c74a411e"
            }
          ],
          "SourceDestCheck": true,
          "StateReason": {
            "Code": "Client.UserInitiatedShutdown",
            "Message": "Client.UserInitiatedShutdown: User initiated shutdown"
          },
          "Tags": [
            {
              "Key": "Name",
              "Value": "Test-Instance"
            }
          ],
          "VirtualizationType": "hvm",
          "CpuOptions": {
            "CoreCount": 2,
            "ThreadsPerCore": 1
          },
          "CapacityReservationSpecification": {
            "CapacityReservationPreference": "open"
          },
          "HibernationOptions": {
            "Configured": false
          },
          "MetadataOptions": {
            "State": "applied",
            "HttpTokens": "optional",
            "HttpPutResponseHopLimit": 1,
            "HttpEndpoint": "enabled"
          },
          "EnclaveOptions": {
            "Enabled": false
          },
          "BootMode": "uefi"
        }
      ],
      "OwnerId": "758357942546",
      "ReservationId": "r-0533aa0fa263f1372"
    }
  ],
  "ResponseMetadata": {
    "RequestId": "977f1891-045c-4b73-85d7-f47a2700c0bd",
    "HTTPStatusCode": 200,
    "HTTPHeaders": {
      "x-amzn-requestid": "977f1891-045c-4b73-85d7-f47a2700c0bd",
      "cache-control": "no-cache, no-store",
      "strict-transport-security": "max-age=31536000; includeSubDomains",
      "content-type": "text/xml;charset=UTF-8",
      "content-length": "6928",
      "vary": "accept-encoding",
      "date": "Mon, 29 Mar 2021 12:17:31 GMT",
      "server": "AmazonEC2"
    },
    "RetryAttempts": 0
  }
}

The value of the default parameter in json.dumps should be a function that will convert a non-JSON serializable object, like datetime, into a JSON encodable version of the object.

Since Strings are JSON serializable, the str() function will be a perfect to convert almost all object types to Strings in Python.

---

Advantage of printing JSON Strings in Lambda functions and CloudWatch Logs

When developing Lambda Functions, an advantage of converting the boto3 response to JSON and printing it is that CloudWatch Logs automatically adjusts the print to an easier to read format.

On the code below, the first print will simply print the response of the EC2 client’s describe_regions function.

The second print will convert the response to JSON using json.dumps then print it. Notice that in this print, I am not using the indent=2 parameter so it will print the JSON string in one line only. I am only using the parameter default=str to make sure if a datetime object appears it will still be converted to JSON properly.

import boto3
import json

def lambda_handler(event, context):
    
    client = boto3.client('ec2')
    
    response = client.describe_regions()
    
    print(response)
    
    print(json.dumps(response, default=str))

On the screenshot of the Lambda function’s CloudWatch Logs below, we can see that even if I did not add the indent=2 parameter in json.dumps, CloudWatch Logs will automatically format the JSON string to an easier to understand layout.

Ever since I noticed this, I have been printing the logs of my Lambda Functions in JSON format rather than normal print or PrettyPrinter (pprint).

---

We hope this helps you develop Python scripts easier when working with AWS Boto3 by converting the response dict to JSON strings.