Automating AWS with boto3 · Prasanta Kumar Behera

Automate AWS with boto3: sessions, profiles and regions, how credentials are found (including roles for EC2 and EKS pods), paginators for complete results, waiters, error handling, and least-privilege IAM for your scripts.

boto3 in one picture

boto3 is the AWS SDK for Python. Every AWS service has a client whose methods map one-to-one to API actions (describe_instances, list_clusters, put_object…). The same credentials and regions you use with the aws CLI work here.

AWS is a giant office building with a different department for everything: computers (EC2), storage (S3), Kubernetes (EKS). boto3 is your phone with every department on speed dial. Before anyone answers, they check your badge (credentials) and which city branch (region) you're calling.

$ pip install boto3

Sessions, profiles and regions

import boto3

session = boto3.Session(profile_name="dev", region_name="eu-west-1")   # or rely on AWS_PROFILE / AWS_REGION
sts = session.client("sts")
me = sts.get_caller_identity()
print(f"account={me['Account']} arn={me['Arn']}")

Log the identity at start-up. It removes all doubt about which account a script is about to change.

How boto3 finds credentials

boto3 checks, in order (simplified): explicit parameters → environment variables → shared config files (~/.aws/credentials, ~/.aws/config, including SSO profiles) → container or pod credentials (EKS Pod Identity, IRSA via web identity) → EC2 instance role.

No long-lived keys in code

Never put access keys in scripts, repositories or images. On laptops use SSO profiles; on EC2 use instance roles; on EKS use Pod Identity or IRSA (see Amazon EKS in Production). All give short-lived credentials automatically.

Paginators: get everything

ec2 = session.client("ec2")
paginator = ec2.get_paginator("describe_instances")
pages = paginator.paginate(Filters=[{"Name": "instance-state-name", "Values": ["running"]}])

for page in pages:
    for reservation in page["Reservations"]:
        for inst in reservation["Instances"]:
            tags = {t["Key"]: t["Value"] for t in inst.get("Tags", [])}
            print(inst["InstanceId"], inst["InstanceType"], tags.get("Name", "-"), tags.get("owner", "MISSING"))

Waiters: wait for a state

ec2.stop_instances(InstanceIds=["i-0123456789abcdef0"])
ec2.get_waiter("instance_stopped").wait(
    InstanceIds=["i-0123456789abcdef0"],
    WaiterConfig={"Delay": 10, "MaxAttempts": 30},     # poll every 10 s, give up after ~5 min
)

Handle errors by code

from botocore.exceptions import ClientError

s3 = session.client("s3")
try:
    s3.head_bucket(Bucket="my-terraform-state")
except ClientError as e:
    code = e.response["Error"]["Code"]
    if code in ("404", "NoSuchBucket"):
        print("bucket does not exist")
    elif code in ("403", "AccessDenied"):
        print("bucket exists but we can't access it (or it belongs to another account)")
    else:
        raise

A useful example: an EKS inventory

eks = session.client("eks")
for name in eks.list_clusters()["clusters"]:          # (use a paginator for many clusters)
    c = eks.describe_cluster(name=name)["cluster"]
    print(f"{name:<20} version={c['version']} status={c['status']} endpoint_public={c['resourcesVpcConfig']['endpointPublicAccess']}")

Least privilege for scripts

Give the role your script uses only what it calls. For the read-only inventory above:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["ec2:DescribeInstances", "eks:ListClusters", "eks:DescribeCluster"],
      "Resource": "*"
    }
  ]
}

(Describe/List actions often require "Resource": "*"; mutating actions should be scoped to specific ARNs or tags.)

Try it: a tagging-compliance report (read-only)

Use a sandbox account or the free tier, with a read-only role.

Print your caller identity and region.
Using a paginator, list all running EC2 instances in the region with Name and owner tags; mark instances missing owner.
Output the report as CSV (lesson 02) and exit 1 if any instance is non-compliant.
Loop over all regions (ec2.describe_regions()), creating a client per region, and summarise counts per region.
Write the minimal IAM policy the script needs, and test it: does anything fail with AccessDenied?

Going deeper: boto3 in production

Tune retries and timeouts with botocore.config.Config(retries={"max_attempts": 10, "mode": "standard"}, connect_timeout=5, read_timeout=30).
Prefer clients over the older "resource" interface for new code; clients cover every API and are actively extended.
Mock AWS in tests with moto (lesson 08), so tests never touch a real account.
For provisioning, prefer Terraform (declarative, with plans and state); use boto3 for reports, glue and operational automation (see Terraform & Infrastructure as Code).

Recap

boto3.Session(profile, region) → session.client("service"); log sts.get_caller_identity().
Credentials come from the chain: SSO profiles on laptops, roles on EC2 and EKS. Never hard-coded keys.
Paginators for complete results, waiters for states, ClientError codes for handling.
Least-privilege IAM for every script.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.