S3 Operations with Boto3

S3 (Simple Storage Service) is AWS’s object storage. It stores files as objects in buckets with virtually unlimited capacity. This guide covers the most common operations using Python’s boto3 SDK.

Setup

import boto3

s3 = boto3.client("s3")
s3_resource = boto3.resource("s3")

Bucket operations

# Create
s3.create_bucket(
    Bucket="my-app-data",
    CreateBucketConfiguration={"LocationConstraint": "us-west-2"},
)

# List
response = s3.list_buckets()
for bucket in response["Buckets"]:
    print(bucket["Name"])

# Delete (must be empty)
s3.delete_bucket(Bucket="my-app-data")

Upload files

# Upload a file
s3.upload_file("local_file.txt", "my-bucket", "folder/remote_file.txt")

# Upload with metadata
s3.upload_file(
    "report.pdf", "my-bucket", "reports/2026/q1.pdf",
    ExtraArgs={
        "ContentType": "application/pdf",
        "Metadata": {"author": "data-team"},
    }
)

# Upload from memory
s3.put_object(
    Bucket="my-bucket",
    Key="data/config.json",
    Body=json.dumps({"key": "value"}),
    ContentType="application/json",
)

Download files

# Download to file
s3.download_file("my-bucket", "folder/remote_file.txt", "local_copy.txt")

# Download to memory
response = s3.get_object(Bucket="my-bucket", Key="data/config.json")
content = response["Body"].read().decode("utf-8")
data = json.loads(content)

List objects

# List objects with prefix
response = s3.list_objects_v2(Bucket="my-bucket", Prefix="reports/2026/")
for obj in response.get("Contents", []):
    print(f"{obj['Key']} — {obj['Size']} bytes")

# Paginate for large buckets
paginator = s3.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket="my-bucket"):
    for obj in page.get("Contents", []):
        print(obj["Key"])

Delete objects

# Single
s3.delete_object(Bucket="my-bucket", Key="old-file.txt")

# Batch delete
s3.delete_objects(
    Bucket="my-bucket",
    Delete={
        "Objects": [
            {"Key": "file1.txt"},
            {"Key": "file2.txt"},
            {"Key": "file3.txt"},
        ]
    }
)

Presigned URLs

Generate temporary URLs for secure access without AWS credentials.

# Upload URL (PUT)
url = s3.generate_presigned_url(
    "put_object",
    Params={"Bucket": "my-bucket", "Key": "uploads/photo.jpg"},
    ExpiresIn=3600,
)

# Download URL (GET)
url = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": "my-bucket", "Key": "reports/q1.pdf"},
    ExpiresIn=3600,
)
print(url)  # share this URL — valid for 1 hour

Copy and move

# Copy
s3.copy_object(
    CopySource={"Bucket": "source-bucket", "Key": "old/path.txt"},
    Bucket="dest-bucket",
    Key="new/path.txt",
)

# Move (copy + delete)
s3.copy_object(
    CopySource={"Bucket": "my-bucket", "Key": "old.txt"},
    Bucket="my-bucket",
    Key="new.txt",
)
s3.delete_object(Bucket="my-bucket", Key="old.txt")

Multipart uploads

For files larger than 100MB, use multipart uploads.

from boto3.s3.transfer import TransferConfig

config = TransferConfig(
    multipart_threshold=100 * 1024 * 1024,  # 100 MB
    multipart_chunksize=50 * 1024 * 1024,   # 50 MB
    max_concurrency=10,
)

s3.upload_file(
    "large_file.zip", "my-bucket", "backups/large_file.zip",
    Config=config,
)

Summary

S3 is the backbone of AWS storage. Use upload_file/download_file for simple operations, put_object/get_object for in-memory data, presigned URLs for temporary access, and multipart uploads for large files. Paginate when listing large buckets and use lifecycle policies to manage storage costs.