AWS interview questions and answers

Core Service Categories

AWS groups capabilities into specialized domains to address specific architectural needs. Compute services include Amazon EC2 for resizable virtual servers and AWS Lambda for event-driven serverless execution. Storage solutions feature Amazon S3 for durable object storage and Amazon EBS for block volumes attached to instances. Database offerings span relational workloads with Amazon RDS and key-value stores with Amazon DynamoDB. Messaging services are distinct: Amazon SQS handles decoupled queues while Amazon SNS manages pub/sub notifications.

Key Characteristics

• Elasticity: Resources scale automatically based on demand using AWS Auto Scaling or Lambda concurrency controls.

• Global Reach: Infrastructure spans geographic Regions, each containing multiple isolated Availability Zones to ensure high availability and fault tolerance.

• Managed Operations: AWS handles patching, backups, and hardware maintenance for services like Amazon RDS, reducing operational overhead.

• Cost Model: Charges are based on consumption rather than upfront capital expenditure, though exact rates vary by region and change over time.

Security and Identity Best Practices

Security responsibilities are shared between AWS and the customer. Access control utilizes IAM Identity Center for workforce single sign-on and IAM for fine-grained resource permissions. You must follow least privilege by granting only specific actions on required resources within IAM policies. Always use IAM roles for applications running on EC2 or Lambda instead of long-lived access keys, and never use the root account for daily operational tasks. Enable MFA for all human users and encrypt data at rest using AWS KMS.

Accessing AWS via SDK and CLI

Developers interact with AWS using the AWS CLI for administrative operations or SDKs such as boto3 for Python. Infrastructure should be provisioned using AWS CDK, CloudFormation, or Terraform rather than manual console clicks to ensure reproducibility. For example, the CLI command aws s3 ls lists buckets, while equivalent SDK calls rely on the default credential chain. This infrastructure-as-code approach enables version control and automated deployment pipelines.

AWS Regions

An AWS Region is a named geographic area, such as us-east-1 (Northern Virginia) or eu-west-1 (Ireland). Each Region operates independently of others, meaning services and resources in one Region are isolated from those in another by default. This isolation supports data residency requirements and compliance laws, as data stored in a Region generally remains within that geographic boundary unless explicitly transferred. Users select a Region based on latency requirements for end-users, regulatory constraints, and service availability, since not all AWS services are launched in every Region simultaneously. Some specialized infrastructure, such as AWS Local Zones, extends compute capacity closer to specific metropolitan areas but remains associated with a parent Region.

Availability Zones

Within each Region, AWS provisions multiple Availability Zones. An AZ is one or more discrete data center locations equipped with redundant power, networking, and connectivity. They are designed to be insulated from failures in other AZs, such as power outages or natural disasters. Typically, a Region consists of three or more AZs, though this varies by location. The physical distance between AZs within a Region is sufficient to reduce correlated risks but close enough to maintain low-latency network connectivity, usually under a few milliseconds. This design allows synchronous replication between AZs without significant performance degradation.

Comparison and Hierarchy

The hierarchy is strict: AZs exist only within a Region. This structure creates nested failure domains. The following table summarizes the key distinctions:

If an entire Region becomes unavailable, resources in other Regions remain unaffected. Similarly, if one AZ fails, resources in other AZs within the same Region typically remain operational. However, AZ names like us-east-1a are mapped to physical locations differently for each AWS account. To ensure consistent placement across accounts, AWS provides AZ IDs (e.g., use1-az1), which remain constant regardless of the account. Understanding this distinction is critical when designing shared architectures or peering connections.

Deployment Best Practices

• High Availability: Distribute applications across multiple AZs within a single Region using services like Elastic Load Balancing or Amazon RDS Multi-AZ deployments to survive single-AZ failures.

• Disaster Recovery: For resilience against regional outages, replicate critical data and infrastructure across separate Regions, acknowledging the higher latency and cost involved.

• Latency Optimization: Choose the Region geographically closest to your user base to minimize network round-trip time.

• Least Privilege: When configuring IAM policies for cross-Region access, specify exact actions and resource ARNs rather than using wildcards.

• Service Availability: Verify that required services are supported in the target Region before architecting, as feature parity varies.

• AZ Identification: Use AZ IDs instead of AZ names when coordinating infrastructure across multiple AWS accounts to avoid misalignment of physical locations.

What an AMI Contains

• AMI ID: a unique identifier of the form ami-0123456789abcdef0. AMI IDs are region-scoped: the same AMI published in multiple regions has a different ID in each.

• Root volume template: for an EBS-backed AMI, this is an EBS snapshot of the boot volume; for an instance store-backed AMI, the template is stored in Amazon S3 and written to the instance store root volume at launch.

• Block device mapping: declares which volumes attach at launch — the root device plus any additional EBS volumes or ephemeral instance store volumes.

• Launch permissions: control who can use the AMI — implicit for the owning account, explicit for named AWS accounts you share it with, or public for everyone.

• Tags and description: metadata used to track build versions, owners, and compliance state.

EBS-Backed vs Instance Store-Backed AMIs

How AMIs Are Used

1. Choose a base AMI: an AWS-managed AMI such as Amazon Linux 2023, an AWS Marketplace AMI, a community AMI, or one of your own previously built AMIs.

2. Launch and configure a build instance: install packages, bake application code, apply hardening, and install the SSM Agent or CloudWatch agent.

3. Create a “golden” AMI from that instance: EC2 takes a consistent EBS snapshot and registers a new AMI ID.

4. Distribute the AMI: copy it to other regions with copy-image, or share it with other AWS accounts through launch permissions; encrypted AMIs can be shared with a cross-account KMS key policy.

5. Launch a fleet from the AMI: every instance gets identical software, so autoscaling, drift detection, and replacement of unhealthy nodes become routine.

6. Patch by rebuilding, not by editing: the immutable-image pattern is to build a new AMI version in a pipeline and roll instances onto it, rather than mutating running instances.

7. Retire old AMIs: deregister the AMI, then delete the underlying EBS snapshots — deregistration alone does not delete the snapshots, so storage costs continue.

Code Example

Resolve the current Amazon Linux 2023 AMI ID from the AWS-published SSM Parameter Store path, so the ID is never hardcoded:

AMI_ID=$(aws ssm get-parameters \
  --names /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 \
  --query 'Parameters[0].Value' \
  --output text)

aws ec2 run-instances \
  --image-id "$AMI_ID" \
  --count 1 \
  --instance-type t3.micro \
  --key-name my-key-pair \
  --security-group-ids sg-0123456789abcdef0 \
  --subnet-id subnet-0123456789abcdef0

Create a versioned AMI from a configured instance; omitting --no-reboot lets EC2 reboot for a consistent snapshot:

aws ec2 create-image \
  --instance-id i-0123456789abcdef0 \
  --name "webapp-1.2.3" \
  --description "Web app golden image, build 1.2.3" \
  --tag-specifications 'ResourceType=image,Tags=[{Key=Owner,Value=platform-team},{Key=Build,Value=1.2.3}]'

Best Practices and Common Pitfalls

• Treat AMIs as immutable artifacts: version them in the name and tags, and rebuild through a pipeline rather than patching instances in place.

• Pin and resolve AMI IDs programmatically: use SSM Parameter Store public parameters or an infrastructure-as-code data source instead of copying IDs between scripts.

• Follow least privilege on launch: grant ec2:RunInstances only for specific AMIs, for example with a condition on ec2:Owner so users can only boot AMIs owned by your account.

• Clean up properly: deregistered AMIs leave orphaned EBS snapshots; automate snapshot deletion in the same pipeline step.

• Plan encryption and sharing together: AMIs with encrypted snapshots require the target account or region to have access to the KMS key, which is a frequent cause of failed cross-account or cross-region launches.

• Avoid --no-reboot for production builds: it can produce an AMI whose filesystem state is not fully consistent with a clean shutdown.

How Each Address Works

• Auto-assigned public IP: When you launch an instance into a public subnet with the public IPv4 addressing attribute enabled, AWS assigns an address from its own pool. The address is tied to that instance only while the instance is running; it is released when the instance is stopped or terminated, and a new one is assigned on the next start.

• Elastic IP (EIP): You explicitly allocate the address to your account in a specific Region and then associate it with an instance, an elastic network interface, a NAT gateway, or another supported resource. The association and the address itself persist independently of the instance’s lifecycle until you change or release them.

Key Differences

When to Use Each

• Use an auto-assigned public IP for ephemeral or horizontally scaled instances where the address does not matter: batch workers, stateless web tiers behind a load balancer, bastion hosts rebuilt by automation, or instances that only need outbound internet access.

• Use an Elastic IP when an external system must reach a fixed address: an on-premises firewall allowlist, a partner API callback, a primary/standby failover where you remap the address to the standby instance, or a NAT gateway that needs a stable source address for outbound traffic.

• For most public-facing applications, the better pattern is a stable DNS name fronting a load balancer or Auto Scaling group, so instances can come and go without any address management at all.

Best Practices and Common Pitfalls

• Prefer DNS over addresses: publish a Route 53 record to a load balancer or instance instead of handing out raw IPs, so you can change the underlying target without updating clients.

• Release unused Elastic IPs: addresses allocated but not associated with a running instance are billed, and they are a scarce Regional resource; use the EC2 console or aws ec2 describe-addresses to find and release them.

• Follow least privilege: grant only the specific ec2:AllocateAddress, ec2:AssociateAddress, or ec2:ReleaseAddress IAM actions on the resources needed, and use roles rather than long-lived access keys for any automation that manages addresses.

• Remember the stop behavior: a common surprise is that stopping an instance with an auto-assigned public IP changes its address, which breaks hardcoded clients or firewall rules; use an Elastic IP or a DNS name when stability matters.

• Don’t confuse private IPs: instances in a VPC also get a stable private IPv4 address that never changes on stop; that is for internal VPC communication and is unrelated to public reachability.

• Security is independent of address type: either address exposes the instance to the internet, so restrict inbound traffic with security groups and the principle of least exposure, and keep outbound traffic through a controlled path such as a NAT gateway when appropriate.

Core Functionality

The Console provides a graphical abstraction over AWS APIs, allowing users to perform operations through clicks and forms rather than code. Key capabilities include: - Resource Provisioning: Launching and configuring compute instances (Amazon EC2), storage buckets (Amazon S3), and databases (Amazon RDS) via guided workflows. - Monitoring and Observability: Viewing Amazon CloudWatch metrics, logs, and alarms directly within service dashboards to track health and performance. - Billing and Cost Management: Accessing the Billing Console to review charges, set budgets, and analyze cost allocation tags. - Security Configuration: Managing IAM users, groups, roles, and policies, as well as configuring multi-factor authentication (MFA) and IAM Identity Center for single sign-on.

Comparison with Other Interfaces

While the Console is intuitive, it is one of several ways to interact with AWS. Choosing the right interface depends on the task.

Security and Access Management

Security best practices dictate that the Console should not be accessed using the AWS root account except for initial setup. Instead, organizations should follow least privilege principles: - IAM Roles and Policies: Grant users only the specific actions and resources they need, avoiding broad administrative permissions. - IAM Identity Center: Use this service (formerly AWS SSO) to manage access across multiple AWS accounts and business applications centrally. - MFA Enforcement: Require multi-factor authentication for all human users accessing the Console to prevent unauthorized access even if credentials are compromised. - Session Management: Use temporary credentials via AWS STS rather than long-lived access keys for Console access where possible.

Best Practices and Limitations

The Console is invaluable for learning and troubleshooting, but reliance on it for production infrastructure management introduces risk. - Infrastructure as Code (IaC): Define production resources using AWS CDK, CloudFormation, or Terraform so infrastructure is version-controlled and reproducible. - Avoid Manual Drift: Changes made manually in the Console can drift from the declared state in IaC templates, causing deployment failures. - Region Awareness: The Console is region-specific; users must ensure they are operating in the correct AWS Region to avoid managing resources in the wrong location. - Quotas and Limits: Service quotas vary by region and change over time; consult the specific service documentation rather than assuming fixed limits when scaling via the Console.

When to Use the Console

Use the Console for initial exploration of new services, verifying resource states after automated deployments, and investigating alarms or errors. Avoid using it for repetitive tasks or scaling operations, where automation via CLI or SDKs ensures consistency and speed. For enterprise environments, federated access through IAM Identity Center is preferred over local IAM users for Console login.

Elasticity versus scalability

• Scalability: the ability to handle increased load by adding resources, which may be planned or manual.

• Elasticity: automatic, demand-driven scaling that adds capacity when load rises and removes it when load falls.

• In AWS, elasticity is achieved through policies and serverless execution rather than manual resizing.

Core services that deliver elasticity

• Amazon EC2 Auto Scaling: maintains a fleet of EC2 instances within a group, replacing unhealthy instances and adjusting count based on metrics or schedules.

• AWS Auto Scaling: provides a unified scaling plan across EC2 Auto Scaling groups, ECS services, Lambda provisioned concurrency, and DynamoDB.

• AWS Lambda: runs code in ephemeral, per-request compute environments; concurrency scales automatically with incoming events.

• AWS Fargate: runs containers without managing servers and scales task count based on service policies.

• Application Load Balancer: distributes requests across healthy targets in multiple Availability Zones and integrates with Auto Scaling.

• Amazon CloudWatch: collects metrics, evaluates alarms, and drives scaling actions.

Scaling approaches

• Horizontal scaling: adding or removing instances or tasks; preferred for stateless workloads because it avoids single-node limits.

• Vertical scaling: moving to a larger or smaller instance type; usually requires stopping the instance and is less elastic.

• Target tracking: keeps a metric such as average CPU utilization near a target value.

• Step scaling: applies different adjustments based on alarm breach magnitude.

• Scheduled scaling: changes capacity for predictable patterns, such as nightly batch windows.

• Predictive scaling: uses historical load data to provision capacity ahead of anticipated demand.

Benefits

• Cost efficiency: you pay only for compute used during actual demand, reducing spend on idle resources.

• Performance consistency: latency and throughput remain stable during traffic spikes.

• Fault tolerance: automatic replacement of unhealthy instances and distribution across Availability Zones improve availability.

• Operational efficiency: scaling decisions are automated, reducing manual intervention and on-call load.

• Agility: teams can launch experiments or promotions without pre-provisioning peak capacity.

Operational best practices

• Deploy across multiple Availability Zones: configure the Auto Scaling group and load balancer in at least two zones.

• Use load balancer health checks: attach the group to a target group so unhealthy instances are replaced automatically.

• Apply least privilege: give the instance role only the AWS service actions it needs, using IAM roles rather than long-lived access keys.

• Prefer target tracking or AWS Auto Scaling plans: they react more smoothly than simple CloudWatch alarms with cooldowns.

• Use lifecycle hooks: perform stateful tasks such as log upload or connection draining before termination.

• Test scaling policies: validate scale-out and scale-in behavior in a non-production environment before relying on it.

Core Components

IAM revolves around four primary entities that define access logic within an AWS account: - IAM Users: Permanent identities representing people or applications requiring long-term access to the AWS account. - IAM Groups: Collections of users used to simplify permissions management by attaching policies to the group rather than individual users. - IAM Roles: Temporary identities with permissions that can be assumed by users, applications, or AWS services (like EC2 or Lambda) without storing long-term credentials. - IAM Policies: JSON documents that explicitly define permissions, specifying allowed or denied actions on resource ARNs.

Why IAM Is Critical

Without IAM, AWS resources would be inaccessible or overly exposed. Its importance stems from three key areas: - Security Boundary: IAM acts as the first line of defense, preventing unauthorized access to sensitive data and infrastructure configurations. - Compliance and Auditing: Detailed logging via AWS CloudTrail integrates with IAM to track exactly who performed actions, satisfying regulatory requirements like SOC2 or HIPAA. - Granular Control: Permissions can be scoped down to specific API actions, resource ARNs, and contextual conditions (such as source IP address, time of day, or TLS usage).

IAM vs IAM Identity Center

It is vital to distinguish between legacy and modern access management tools to choose the right solution: | Feature | AWS IAM | IAM Identity Center | | :— | :— | :— | | Primary Use | Resource-level permissions within an account | Centralized workforce access across multiple accounts | | Identity Source | IAM users/groups or federated identities | External identity providers (e.g., Azure AD, Okta) | | Access Mechanism | Access keys, roles, console login | SSO via user portal | | Best For | Workloads, services, fine-grained resource policies | Human users accessing multiple AWS accounts |

Security Best Practices

Adhering to established security patterns minimizes risk exposure significantly: 1. Follow Least Privilege: Grant only the permissions necessary to perform a task, starting with no permissions and adding as needed based on CloudTrail analysis. 2. Use Roles Instead of Keys: For EC2 instances or Lambda functions, assign an IAM role rather than embedding access keys in code or configuration files. 3. Enable Multi-Factor Authentication (MFA): Require MFA for all human users, especially those with administrative privileges, to prevent credential theft. 4. Rotate Credentials Regularly: If access keys are unavoidable, enforce rotation policies and monitor for unused keys via IAM Access Analyzer. 5. Avoid Root Account Usage: The AWS root user has unrestricted access; secure it with MFA and use IAM users with administrative policies for daily tasks.

Common Pitfalls

Misconfigurations often lead to security incidents. Avoid granting wildcard permissions (*) on resources unless absolutely necessary for specific service-linked roles. Do not share IAM user credentials between individuals, as this breaks accountability and audit trails. Finally, ensure policies are attached to the correct level; attaching overly broad policies to groups can inadvertently grant excessive access to new members added later, violating the principle of least privilege.

Core Concept

The model splits security into two distinct domains. - Security of the Cloud: AWS is responsible for protecting the infrastructure that runs all of the services offered in the AWS Cloud. - Security in the Cloud: The customer is responsible for security in the cloud, determined by the AWS Cloud services used.

This division ensures that customers know exactly what they must configure versus what AWS manages automatically.

AWS Responsibilities

AWS manages the physical and virtual infrastructure layers. - Hardware and Facilities: Data centers, servers, storage devices, and networking hardware are maintained by AWS. - Virtualization Layer: The hypervisor and underlying host operating system are patched and secured by AWS. - Network Infrastructure: Global networking backbone, DNS infrastructure, and edge locations are managed by AWS. - Compliance Certifications: AWS maintains compliance programs (e.g., SOC, ISO, PCI-DSS) for the infrastructure itself.

Customer Responsibilities

Customers control what they place on the infrastructure. - Data Protection: Encryption at rest and in transit, classification, and integrity of customer data. - Identity and Access Management: Configuring users, roles, and policies using IAM or IAM Identity Center. - Operating System and Applications: Patching guest OS, managing application code, and configuring firewalls (security groups). - Network Configuration: Setting up VPCs, subnets, route tables, and network ACLs.

Variation by Service Type

Responsibility shifts based on the service abstraction level.

As abstraction increases, customer responsibility for infrastructure management decreases.

Best Practices

Adhering to security best practices ensures the customer side of the model is robust. - Enable Logging: Use AWS CloudTrail to record API calls for governance and auditing. - Encrypt Data: Use AWS KMS to manage encryption keys for sensitive data. - Monitor Continuously: Use Amazon GuardDuty to detect malicious activity. - Review Regularly: Use IAM Access Analyzer to identify unintended resource access. - Apply Least Privilege: Grant only necessary permissions to IAM entities instead of using broad policies.

How Each Works

• Vertical scaling: You change the capacity of one resource. In EC2, you stop the instance, change the instance type (for example from t3.medium to m6i.xlarge), and start it again; the instance keeps its EBS volumes, private IPv4 address, and instance ID. In RDS, you modify the DB instance class, which triggers a customer-initiated failover for Multi-AZ deployments. Vertical scaling has an upper bound — the largest available instance class — and usually involves a period of downtime or reduced availability.

• Horizontal scaling: You add or remove resources in response to demand. Amazon EC2 Auto Scaling adds instances to an Auto Scaling group across multiple Availability Zones, an Application Load Balancer or Network Load Balancer distributes traffic across them, and Amazon EC2 Auto Scaling terminates instances when demand falls. For data, horizontal read scaling uses Aurora or RDS read replicas; DynamoDB scales horizontally and transparently by distributing data across partitions behind a table’s partition key.

Key Differences

When to Choose Each

• Prefer vertical scaling for stateful systems that are hard to partition: a relational database that must remain strongly consistent, a legacy monolith with in-memory session state, or a workload with a single bottleneck such as CPU or memory on one node. RDS and Aurora still scale reads horizontally with read replicas, but writes remain on the primary instance.

• Prefer horizontal scaling for stateless web and API tiers, batch and event-driven workloads, and systems with unpredictable or spiky traffic. ECS with Fargate, EKS, Lambda, and EC2 Auto Scaling all assume horizontal elasticity. Horizontal scaling is also the only way to survive an Availability Zone failure when combined with Multi-AZ deployment.

• Combine both in practice: a common pattern is a horizontally scaled, stateless application tier in front of a vertically scaled primary database with horizontally scaled read replicas.

Best Practices and Common Pitfalls

• Make instances stateless before scaling out: move session state, caches, and file uploads to ElastiCache, DynamoDB, or S3 so any instance can serve any request.

• Distribute across Availability Zones: an Auto Scaling group should span at least three AZs; a single AZ of instances is horizontal scaling without high availability.

• Use load balancer deregistration delay: configure deregistration delay (called connection draining on the legacy Classic Load Balancer) so in-flight requests finish before an instance is terminated during scale-in.

• Design data tiering deliberately: horizontal write scaling requires partitioning (DynamoDB partition key, application-level sharding, or Aurora Limitless Database for MySQL-compatible Aurora); naive read replicas do not scale writes.

• Watch for “scale-up as a workaround”: repeatedly moving to a larger instance often masks an unindexed query, a memory leak, or a missing cache — profile before paying for a bigger node.

• Respect service quotas: horizontal growth is bounded by account and Regional quotas (for example, vCPU quotas for EC2 or concurrent executions for Lambda); request increases in advance rather than during an incident.

What High Availability Actually Means

High availability is usually discussed alongside two related properties, and mixing them up is a common interview mistake:

• Reliability/availability: the proportion of time a system is working, often expressed as a percentage of uptime (“nines”). AWS publishes service-level agreements per service, and the exact targets vary by service and Region.

• Fault tolerance: the ability to keep working while a failure is happening. A system can be highly available overall yet still degrade briefly during a failover.

• Disaster recovery: recovering an entire Region after a large-scale event. HA within one Region does not replace a multi-region DR strategy.

HA targets are typically expressed as RTO (recovery time objective — how quickly service must return) and RPO (recovery point objective — how much data loss is acceptable). Multi-AZ designs drive both toward near zero; multi-region designs are needed when an entire Region must be recovered.

The AWS Building Blocks

• Regions: fully independent geographic areas with their own power, cooling, and networking. Choosing a Region is the first isolation boundary.

• Availability Zones (AZs): distinct, physically separated locations inside a Region with low-latency, redundant connectivity. HA in AWS almost always means “span at least two or three AZs.”

• Edge locations and Regional edge caches: used by Amazon CloudFront and Route 53 to serve content and DNS close to users, which improves performance and absorbs regional load.

• Managed, inherently HA services: services such as Amazon S3, DynamoDB, Lambda, and Amazon SQS replicate across multiple AZs or facilities by default, so you inherit availability without designing it yourself.

Common HA Patterns

Design Principles

• Design for failure of any single component: assume an AZ, instance, node, or network path will fail, and verify the application still serves traffic.

• Make every tier stateless and horizontally scalable: put session state in ElastiCache, DynamoDB, or S3 so any instance can serve any request and instances are disposable.

• Use health checks everywhere: load balancer target group health checks, Route 53 health checks, and Auto Scaling health checks so failed components are removed automatically.

• Prefer managed services: services such as S3, DynamoDB, Lambda, Fargate, and Aurora remove undifferentiated HA work (patching, replication, failover) from your team.

• Automate deployment and recovery: infrastructure as code (CloudFormation, CDK, or Terraform) lets you rebuild an environment deterministically, which is essential for recovery and for testing failover.

• Apply least privilege: HA automation only works if its IAM roles have exactly the actions and resources it needs — for example, an Auto Scaling service role limited to EC2 actions on the specific resources, rather than broad access. Use roles, not long-lived access keys, and never the root account.

Common Pitfalls

• Single-AZ deployments: a “highly available” application with its database and compute in one AZ is not HA; an AZ outage takes it down.

• Forgotten single points of failure: a single NAT gateway, a single bastion host, one cache node, or a hard-coded dependency on one instance can defeat an otherwise multi-AZ design.

• Treating a snapshot as HA: backups protect against data loss but have a long RTO; they are recovery, not availability.

• Untested failover: Multi-AZ, read-replica promotion, and Route 53 failover should be exercised regularly, because misconfigured health checks or long DNS TTLs are often discovered only during a real incident.

• Ignoring service quotas: capacity limits vary by account, service, and Region and change over time; request increases in advance rather than assuming you can scale on demand. Check the documentation for each service’s current limits.

How It Works

You select an Amazon Machine Image (AMI), which is a template containing an operating system and often application software, then choose an instance type that defines the vCPUs, memory, storage and networking capacity. The instance launches into a VPC subnet you control, with security groups acting as stateful firewalls, and optionally an IAM role attached so the instance can call AWS APIs securely. You can connect using modern options such as EC2 Instance Connect or AWS Systems Manager Session Manager instead of relying solely on long-lived SSH keys. Instances run on the AWS Nitro System, which provides dedicated hardware for networking, storage and security functions and isolates workloads from each other.

Key Features

• Instance types and families: general purpose instances for balanced workloads, compute optimized for batch and gaming, memory optimized for databases and analytics, storage optimized for high local I/O, and accelerated computing with GPUs or AWS-designed accelerators. AWS Graviton-based instances use ARM processors and are commonly chosen for efficient, scalable application tiers.

• AMIs and bootstrapping: AMIs let you bake patched, hardened, repeatable images, while user data scripts run at first boot to install packages or join configuration management systems.

• Flexible storage: Amazon EBS volumes provide persistent block storage, instance store volumes provide low-latency temporary storage tied to the instance lifecycle, and EBS snapshots back up volumes to Amazon S3.

• Fine-grained networking: instances live in your VPC, can have multiple elastic network interfaces, use Elastic IPs for fixed public addresses, and can be grouped with placement groups to control proximity or hardware isolation.

• Purchasing and scaling models: On-Demand Instances suit unpredictable short-term workloads, Reserved Instances and Savings Plans suit steady-state usage, and Spot Instances suit flexible, interruptible workloads. Amazon EC2 Auto Scaling works with Elastic Load Balancing to add or remove instances based on demand.

• Operational controls: stop, start, reboot, hibernate where supported, and terminate instances; protect them from accidental termination; use Amazon CloudWatch for metrics and AWS Systems Manager for patching, inventory and runbooks.

EC2 Compared with Other AWS Compute Services

Advantages and Limitations

• Advantages: broad instance choice, per-second or per-hour billing models, fast provisioning, deep integration with AWS services, and the ability to run almost any compatible operating system or application stack.

• Limitations: you remain responsible for guest OS patching, hardening, licensing where applicable, and capacity planning unless you use managed services on top. Instance store data does not persist when the instance terminates, and certain instance types and AMIs vary by Region and Availability Zone.

Best Practices

• Provision instances with infrastructure as code using AWS CloudFormation, AWS CDK, Terraform, or the AWS CLI, rather than manual console clicks.

• Attach an IAM role with least-privilege permissions to the instance profile instead of embedding long-lived access keys.

• Keep AMIs patched and versioned, enforce IMDSv2, place instances in private subnets where practical, and expose them only through load balancers or controlled egress.

• Choose a mix of On-Demand, capacity reservation or savings commitments, and Spot capacity based on workload criticality and flexibility.

• Use Amazon EC2 Auto Scaling with Elastic Load Balancing, monitor with CloudWatch, and back up EBS snapshots regularly, ideally through AWS Backup.

How Instance Type Names Work

An instance type name is read left to right:

• Family letter: the workload profile, e.g. m (general purpose), c (compute), r (memory), i (storage), p/g (accelerated).

• Generation number: newer is generally better price/performance, e.g. c7 over c6 over c5.

• Optional attribute letters: CPU or accelerator, e.g. i for Intel (m6i), a for AMD (m7a), g for AWS Graviton (m7g), n for enhanced networking, z for high frequency.

• Size: nano, micro, small, medium, large, xlarge, then 2xlarge through 48xlarge, plus bare-metal metal sizes. Available sizes differ by family.

Instance Families and Their Use Cases

Choosing the Right Family

• Match the bottleneck, not the label. CPU-bound workloads belong on C, memory-bound on R/X, latency-sensitive local storage on I, throughput-sensitive storage on D/H, and massively parallel math on P/G/Inf/Trn.

• Compare memory per vCPU. c7i.2xlarge and r7i.2xlarge both have 8 vCPUs, but the r7i has roughly four times the memory; pick based on your application’s RAM footprint, not vCPU count alone.

• Prefer current generations. A c7i or c7g generally outperforms an equivalent c5 on price/performance and supports newer Nitro features; previous generations remain available for legacy compatibility.

• Evaluate Graviton (g suffix) seriously. Graviton instances often deliver the best price/performance, but the workload must be ARM-compatible and you need to rebuild or re-pull container images; benchmark before migrating.

• Use burstable T deliberately. T families suit intermittent traffic; sustained high utilization can exhaust credits, so enable Unlimited mode only with cost monitoring.

• Verify availability before you design around a type. Not every family or size is offered in every Region or Availability Zone, and offerings change over time; check the EC2 console, describe-instance-types, or the service documentation for your Region.

Best Practices and Common Pitfalls

• Right-size first, scale out second. Start with the smallest type that meets your capacity target, then use Auto Scaling groups horizontally; over-provisioning “just in case” is the most common waste.

• Let tooling recommend changes. AWS Compute Optimizer and Cost Explorer surface under-utilized instances and right-sizing candidates continuously.

• Use roles, not long-lived keys. Give instances an IAM role with least-privilege actions on specific resources rather than embedding access keys in user data or AMIs.

• Pitfall: ignoring instance store vs. EBS. I and D families only shine when the workload actually uses the local disks; if you attach only EBS volumes, you are paying for hardware you never touch.

• Pitfall: assuming one family everywhere. Homogeneous fleets are simpler, but a typical platform benefits from mixing types, e.g. m7i for services, r7i for caches, inf2 or g6 for inference tiers.

How It Works

ECS separates the control plane from the data plane. AWS manages the scheduling and orchestration software; you provide either your own Amazon EC2 instances or serverless capacity through AWS Fargate. The flow is:

1. You build a container image and push it to Amazon ECR (or another registry).

2. You register a task definition, a JSON document describing the containers, their images, CPU and memory requirements, ports, environment variables, logging and the IAM role the task assumes.

3. You create a cluster, a logical grouping of compute capacity and the tasks running on it.

4. You run tasks either ad hoc or through an ECS service, which keeps a desired number of tasks running and, optionally, spreads them across an Application Load Balancer or Network Load Balancer.

5. ECS reports state and metrics to Amazon CloudWatch, and you can attach auto scaling policies to the service or the cluster capacity.

Key Components

• Task definition: the blueprint for your application; it can hold one or more containers that share a network namespace and lifecycle.

• Task: a running instantiation of a task definition, the atomic unit ECS schedules.

• Service: a scheduler configuration (the REPLICA or DAEMON scheduling strategy) that maintains the desired task count, performs rolling updates and, for REPLICA, integrates with Elastic Load Balancing.

• Cluster: the regional grouping of capacity, tasks and services.

• Capacity provider: links a cluster to its compute, for example a Fargate capacity provider or an Auto Scaling group-backed provider for EC2, and manages scale-out and scale-in.

• Task execution role and task role: the execution role lets the ECS agent pull images and write logs; the task role grants the application itself AWS permissions, following least privilege instead of embedding long-lived credentials in the image.

EC2 Launch Type vs Fargate

ECS also supports ECS Anywhere to run tasks on your own on-premises servers registered as external capacity, and Amazon EKS is the alternative to choose when you specifically need Kubernetes APIs, Helm charts or a portable, multi-cloud control plane.

Code Example

A minimal Fargate task definition in CloudFormation, showing the fields ECS actually reads:

TaskDefinition:
  Type: AWS::ECS::TaskDefinition
  Properties:
    Family: orders-api
    RequiresCompatibilities:
      - FARGATE
    NetworkMode: awsvpc
    Cpu: "512"
    Memory: "1024"
    ExecutionRoleArn: !Ref ExecutionRole
    TaskRoleArn: !Ref TaskRole
    ContainerDefinitions:
      - Name: api
        Image: !Sub "${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/orders-api:1.0.0
        Essential: true
        PortMappings:
          - ContainerPort: 8080
            Protocol: tcp
        LogConfiguration:
          LogDriver: awslogs
          Options:
            awslogs-group: /ecs/orders-api
            awslogs-region: !Ref AWS::Region
            awslogs-stream-prefix: api

The equivalent inspection command with the AWS CLI is aws ecs describe-task-definition --task-definition orders-api, and aws ecs describe-tasks shows the runtime placement and health of individual tasks.

Best Practices

• Prefer Fargate for new stateless services unless you need host-level control; it removes AMI patching from your scope.

• Use awsvpc network mode with Fargate so each task gets its own elastic network interface and security group, which is also required for the launch type.

• Grant the task role only the specific actions and resources the application needs, and never place IAM access keys in container environment variables.

• Reference secrets with AWS Secrets Manager or Parameter Store in the task definition instead of hardcoding them, and keep the execution role limited to reading just those secrets.

• Put services behind an Application Load Balancer, use target tracking auto scaling on average CPU or memory, and set minimumHealthyPercent and maximumPercent for controlled rolling deployments.

• Send logs through the awslogs driver to CloudWatch Logs from day one; without it, container stdout is lost when a task is replaced.

How Each Works

• Amazon EC2: You select an instance type (a fixed combination of vCPUs, memory, and storage), choose an Amazon Machine Image (AMI), pick a networking and security configuration, and launch instances. You then connect to them, install your runtime and application, and keep them running. Billing is per-second (with a minimum of 60 seconds) while the instance is running, regardless of whether it is doing useful work.

• AWS Lambda: You upload code (a function) and configure a trigger, such as an Amazon API Gateway request, an Amazon S3 event, an Amazon DynamoDB stream record, or a scheduled Amazon EventBridge rule. AWS runs the function only when invoked, starts compute capacity automatically to match the request rate, and stops charging when execution finishes. You never see or patch the underlying servers.

Key Differences

When to Choose Each

• Choose EC2 when you need a long-running process, a fixed host identity, deep OS-level tuning, custom software that must stay resident in memory, or a workload that runs continuously and can be cost-efficiently packed onto instances. Typical examples are game servers, legacy enterprise applications, database servers you self-manage, and container hosts when you want control without a managed orchestrator.

• Choose Lambda when you have short-lived, event-driven workloads with bursty or unpredictable traffic. Typical examples are request processing behind API Gateway, image or video processing triggered by S3 uploads, real-time data transformation from DynamoDB Streams or Amazon Kinesis, chatbots, and scheduled automation jobs. Lambda pairs naturally with other serverless building blocks such as API Gateway, DynamoDB, and EventBridge.

• Consider the middle ground: AWS Fargate gives you containers without managing servers when Lambda’s constraints do not fit but you still want to avoid instance operations, while Amazon EKS or ECS on EC2 suits large container platforms where you want cluster-level control.

Best Practices and Common Pitfalls

• Follow least privilege: for both services, grant permissions to an IAM role scoped to specific actions and resources rather than embedding long-lived access keys. EC2 instances should use an instance profile; Lambda functions should use an execution role.

• Design Lambda for statelessness: keep connection objects and caches at module scope so they can be reused across invocations within a warm execution environment, but never rely on local state persisting between invocations.

• Mind the cold starts: the first invocation of a Lambda function after a period of inactivity takes longer to serve. For latency-critical paths, consider provisioned concurrency and keep deployment packages small.

• Avoid over-provisioned EC2: use EC2 Auto Scaling and right-sized instance types, and stop or terminate idle instances to control cost. Actual pricing varies by Region and changes over time; consult the EC2 and Lambda pricing documentation.

• Do not treat Lambda as a replacement for everything: workloads that run continuously or that need very long execution times are usually a poor fit and can be more expensive on Lambda.

How It Works

• Environment: the logical container for your application. An environment maps to a set of AWS resources (Amazon EC2 instances or ECS tasks, an Application Load Balancer, an Auto Scaling group, an Amazon S3 bucket for logs and uploaded versions, and Amazon CloudWatch alarms).

• Application version: a labeled, immutable artifact — a source bundle in Amazon S3 or a container image in Amazon ECR — that you deploy to one or more environments.

• Platform: a preconfigured, AWS-maintained stack (language runtime, web server or application container, and operating system) that your code runs on. You pick the platform branch and version; AWS manages patching through managed platform updates.

• Deployment policies: Elastic Beanstalk supports rolling, rolling with additional batch, immutable, and traffic-splitting (canary) deployments, plus in-place and blue/green (swap environment URL) strategies.

• Infrastructure as code: every environment is backed by AWS CloudFormation, and you can extend it with .ebextensions configuration files or, for newer platform branches, platform.yaml-based custom platforms. You can also manage environments through the AWS CLI, the EB CLI, AWS SDKs (for example AWS SDK for JavaScript v3 or boto3), AWS CloudFormation, AWS CDK, or Terraform.

Supported Platforms

Elastic Beanstalk supports Docker (single-container, preconfigured multi-container, and Amazon Linux 2/2023 Docker platforms), Java with Tomcat, .NET on Windows Server, Node.js, PHP, Python, Ruby, and Go. For containerized workloads, the ECS-based Docker platform runs your image as an Amazon ECS task on EC2 instances launched and managed by Elastic Beanstalk.

When to Use It

• You want to run code, not manage clusters: Elastic Beanstalk removes the day-to-day work of patching OS images, configuring load balancers, and writing scaling policies.

• Long-running web apps and APIs: anything that needs to be always on and scaled horizontally — a public-facing website, a REST API, a background worker tier — fits the model well.

• Predictable, simple architectures: when a single application plus optional worker tier matches your topology, Elastic Beanstalk gives you production best practices almost out of the box.

• You still need control: unlike a fully abstracted platform, you can SSH into instances, choose instance types and AMIs, attach custom security groups and IAM instance profiles, and override CloudFormation resources — useful when a pure serverless or container service is too restrictive.

When to Choose Alternatives

Best Practices and Common Pitfalls

• Follow least privilege: let Elastic Beanstalk use its service-linked role (AWSElasticBeanstalkManagedUpdatesCustomerRolePolicy and related) and grant your EC2 instance profile only the actions your application needs — never broad AdministratorAccess. Avoid long-lived access keys on instances; use the instance profile or IAM roles for service accounts instead.

• Use immutable or traffic-splitting deployments for production to avoid a bad version affecting all instances at once.

• Keep environments stateless: store session state and files in Amazon ElastiCache, Amazon DynamoDB, or Amazon S3 so auto scaling and instance replacement are safe.

• Pin and test platform versions deliberately: automatic minor version updates are helpful, but validate them in a staging environment first.

• Watch cost, not quotas: Elastic Beanstalk itself adds no charge — you pay for the EC2 instances, load balancer, NAT gateways, S3 storage, and CloudWatch it provisions, and prices vary by region and change over time. Right-size instances and enable scheduled scaling for predictable traffic patterns.

• Don’t treat it as a black box: because it is CloudFormation under the hood, changes made directly in the console can drift from your configuration source; manage environments through code and pipelines (for example, AWS CodePipeline) so deployments are repeatable.

Core Components

• Scaling group / resource: the thing being scaled — an EC2 Auto Scaling group, an ECS service, a DynamoDB table or global secondary index, an Aurora replica set, and so on.

• Launch template (EC2): defines the AMI, instance type, key pair, security groups, and user data used to start new instances. Launch templates are the recommended replacement for the older launch configurations, which are legacy.

• CloudWatch alarms and metrics: scaling decisions are driven by metrics such as CPUUtilization, RequestCountPerTarget, or custom metrics. EC2 Auto Scaling also emits built-in metrics like GroupInServiceInstances.

• Scaling policies: the rules that decide when to scale and by how much.

• Instance refresh (EC2): rolls instances in a group to a new launch template version, keeping a minimum number of instances in service during the rollout.

Scaling Strategies

• Dynamic scaling: reacts to current demand. Three policy types:

• Target tracking scaling — the most common; keeps a metric at a target value, e.g. average CPU at 50%. Simple to reason about and usually the default choice.

• Step scaling — defines thresholds and adjustment amounts, e.g. “add 2 instances when CPU is between 60% and 70%, add 4 above 70%”.

• Simple scaling — a single adjustment per alarm breach with a cooldown; largely superseded by step and target tracking policies.

• Scheduled scaling: scales at known times, e.g. a flash sale or business-hours traffic pattern.

• Predictive scaling (EC2): uses machine learning on historical CloudWatch data to provision capacity ahead of forecasted demand; useful for recurring daily or weekly patterns.

• Warm pools (EC2): pre-initialize instances in a Stopped or Running state (including Hibernated) so scale-out is faster, which helps latency-sensitive workloads.

How a Scale-Out Happens

1. Traffic increases and CloudWatch metric data points breach the alarm threshold.

2. The alarm transitions to ALARM and invokes the scaling policy.

3. The scaling service computes the desired capacity from the policy.

4. For EC2, the Auto Scaling group launches instances from the launch template into the specified subnets and attaches them to the target group.

5. The load balancer health check marks the new targets healthy, and traffic flows to them.

6. When demand drops, a scale-in policy removes instances, respecting the group’s minimum size and instance termination policies (e.g. terminating the oldest launch template first, or the oldest instances).

Choosing Between Related Services

Best Practices and Common Pitfalls

• Prefer target tracking over hand-tuned step policies unless you need asymmetric, threshold-based control.

• Scale on application-level metrics such as RequestCountPerTarget or queue backlog (ApproximateNumberOfMessagesVisible for SQS) rather than CPU alone; CPU lags real user-facing load.

• Set realistic cooldowns: too short causes thrashing, too long delays response to spikes; pair with warm pools when scale-out latency matters.

• Keep capacity buffers: a minimum size that absorbs single-instance failure, and a maximum that protects against runaway cost from a misconfigured policy or metric anomaly.

• Use instance refresh to deploy new AMIs or instance types without downtime.

• Follow least privilege: grant only autoscaling:* actions scoped to specific group ARNs, and pass an instance profile role rather than embedding credentials in user data.

• Watch for metric gaps: custom metrics must be published reliably; a missing metric means scaling policies stop reacting.

• Do not mix scaling controls on the same resource (for example, manual capacity changes fighting dynamic policies) unless you understand which wins.

How It Works

An EKS deployment consists of two parts:

• The control plane: AWS runs the Kubernetes API servers, etcd and scheduler across multiple Availability Zones in an AWS-managed account, with cluster endpoints in your VPC. You interact with the cluster through the standard Kubernetes API using kubectl or any Kubernetes client.

• The data plane: You provision the compute that runs your Pods. EKS supports several node types, and you choose based on workload shape and operational preference.

Node Types

• Amazon EKS Managed Node Groups: EC2 instances that AWS provisions and patches; you still control the instance type, AMI and scaling.

• Self-managed EC2 nodes: you own the instances and the kubelet configuration entirely.

• AWS Fargate with EKS: serverless compute where AWS provides the infrastructure per Pod, so there are no instances to manage.

• EKS Hybrid Nodes: lets you attach on-premises or edge infrastructure as nodes to an EKS control plane.

Key Benefits

• Managed, highly available control plane: AWS runs the API servers and etcd across multiple Availability Zones, which removes a large part of the operational burden of running Kubernetes yourself.

• Upstream compatibility: EKS uses the same upstream Kubernetes releases, so existing Kubernetes manifests, Helm charts, controllers and tools work without modification, and you are not locked into a proprietary orchestration API.

• Deep AWS integration: EKS works directly with IAM for authentication, VPC networking via the Amazon VPC CNI, Elastic Load Balancing for LoadBalancer and Ingress services, ECR for images, and CloudWatch for logging and metrics.

• Security and compliance: clusters integrate with IAM, support private endpoints and fine-grained network policies, and EKS is available in AWS GovCloud and on AWS Outposts for regulated or on-premises workloads.

• Choice of compute: the same cluster can run EC2, Fargate and hybrid nodes, so you can match infrastructure to each workload instead of committing to one model.

• Ecosystem compatibility: because it is standard Kubernetes, you can use CNCF projects such as Argo CD, Istio and Prometheus, plus AWS add-ons like the AWS Load Balancer Controller.

When to Choose EKS Versus Alternatives

Considerations and Best Practices

• You still manage parts of it: node patching, workload scaling, upgrades of node AMIs and day-2 operations remain your responsibility, especially with EC2-based nodes.

• Follow least privilege: grant IAM roles to nodes and service accounts scoped to specific actions and resources, and use IAM Roles for Service Accounts rather than broad node-instance permissions.

• Plan upgrades: Kubernetes versions reach end of support, so track the EKS release calendar and upgrade control plane and nodes on a regular cadence.

• Cost awareness: pricing varies by region and changes over time; you pay for the control plane plus the underlying compute and any add-ons. See the EKS pricing documentation, and consider Savings Plans or Fargate for suitable workloads.

• Start with infrastructure as code: define clusters with CloudFormation, AWS CDK, Terraform or eksctl, and keep node groups, add-ons and policies versioned rather than creating clusters by hand.

How It Works

AWS continuously has spare capacity in its Availability Zones, and the Spot price for each instance type and Availability Zone adjusts with long-term supply and demand. When you request Spot capacity, you either launch instances directly at the current Spot price or use a fleet mechanism that requests capacity across many instance types and Availability Zones.

• No bidding auction: the earlier auction model was replaced, so you no longer place a maximum bid against other customers; you pay the prevailing Spot price, and interruptions are driven by capacity reclamation rather than by other customers’ bids.

• Interruption notice: when AWS needs the capacity back, it emits an Amazon EventBridge event (formerly CloudWatch Events) and exposes the notice in the instance metadata, giving you up to two minutes before the instance is stopped, hibernated, or terminated.

• Interruption behavior: for EBS-backed instances you can choose terminate (default), stop, or hibernate when the Spot Instance is interrupted.

• Capacity pools: each instance type in each Availability Zone is a separate pool, so spreading requests across many pools is what makes Spot capacity reliable at scale.

Spot Compared with Other EC2 Purchasing Options

Spot Blocks, which previously let you reserve Spot capacity for a fixed 1–6 hour window, are deprecated and no longer offered; use Spot Instances with a Spot Fleet or an Auto Scaling group mixed instances policy for durable, diversified Spot capacity instead.

Interruption Handling

Because a Spot Instance can be reclaimed, the workload has to expect interruptions rather than treat them as failures:

1. Amazon EventBridge delivers the Spot interruption notice roughly two minutes ahead of time.

2. The same notice is available through the instance metadata service, so in-instance code can poll for it.

3. Your application uses the notice to checkpoint state, drain connections, finish in-flight work, and shut down cleanly.

4. The fleet mechanism (Spot Fleet, EC2 Fleet, or an Auto Scaling group) automatically replaces the interrupted instance from another capacity pool.

When to Use Spot Instances

Spot fits workloads where the loss of any single instance is tolerable and the overall job still completes:

• Batch and big data processing, such as Amazon EMR, AWS Batch, and Apache Spark jobs that can retry or resume tasks.

• Containerized workloads on Amazon ECS, Amazon EKS, or AWS Fargate Spot, where the scheduler reschedules pods or tasks onto other nodes.

• CI/CD and build fleets, where an interrupted build is simply retried.

• HPC, rendering, and machine learning training that checkpoint progress, including Amazon SageMaker managed Spot training.

• Stateless web and API tiers behind a load balancer with enough surplus capacity to absorb node loss.

Avoid Spot for workloads that cannot tolerate interruption: primary relational or in-memory databases, long-running jobs with no checkpointing, monolithic stateful services, and anything with a hard availability SLA. For guaranteed capacity, use On-Demand with On-Demand Capacity Reservations, and use Reserved Instances or Savings Plans for the predictable baseline.

Best Practices and Common Pitfalls

• Diversify: request many instance types across many Availability Zones so a single pool drying up does not stall the workload.

• Use the capacity-optimized allocation strategy in Spot Fleet, which launches from the pools with the most available capacity and so reduces interruption risk compared with lowest-price selection.

• Make workloads idempotent and resumable: checkpoint to Amazon S3, Amazon EFS, or a database so a retried task does not duplicate work.

• Handle the two-minute notice: drain load balancer registrations, close connections, and flush buffers before shutdown.

• Do not rely on Spot for capacity you must have: treat Spot as elastic capacity and keep a non-Spot baseline where availability matters.

• Watch cost behavior: Spot prices vary by instance type, Availability Zone, and over time, and Spot usage counts toward Savings Plans coverage, so review the EC2 console or AWS Cost Explorer rather than assuming a fixed discount.

How Each Service Works

• Amazon ECS provides the orchestration layer. You define a task definition (the container image, CPU/memory, networking mode, IAM task role, logging, and other settings), then run it as a standalone task or as a long-running service that maintains a desired count and integrates with an Application Load Balancer, service auto scaling, and capacity providers. ECS keeps the cluster in the desired state, restarts failed tasks, and schedules them onto available capacity.

• The EC2 launch type means you supply that capacity: you create an Auto Scaling group of EC2 instances (or register existing instances), choose the AMI, patch the instances, and scale the instance fleet to match aggregate task demand. On-premises or VM hosts are supported through the separate External launch type via ECS Anywhere.

• The Fargate launch type means AWS supplies the capacity. When you request a task, Fargate allocates matching compute for exactly that task, launches it, and tears it down when the task stops. There is no cluster of instances to size, patch, or rightsize, and each task runs in its own isolated compute environment with its own elastic network interface and security group.

Key Differences

When to Choose Which

• Choose Fargate when you want to minimize operational overhead: microservices, batch jobs, CI/CD runners, scheduled tasks, and applications with bursty or unpredictable demand. It is also a good default when teams should focus on application code rather than capacity management, and it simplifies least-privilege networking because every task gets its own security group.

• Choose the EC2 launch type when you need control or cost density that Fargate cannot provide: GPU or specialized instance types, software licensed per host, tasks that must run on every instance (daemon scheduling), large memory footprints beyond the supported Fargate task sizes, workloads needing fine-grained Spot strategies across many instance families, or workloads requiring host-level access and custom AMI hardening.

• A single ECS cluster can use both launch types through capacity providers, so a common pattern is to run stateless services on Fargate while keeping a small EC2 capacity provider for workloads with special requirements.

Best Practices and Common Pitfalls

• Rightsize tasks deliberately. With Fargate you specify exact vCPU/memory combinations; over-provisioning directly increases cost because you are billed per task, not per shared host.

• Use an IAM task role, not long-lived credentials. Grant the role only the specific actions and resources the container needs, and never bake access keys into an image.

• Check current task sizes and regional availability before designing around Fargate. Supported vCPU/memory combinations, Windows container support details, and feature availability vary and change over time; consult the AWS Fargate documentation for the authoritative, current list rather than assuming a fixed limit.

• Remember there is no host to log into. For troubleshooting, rely on container logs sent to Amazon CloudWatch Logs and ECS Exec; do not design operational procedures around SSH access to a Fargate task’s host.

• Do not confuse Fargate with ECS. “Moving to Fargate” usually means switching an ECS service’s launch type from EC2 to FARGATE in the task definition and service configuration — ECS remains the orchestrator.

How It Works

• Registry and repositories: Each AWS account gets a default private registry, and you create repositories inside it to hold images. An image URI looks like 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest, where the host encodes the account, service, and Region.

• Push and pull: You authenticate your Docker or Podman client to ECR, then use standard docker push and docker pull commands. ECR stores each image as a set of immutable layers, so layers shared between images are stored once.

• OCI support: In addition to Docker images, ECR accepts OCI images and OCI artifacts, including Helm charts used with Amazon EKS.

• Public images: Amazon ECR Public provides a separate public registry (public.ecr.aws) for images anyone can pull, with a public gallery for discovery.

Key Features

• Integrated authentication: Image pulls use IAM credentials. Workloads running on ECS, EKS, Lambda, or EC2 get temporary credentials through their execution or instance role, so no long-lived registry passwords are stored in manifests or Kubernetes image-pull secrets.

• Lifecycle policies: JSON rules automatically expire untagged or old images, controlling storage growth and cost.

• Image scanning: Basic and enhanced scanning identify software vulnerabilities in pushed images; enhanced scanning findings flow to Amazon Inspector and Security Hub.

• Tag immutability and pull-through cache: Tag immutability prevents overwriting a tag such as v1.0.0. Pull-through caching proxies images from upstream public registries so external rate limits and network egress are reduced.

• Replication: Cross-Region and cross-account replication copy images automatically, supporting disaster recovery and multi-account deployment models.

• Integration with deployment services: ECS and EKS resolve ECR URIs directly, and CodePipeline/CodeBuild can build and push to ECR as a build stage.

ECR Compared with Related AWS Services

Best Practices

• Use IAM roles, not access keys: Grant only the needed pull actions (such as ecr:BatchGetImage, ecr:GetDownloadUrlForLayer, and ecr:BatchCheckLayerAvailability) to the task or pod execution role, scoped to specific repository ARNs, following least privilege.

• Pin image tags or digests: Deploy by digest or immutable tags rather than latest to make rollouts reproducible.

• Automate vulnerability scanning and lifecycle policies: Fail builds on critical findings and expire stale images automatically.

• Choose ECR when you run on AWS: It removes registry maintenance and keeps traffic inside the AWS network; use a public registry only when images must be consumable outside your account.

Core Concepts

• Bucket: the top-level container that holds objects. Bucket names are globally unique across all of AWS, are created in a specific Region, and now default to private access with S3 Object Ownership set to bucket owner enforced, which disables legacy access control lists (ACLs).

• Object: the actual file plus its metadata, stored in a bucket. Each object can be up to 5 TB; the practical unit for large uploads is multipart upload.

• Key: the unique name (path) of an object within a bucket, for example reports/2025/q3/summary.csv. There is no real hierarchy — the slashes create a logical prefix structure that tools render as folders.

• Versioning: when enabled, S3 keeps every variant of an object, so accidental overwrites or deletes can be recovered. Versioning is also a prerequisite for replication and for S3 Object Lock.

Storage Classes

Choosing the right class is the main cost lever; pricing varies by Region, storage size, request volume, and data transfer, so check the AWS S3 pricing documentation rather than assuming fixed rates.

Security and Access Control

• Encryption: S3 applies server-side encryption with S3-managed keys (SSE-S3, AES-256) by default to new object uploads. You can instead use SSE-KMS with AWS KMS keys for control over key policy and audit trail, SSE-C when you supply your own keys, or client-side encryption before upload.

• Access control: use IAM identity-based policies and resource-based bucket policies, following least privilege — specific actions on specific bucket ARNs, and roles for applications instead of long-lived access keys. Never use the root account for routine S3 access.

• S3 Access Points: named endpoints with dedicated policies, useful when many teams or applications share one bucket.

• S3 Object Lock: write-once-read-many (WORM) protection with retention periods and legal holds, often used for compliance.

• Presigned URLs: time-limited URLs for granting temporary download or upload permission without exposing credentials.

Data Management and Automation Features

• Lifecycle rules: automatically transition objects to cheaper classes or expire them, for example moving logs to S3 Glacier Flexible Retrieval after 90 days and deleting them after a year.

• Replication: same-region replication (SRR) and cross-region replication (CRR) copy objects to another bucket, requiring versioning on both sides.

• Event notifications: object creation or deletion events can be delivered to Amazon SQS, Amazon SNS, AWS Lambda, or Amazon EventBridge, which keeps S3 distinct from those messaging services — S3 stores the data, they react to it.

• Batch Operations and S3 Inventory: run large-scale jobs (copy, replace tags, restore) and report on bucket contents.

Performance, Integration, and Observability

• S3 Transfer Acceleration and Multi-Region Access Points speed up global uploads and route requests to the closest healthy Region.

• S3 Storage Lens provides dashboards on storage trends, and access logging plus AWS CloudTrail record who did what to which object.

• S3 integrates natively with CloudFront for content delivery, Athena for querying data in place, and IAM Identity Center for workforce access.

When to Choose S3 (and When Not To)

Choose S3 for durable, virtually unlimited, API-accessible object storage with rich lifecycle and security controls. Choose Amazon EBS when an EC2 instance needs a boot volume or block device, and Amazon EFS when multiple compute instances need shared POSIX file access — S3 is not a mounted file system, and its per-object latency makes it unsuitable as a transactional database.

Storage Architecture and Data Model

• Amazon S3: organizes data as objects within buckets using a flat namespace; each object comprises data, metadata, and a unique key, enabling massive scalability without hierarchical directory constraints.

• Amazon EBS: presents a raw block device that must be formatted with a file system (e.g., ext4, NTFS) before use; volumes persist independently of the EC2 instance lifecycle but are bound to a single Availability Zone.

Access Methods and Connectivity

• S3: accessed via REST APIs, SDKs, or the AWS CLI over HTTPS; reachable from any network location with proper credentials within the region.

• EBS: mounted to EC2 instances like physical hard drives and accessed via standard file system operations; cannot be accessed directly over the network or shared across multiple instances simultaneously, except for Multi-Attach enabled io1/io2 volumes.

Durability and Availability Model

• S3: replicates data across multiple facilities within a region, offering 99.999999999% (11 nines) durability; designed for critical data preservation.

• EBS: data resides within a single Availability Zone, with synchronous replication within that zone for standard volumes (gp2/gp3); durability is enhanced through snapshots stored in S3, which protect against volume failure but require explicit management and scheduling.

Comparison Table

Use Cases and Performance Characteristics

• S3: ideal for static website hosting, backup archives, data lakes, and big data analytics where data is accessed via APIs; scales automatically.

• EBS: critical for boot volumes, transactional databases (e.g., PostgreSQL, MySQL), and applications requiring POSIX-compliant file system access; performance depends on volume type and instance bandwidth, offering lower latency for frequent read/write operations.

Security and Best Practices

• Apply least-privilege IAM policies: restrict S3 actions (e.g., s3:GetObject, s3:PutObject) to specific buckets and EBS actions (e.g., ec2:AttachVolume, ec2:CreateSnapshot) to specific volume IDs.

• Enable S3 versioning and encryption at rest for both services using AWS KMS; enforce bucket policies and block public access for S3.

• Use S3 lifecycle policies to transition objects to cheaper storage classes (e.g., S3 Standard-IA, Glacier) and automate EBS snapshots for disaster recovery.

• Monitor EBS volume performance with CloudWatch and adjust volume types (gp3, io2) as needed; remember that EBS costs accrue while provisioned, whereas S3 costs are based on actual storage consumed and request rates.

What RDS Handles for You

• Managed lifecycle: instance creation, storage attachment, engine version upgrades, and maintenance windows are handled by the service, with options to control when they are applied.

• High availability: Multi-AZ deployments keep a synchronous standby replica in another Availability Zone and fail over automatically, so they are for resilience rather than read scaling.

• Scaling: you can scale compute and storage vertically, add read replicas to distribute read traffic, and for Amazon Aurora use serverless capacity that adjusts automatically.

• Backup and recovery: automated backups with point-in-time restore, manual snapshots, and retention settings you configure.

• Security: encryption at rest with AWS KMS, encryption in transit with TLS, Amazon VPC isolation, security groups, IAM authentication policies, and integration with AWS Secrets Manager for credentials.

Supported Database Engines

• Amazon Aurora: AWS-designed, MySQL-compatible and PostgreSQL-compatible engines with a distributed, self-healing storage layer. Aurora is part of the RDS family but has distinct capabilities such as up to 15 low-latency read replicas, fast failover, and Aurora Serverless v2 for workloads with variable load.

• MySQL: the community editions offered by AWS under a license-included model, suited to general-purpose open-source applications.

• PostgreSQL: license-included open-source engine, chosen for features such as advanced data types, indexing options, and extensions.

• MariaDB: license-included open-source engine, a community-developed fork of MySQL.

• Oracle: available as License Included or Bring Your Own License (BYOL) for organizations with existing Oracle licenses; RDS Custom for Oracle adds OS-level and DB-level access for workloads that need customizations.

• Microsoft SQL Server: available as License Included or BYOL, commonly used for Windows-centric enterprise applications; RDS Custom for SQL Server provides deeper access where required.

• Db2: Amazon RDS for Db2 runs IBM Db2 under a license-included model, aimed at customers modernizing existing Db2 workloads.

Choosing the Right Engine

If an application needs an engine RDS does not offer, or requires deep operating system access, alternatives include Amazon EC2 for full self-management, RDS Custom where supported, or purpose-built AWS services such as Amazon DynamoDB, Amazon Neptune, or Amazon ElastiCache.

Best Practices

• Use Multi-AZ for production resilience and test failover behavior before you rely on it.

• Keep automated backups enabled and verify restore drills; backups are only as good as your tested recovery path.

• Store database credentials in AWS Secrets Manager and rotate them, rather than embedding them in application configuration.

• Follow least privilege: grant IAM and security group access to specific actions, ports, and resources only, and avoid using the AWS account root user for any database operation.

• Use read replicas to scale reads, but monitor replication lag because they are asynchronous by default.

• Enable encryption at rest when you create the instance, since it cannot be applied retroactively to an unencrypted instance without a snapshot restore.

How S3 Organizes Data

S3 is flat object storage, not a file system. There are no real folders or directories; an object key is simply a string, and tools display prefixes like reports/2025/q1.csv as folders for convenience. This differs from block storage (Amazon EBS) attached to an EC2 instance or file storage (Amazon FSx) mounted by multiple servers. S3 is best suited for unstructured data: backups, media files, application artifacts, static website assets, and data-lake inputs.

Buckets

• Purpose: a bucket groups related objects and is the unit where you apply settings such as versioning, logging, encryption, lifecycle rules, and access policy.

• Naming: bucket names must be globally unique across all of AWS, so a name you choose cannot be used by another account. Names are constrained in length and character set (lowercase letters, numbers, hyphens, dots) and must not look like an IP address; see the S3 documentation for the exact rules.

• Region: each bucket lives in a single region you select at creation time. Data stays in that region unless you explicitly configure replication to another region.

• Ownership and access: the creating account owns the bucket. Access is controlled with IAM policies, bucket policies, and (for large-scale access management) S3 Access Points. Access control lists (ACLs) are a legacy mechanism; new buckets default to the BucketOwnerEnforced object ownership setting, which disables ACLs so that bucket policies and IAM are the single source of truth.

Objects

• Key: the unique identifier of an object within the bucket. The full key plus bucket name uniquely locates the object.

• Value: the actual data, from a few bytes to very large files. S3 documents a maximum object size and requires multipart upload above a certain threshold; always check the S3 documentation for the current limits, and use multipart upload for large or unreliable uploads because it retries failed parts instead of the whole object.

• Metadata: name/value pairs stored with the object, including system metadata such as content type and size plus custom metadata you supply.

• Version ID: when versioning is enabled, every write to the same key creates an immutable version with its own ID, so you can recover from accidental overwrites and deletes.

• Consistency: S3 provides strong read-after-write consistency for object PUTs and DELETEs, so a successful write is immediately visible to subsequent reads.

Bucket-Level Versus Object-Level Concerns

Key Features

• Storage classes: S3 offers tiers such as Standard, Intelligent-Tiering, and archive classes, so you can trade retrieval time and cost for infrequently accessed data.

• Lifecycle rules: automatically transition objects between storage classes or expire them, which is ideal for logs and backups with known retention needs.

• Protection: versioning, MFA delete for privileged changes, S3 Object Lock for write-once compliance data, and Block Public Access to prevent accidental exposure.

• Encryption: data is encrypted in transit over HTTPS and at rest with S3-managed keys, or customer-managed keys in AWS KMS for tighter control.

• Access without long-lived credentials: pre-signed URLs grant time-limited read or write access to a specific object, and S3 Access Points give applications dedicated named endpoints with their own policies.

Best Practices

• Follow least privilege: write bucket policies that grant specific actions on specific resources, and prefer IAM roles over long-lived access keys.

• Keep Block Public Access enabled unless you are deliberately hosting public content, and then scope the policy to exactly the objects that must be public.

• Enable versioning and default encryption on every bucket that holds data you cannot afford to lose.

• Use lifecycle rules and storage classes so aging data moves to cheaper tiers automatically.

• Use HTTPS endpoints, pre-signed URLs, or CloudFront for content delivery instead of opening buckets to the world.

• Treat ACLs as deprecated for new work and manage permissions through bucket policies, IAM, and Access Points.

Core Characteristics

DynamoDB stores data in tables, items, and attributes. Every table requires a partition key, which determines how data is distributed across partitions, and optionally a sort key for ordering items within a partition. Capacity modes are provisioned (you set read/write capacity units, optionally with DynamoDB auto scaling) and on-demand (capacity adjusts automatically with traffic). Other core features include global secondary indexes (GSIs) for querying non-key attributes, DynamoDB Streams for change data capture, Time to Live (TTL) for automatic item expiration, point-in-time recovery and continuous backups, and global tables for multi-region active-active replication.

Ideal Use Cases

Choose DynamoDB when your workload demands high throughput and low latency regardless of data volume, and when access patterns are known up front: - Real-time personalization: user profiles and recommendation lookups requiring fast, predictable reads. - Gaming: leaderboards, player state, and session data with high write concurrency. - IoT telemetry: ingesting large volumes of device and sensor data. - Session stores: web and mobile session management with single-digit millisecond latency. - Metadata and catalog data: flexible schemas for product catalogs, content metadata, and entity lookup. - Serverless backends: natural pairing with AWS Lambda and API Gateway for event-driven applications.

Comparison with Other AWS Data Stores

DynamoDB trades relational richness for scale and operational simplicity:

DynamoDB does support ACID transactions, but only within a single transaction request: TransactWriteItems and TransactGetItems can operate on up to 100 items in one all-or-nothing operation, which is far narrower than multi-statement relational transactions.

Security and Access Control

Access is governed by IAM policies following the least privilege principle. Prefer short-lived credentials by attaching IAM roles to compute resources such as EC2 instances, ECS tasks, EKS pods, or Lambda functions, rather than long-lived access keys. Policies should name exact actions (for example dynamodb:GetItem, dynamodb:PutItem, dynamodb:Query) and restrict resources to specific table ARNs. Encryption at rest is enabled by default using AWS owned keys; customer managed keys in AWS KMS are available where stricter key control or compliance auditing is required.

Best Practices and Pitfalls

• Partition key design: choose high-cardinality, evenly distributed keys to avoid hot partitions that throttle throughput.

• Query over Scan: prefer Query (key-based access) over Scan, which reads every item and consumes capacity proportional to table size.

• Model access patterns first: design keys, GSIs, and projections from the queries the application will actually run; refactoring key design later is costly.

• Caching: for read-heavy, latency-critical workloads, Amazon DynamoDB Accelerator (DAX) adds in-memory microsecond latency.

• Error handling: implement exponential backoff and jitter for ProvisionedThroughputExceededException and throttling errors.

• Schema evolution: DynamoDB is schema-less, so handle attribute versioning and optional attributes in application logic.

• Cost awareness: capacity mode, storage, GSI replication, and backup costs vary by region and change over time; check the DynamoDB documentation and Pricing page for current figures.

Core Architecture

• Amazon RDS: Manages traditional engines (MySQL, PostgreSQL, SQL Server, Oracle, MariaDB, Db2) where storage volumes are attached to a specific DB instance.

• Amazon Aurora: Compatible with MySQL and PostgreSQL but uses a distributed, fault-tolerant storage system shared across multiple compute instances.

• Decoupling: Aurora allows compute to scale independently of storage, enabling faster failover and read scaling without data copying.

Performance and Scalability

• Throughput: Aurora typically delivers higher throughput due to storage layer optimizations and parallel I/O.

• Scaling: Aurora storage grows automatically as data is written. RDS storage requires manual modification or a configured maximum with storage autoscaling.

• Read Replicas: Aurora supports low-latency read replicas that share the underlying storage volume, reducing replication lag compared to RDS logical or binary log replication.

Availability and Durability

• Replication: Aurora automatically replicates data across multiple Availability Zones in a single cluster. RDS Multi-AZ maintains a synchronous standby replica in a different Availability Zone for failover.

• Failover: Aurora failover is generally faster because the storage layer persists independently of the compute host; RDS failover depends on the engine and configuration.

• Backups: Both support automated backups and point-in-time recovery, but Aurora snapshots are incremental and faster due to the storage architecture.

Side-by-Side Comparison

Use Case Selection

• Choose RDS: When you need specific engine features (e.g., SQL Server, Oracle, Db2) or require maximum portability for hybrid cloud strategies.

• Choose Aurora: When building cloud-native applications requiring high availability, rapid scaling, and performance comparable to commercial databases at open-source engine cost.

• Migration: RDS is often simpler for lift-and-shift migrations of existing on-premises databases without refactoring; Aurora works well for greenfield applications or when you can accept MySQL/PostgreSQL wire compatibility.

Security and IAM Best Practices

• Authentication: Use IAM database authentication where supported to avoid password management, or retrieve credentials from AWS Secrets Manager.

• Access Control: Assign IAM roles to compute workloads (EC2, ECS, Lambda) instead of long-lived access keys, and grant only the specific RDS or Aurora actions and resources each role needs.

• Encryption: Enable encryption at rest using AWS KMS keys for both services; encryption must usually be chosen at creation time.

• Network: Deploy within private subnets of a VPC and restrict traffic with security groups.

• Infrastructure as code: Provision either service with AWS CloudFormation, AWS CDK or Terraform rather than the console, so security settings (KMS keys, security groups, Multi-AZ) are reviewed and repeatable.

How It Works

• You create a file system in a chosen AWS Region and attach it to a mount target in each subnet where clients need access. Each mount target gets its own security group, which must allow inbound NFS traffic (TCP port 2049) from the clients.

• Clients mount the file system using standard NFS (NFSv4.1 by default; NFSv4.0 is also supported). Linux instances mount with the nfs client, while Windows EC2 instances are not supported as NFS clients — Windows workloads should use Amazon FSx for Windows File Server instead.

• Data is stored redundantly across multiple Availability Zones by default (Standard deployment), or in a single AZ (One Zone deployment) at lower cost with lower durability.

• EFS offers four storage classes: Standard, Standard-Infrequent Access (IA), One Zone, and One Zone-Infrequent Access, plus Archive for rarely accessed data. Lifecycle management automatically moves files to IA or Archive after a period you configure, and EFS Intelligent-Tiering moves files between Standard and IA based on access patterns.

Key Features

• Elastic capacity: no pre-provisioning; usable storage scales to petabytes as files are written.

• Shared access: thousands of concurrent clients can read and write the same file system, with consistent NFS semantics.

• Performance modes: General Purpose (default, lowest latency) and Max I/O (highest aggregate throughput for large-scale parallel workloads).

• Throughput modes: Bursting (throughput scales with stored data), Provisioned (fixed throughput independent of data size), and Elastic (automatically scales throughput up or down with workload).

• Security: encryption at rest using AWS KMS keys, encryption in transit using TLS via the amazon-efs-utils package, and EFS Access Points to enforce per-application root directories, POSIX identities, and read-only permissions.

• Managed backups: automatic daily backups via AWS Backup, and EFS Replication for cross-Region or same-Region disaster recovery.

Common Use Cases

• Containers and serverless: persistent, shared volumes for Amazon ECS, Amazon EKS (through the Amazon EFS CSI driver), and AWS Lambda (via EFS access points) — ideal when pods or functions need a shared scratch space or shared configuration.

• Content management and web serving: WordPress, Drupal, and static site assets shared across an autoscaling fleet of web servers behind an Application Load Balancer.

• Dev/test and CI/CD: shared build artifacts, toolchains, and home directories for development teams.

• Big data and analytics: shared input and output for Hadoop, Spark, and machine learning training jobs that need a common file interface.

• Lift-and-shift: migrating on-premises applications that expect an NFS share without rewriting storage logic.

Choosing Between EFS, EBS, and FSx

Best Practices

• Follow least privilege: grant EC2 roles only the EFS actions they need (for example elasticfilesystem:ClientMount, elasticfilesystem:ClientWrite) scoped to the specific file system resource, and prefer IAM roles over long-lived access keys.

• Restrict mount-target security groups to known client security groups and CIDRs rather than opening NFS to 0.0.0.0/0.

• Use EFS Access Points for application-specific entry points, and enable encryption at rest and in transit for regulated workloads.

• Choose One Zone only for cost-sensitive, non-critical data; use Standard when multi-AZ durability matters.

• Monitor with Amazon CloudWatch metrics such as PercentIOLimit, DataReadIOBytes, and DataWriteIOBytes, and audit file-system API activity with AWS CloudTrail.

• Pricing depends on the storage class, throughput mode, and Region and changes over time; consult the Amazon EFS documentation and pricing page for current details instead of assuming fixed rates.

Data Durability Mechanisms

• Redundant Replication: Objects are stored across multiple facilities (Availability Zones) within a Region, protecting against hardware failure.

• Checksum Validation: S3 supports algorithms such as CRC32, CRC32C, SHA-1 and SHA-256 to verify data integrity during upload, download, and background scanning.

• Automatic Repair: If corruption is detected during integrity checks, S3 uses redundant copies to self-heal the object silently.

Availability Strategies

• Multi-AZ Distribution: Standard storage classes distribute data across at least three Availability Zones to survive single-location outages.

• Design Targets: S3 Standard is designed for 99.99% availability, so data remains accessible even during partial infrastructure failures; the service’s SLA is a separate uptime commitment, so treat the two as distinct.

• Cross-Region Replication: Optional configuration to copy data asynchronously to a different Region for disaster recovery and compliance.

Storage Classes and Trade-offs

Best Practices

• Enable Versioning: Protects against overwrites and deletions by keeping multiple variants of an object.

• Block Public Access: Ensure buckets are not publicly accessible unless explicitly required for static hosting.

• Use Lifecycle Policies: Transition objects to cheaper storage classes automatically to manage costs without manual intervention.

• Monitor with CloudWatch: Set alarms on replication metrics or error rates to detect issues early.

• Apply Least Privilege: Restrict bucket access using IAM roles and bucket policies rather than sharing long-lived credentials.

Workload Type and Use Case

Redshift targets Online Analytical Processing (OLAP). It aggregates data from multiple sources to support business intelligence, reporting, and machine learning. Typical queries scan millions of rows but return few columns. Amazon RDS targets Online Transaction Processing (OLTP). It powers application backends where frequent, short transactions (inserts, updates, deletes) must commit quickly with strong ACID guarantees. Using Redshift for high-frequency transactional writes will result in poor performance, while running full-table analytics on RDS can lock tables and degrade application responsiveness.

Storage Architecture and Query Performance

Redshift uses columnar storage, which stores data by column rather than row. This allows queries to read only the necessary columns, significantly reducing I/O and improving compression ratios. It employs Massively Parallel Processing (MPP) to distribute data and query load across multiple nodes. RDS engines (such as PostgreSQL, MySQL, or SQL Server) typically use row-based storage, optimized for retrieving complete records quickly. Redshift also supports features like Redshift Spectrum to query data directly in Amazon S3 without loading it into the cluster, whereas RDS is confined to its allocated storage volume.

Scaling Mechanisms

Scaling strategies differ fundamentally between the services: - Redshift: Scales horizontally by adding nodes to a cluster (resize) or using concurrency scaling to handle sudden spikes in query volume. Storage and compute are often coupled in provisioned clusters, though Redshift Serverless decouples them. - RDS: Primarily scales vertically by modifying the instance class (CPU/memory). Read scalability is achieved through read replicas, and storage can autoscale up to a defined limit. Failover is managed via Multi-AZ deployments.

Security and Access Management

Both services integrate with AWS IAM, but access patterns differ. For Redshift, you should grant IAM roles access to specific S3 buckets for COPY/UNLOAD operations rather than using long-lived credentials. For RDS, authentication is often database-native (username/password), though IAM database authentication is supported for some engines. Always follow least privilege: restrict network access via security groups, encrypt data at rest using AWS KMS, and avoid using the root account for routine administration. Quotas and limits vary by region and change over time, so consult the latest documentation.

Choosing the Right Service

Select Redshift when you need to analyze historical data, run complex joins across terabytes, or build a data lakehouse architecture. Select RDS when your application requires a reliable source of truth for user sessions, orders, or inventory where row-level locking and fast point-lookups are critical. In many architectures, both coexist: RDS handles transactional writes, and data is periodically ETL’d into Redshift for analysis.

What It Is

S3 Glacier is not a separate service you write to directly for the standard archive flow; you store objects in an S3 bucket and select a Glacier storage class, either at upload time or later through an S3 Lifecycle rule. AWS handles the underlying storage, which is the same 99.999999999% (11 nines) durability design used by the rest of S3, with data automatically distributed across multiple facilities.

• S3 Glacier Instant Retrieval: archive storage with millisecond retrieval, for data accessed a few times per quarter.

• S3 Glacier Flexible Retrieval (formerly “S3 Glacier”): the classic archive tier with retrieval times ranging from minutes to hours.

• S3 Glacier Deep Archive: the lowest-cost tier, for data that can wait half a day or more to come back.

Storage Classes at a Glance

Minimum storage durations matter operationally: deleting an object before that period ends triggers an early-deletion charge. Pricing, retrieval fees, and request costs vary by Region and change over time; consult the Amazon S3 pricing and documentation pages for current figures.

How Retrieval Works

With Flexible Retrieval and Deep Archive, you cannot simply GET an object and read it. You first initiate a restore, AWS copies the object into a frequently accessible storage class, and after the retrieval completes the object is readable for a retention period you select. Instant Retrieval objects can be read directly with normal S3 GET requests.

1. Initiate the restore, choosing a retrieval tier and a retention period (in days).

2. Wait for the tier’s stated retrieval window.

3. Read the restored copy as often as needed during the retention period.

4. Optionally change or extend the retention period before it expires.

For large objects, restore only the data you need and use byte-range GETs on the restored copy. For restoring thousands or millions of objects at once, use S3 Batch Operations rather than scripting individual restores.

Common Use Cases

• Regulatory and compliance archives: financial records, medical images, or audit logs that must be retained for years and are read only during an investigation.

• Backup and disaster recovery: off-site copies of databases, file systems, or application snapshots kept separately from the primary copy.

• Media and entertainment: raw footage, mastered assets, and finished content kept in archive and restored only when re-cut, re-mastering, or licensing requires it.

• Scientific and sensor data: satellite imagery, genomics outputs, and telemetry retained for future analysis.

• Log and telemetry retention: older log partitions moved out of hot analytics storage once their query frequency drops.

Choosing Between S3 Glacier and Other Options

Keep Glacier distinct from services such as AWS Backup, which orchestrates backups across services and can store copies in S3 Glacier, and from Amazon EBS snapshots, which are block-storage backups that can in turn be archived to S3 Glacier.

Best Practices

• Use S3 Lifecycle rules to transition objects to Glacier automatically as they age, rather than moving them manually.

• Enable S3 Object Lock in governance or compliance mode when you need write-once-read-many (WORM) controls and immutability for compliance; Object Lock must be enabled when the bucket is created and applies to objects in any S3 storage class, including Glacier. Do not confuse this with Glacier Vault Lock, which belongs to the original Glacier vault API.

• Estimate both storage and retrieval cost before bulk restores; large Deep Archive restores can be a significant expense.

• Prefer S3 Batch Operations for mass restore, and validate checksums after retrieval.

• Follow least privilege: grant s3:GetObject, s3:RestoreObject, and lifecycle configuration permissions only on the specific buckets and key prefixes required, and use roles or IAM Identity Center federation instead of long-lived access keys.

• Monitor archive growth and restore activity with S3 Storage Lens and CloudWatch so aging data is tiered before it becomes expensive.

Core Architecture

A VPC is scoped to a single AWS Region and uses a CIDR block (e.g., 10.0.0.0/16) to define the available IP address space. Resources launched inside a VPC, such as EC2 instances or Lambda functions configured with VPC access, communicate privately by default. You can extend connectivity to on-premises data centers using AWS Site-to-Site VPN or AWS Direct Connect, and link multiple VPCs via VPC Peering or AWS Transit Gateway.

Fundamental Components

• Subnets: Segments of the VPC CIDR range placed within a single Availability Zone. Public subnets have routes to an internet gateway, while private subnets do not.

• Route Tables: Sets of rules (routes) that determine where network traffic is directed based on destination IP addresses. Each subnet must be associated with a route table.

• Internet Gateway (IGW): A horizontally scaled, redundant gateway that allows communication between instances in the VPC and the internet.

• NAT Gateway: Enables instances in a private subnet to connect to the internet for updates while preventing inbound connections from the internet.

• Security Groups: Act as virtual firewalls for instances to control inbound and outbound traffic at the resource level (stateful).

• Network ACLs (NACLs): Operate at the subnet level as a stateless layer of defense, allowing or denying traffic based on rules.

Traffic Control and Routing

Traffic flow is determined by the route table associated with a subnet. For example, a route 0.0.0.0/0 pointing to an igw-id enables public access, whereas pointing to a nat-id enables outbound-only access for private subnets. DNS resolution within the VPC can be enabled via enableDnsSupport and enableDnsHostnames to resolve public hostnames to private IP addresses, facilitating internal service discovery without exposing endpoints publicly.

Common Use Cases

• Multi-tier applications: Place web servers in public subnets and application/database servers in private subnets, using security groups to restrict traffic between tiers.

• Hybrid connectivity: Use AWS Direct Connect or Site-to-Site VPN to link a VPC to an on-premises data center, extending the corporate network into AWS.

• Isolated environments: Deploy separate VPCs for development, staging, and production to limit blast radius and enforce distinct security policies.

Best Practices

• CIDR Planning: Reserve enough IP space for future growth, considering that AWS reserves five IP addresses in every subnet.

• Security Least Privilege: Configure security groups to allow only necessary ports and source IPs rather than using 0.0.0.0/0 broadly.

• Logging: Enable VPC Flow Logs to capture information about IP traffic going to and from network interfaces for auditing and troubleshooting.

• Isolation: Separate environments (dev, staging, prod) into distinct VPCs to limit blast radius during incidents.

Core Characteristics

• Security Groups (SGs): Associated with Elastic Network Interfaces (ENIs), so they protect individual instances (or, more precisely, each ENI). An ENI can reference multiple SGs, and a single SG can be attached to many ENIs. SGs track connection state, so return traffic is automatically allowed regardless of inbound rules.

• Network ACLs (NACLs): Associated with subnets, so every instance in the subnet is subject to them. They do not track state, so response traffic must be explicitly permitted by outbound rules. Each subnet must be associated with exactly one NACL, and every VPC starts with a default NACL that allows all inbound and outbound traffic.

Statefulness and Rule Evaluation

• Stateful vs stateless: Because SGs are stateful, allowing inbound HTTP (port 80) is enough — the outbound response is permitted automatically. NACLs are stateless, so a NACL that allows inbound HTTP also needs an outbound rule for the ephemeral port range used by the response (for example, the OS-chosen client port on the instance), which is a classic source of connectivity failures.

• Rule processing: SG rules are evaluated as a collective set — if any rule allows the traffic, it passes, and order does not matter. NACL rules are processed in ascending order by rule number (1–32766); the first match wins and evaluation stops.

• Allow vs deny: SGs cannot explicitly deny traffic; unmatched traffic is implicitly denied. NACLs can explicitly deny traffic, which makes them the right tool for blocking a specific IP range even when a broader allow rule would otherwise match.

Comparison Table

Choosing Between Them

• Defence in depth: Use SGs as the primary firewall for application-level access and NACLs as a secondary, coarse-grained layer — for example, a NACL that denies a known-bad IP range for an entire subnet.

• Least privilege: Restrict SG rules to the specific IP ranges and ports the application requires rather than 0.0.0.0/0, and prefer roles and other AWS identity controls over relying on network rules alone.

• Avoid NACL complexity: Because NACLs are stateless and numbered, complex rule sets easily produce ordering mistakes and broken return traffic; keep them simple.

• Do not rely on defaults: The default SG and default NACL are too permissive for production; create custom resources tailored to your architecture.

Core Components

Route 53 manages three primary functions. First, domain registration allows you to purchase and manage top-level domains (TLDs). Second, DNS routing stores records in hosted zones that map names to resources like Elastic Load Balancers, S3 buckets, or EC2 instances. Hosted zones can be public (visible to the internet) or private (visible only within specified Amazon VPCs). Third, health checking monitors the availability of resources and can trigger failover routing automatically. A hosted zone is a container for records defining how traffic is routed for a domain.

DNS Resolution Mechanism

When a user requests a domain, the query reaches a recursive DNS resolver. If the record is not cached, the resolver queries the root servers, then the TLD servers, and finally the authoritative name servers assigned to your hosted zone. Route 53 serves as the authoritative name server, responding with the IP address defined in the record set. Because Route 53 uses anycast routing, each query is automatically directed to the nearest DNS server, reducing latency. A global network of edge locations caches responses and routes users to the optimal endpoint based on configured policies.

Routing Policies

Different application architectures require specific traffic distribution strategies. - Simple Routing: Standard one-to-one mapping, suitable for single-resource applications. - Weighted Routing: Splits traffic based on assigned weights, useful for A/B testing or blue/green deployments. - Latency-Based Routing: Directs users to the AWS region with the lowest network latency. - Failover Routing: Switches traffic to a secondary resource when health checks detect a failure. - Geolocation Routing: Routes traffic based on the geographic location of the user. - Multivalue Answer Routing: Returns multiple healthy records in response to a query, distributing load across resources.

Best Practices and Security

Implement least privilege by attaching IAM policies that specify only required actions like route53:ChangeResourceRecordSets to specific hosted zone resources. Enable DNSSEC to protect against spoofing attacks by signing your zone data. Use health checks with appropriate thresholds to avoid false positives during brief network blips. Remember that Route 53 is distinct from load balancers; it directs traffic at the DNS level, whereas Application Load Balancers operate at the connection level. Avoid setting TTL values too low, as this increases query costs and latency, or too high, which delays propagation of changes.

Core Concepts

• CIDR block: A subnet takes a slice of the VPC’s IPv4 CIDR block (and optionally a /64 IPv6 block from the VPC’s IPv6 range). Subnet CIDRs within a VPC must not overlap.

• Availability Zone binding: A subnet is created in one AZ you choose (or one AWS picks). Resources such as EC2 instances, RDS databases and Lambda functions in a subnet run in that AZ; a subnet cannot span multiple AZs.

• Reserved addresses: For IPv4 subnets of /28 or larger, AWS reserves the first four addresses and the last one (network address, VPC router .1, VPC DNS .2, a reserved .3, and broadcast). A /28 subnet therefore offers 11 usable addresses, which is why AWS recommends larger blocks for most workloads.

• Minimum size: IPv4 subnets can be as small as /28; AWS recommends /27 or larger so you have room to add instances and elastic network interfaces.

Public, Private and Other Types

The “public” or “private” label is not a property of the subnet object itself; it comes from the route table attached to it. Another subnet attribute, auto-assign public IPv4 address, controls whether instances launched in it receive a public IP by default.

How Traffic Is Controlled

• Route tables: Each subnet is associated with one route table (or inherits the VPC main route table). This is what decides whether a subnet can reach the internet, a peered VPC, a transit gateway, or a VPC endpoint.

• Network ACLs: Stateless firewalls applied at the subnet boundary, evaluated by rule number, and useful for blocking specific CIDRs or protocols.

• Security groups: Stateful firewalls attached to individual network interfaces, the primary tool for allowing traffic to instances.

• VPC endpoints: Private subnets reach AWS services such as Amazon S3 and Amazon DynamoDB through gateway or interface endpoints instead of traversing the public internet.

Best Practices and Common Pitfalls

• One subnet per AZ per tier: Create public and private subnets in at least three AZs so services can be spread across AZs for high availability.

• Plan CIDR ranges up front: Reserve spare ranges in the VPC for future subnets, additional AZs, peering, or a migration to a larger VPC; overlapping CIDRs make VPC peering and transit gateway attachments fail.

• Keep data tiers private: Databases and internal services should have no internet gateway route; outbound patching traffic can egress through a NAT gateway in a public subnet.

• Use IPv6 where supported: You can attach an Amazon-provided or BYOIP IPv6 /56 to the VPC and assign /64 blocks to subnets, which helps workloads with very large address needs.

• Share subnets across accounts safely: Subnets can be shared through AWS RAM within an AWS Organizations organization, but the sharing account keeps ownership and the receiving accounts cannot modify or delete the subnet.

How It Works

• Edge locations: CloudFront maintains a worldwide network of points of presence (edge locations and regional edge caches). When a user requests content, the request is routed to the nearest edge location.

• Caching: If the content is cached there and still valid, CloudFront returns it immediately. If not, CloudFront fetches it from your configured origin (an S3 bucket, an Application Load Balancer, a Lambda function URL, or any HTTP/HTTPS server), caches a copy according to your cache behaviour rules, and returns it.

• Cache behaviours: Each distribution maps URL path patterns (for example /images/*) to an origin, with settings for allowed HTTP methods, viewer protocol policy, signed URLs/cookies, and cache key settings.

• Invalidation: When content changes at the origin before its TTL expires, you create an invalidation to remove specific paths from the cache, or use versioned file names (app.v12.js) so new requests bypass the old cache entries.

Key Benefits

• Lower latency: Serving requests from the nearest edge location avoids long round trips to your origin, which is the biggest win for global audiences.

• Reduced origin load and cost: Most requests are served from cache, which lowers compute, database, and data-transfer costs on your origin infrastructure.

• Security at the edge: CloudFront supports HTTPS with AWS Certificate Manager certificates, AWS WAF and AWS Shield Standard integration, and field-level encryption. AWS Shield Standard is applied automatically at no extra charge and helps protect against common DDoS attacks.

• Access control: Signed URLs and signed cookies let you restrict who can view private content, and Origin Access Control (OAC) is the current recommended way to restrict an S3 origin so only CloudFront can read it. The older Origin Access Identity (OAI) still exists but OAC is preferred, including for SSE-KMS buckets.

• Performance features: HTTP/3 and QUIC support, compression (Gzip and Brotli), and TLS 1.3 improve time to first byte and page load times.

• Programmability: CloudFront Functions (lightweight JavaScript for header manipulation, URL rewrites, access control) and AWS Lambda@Edge (Node.js/Python running in regional edge caches) let you customise behaviour close to users.

When to Use CloudFront Compared with Similar Services

Best Practices and Common Pitfalls

• Cache aggressively, but with the right cache key: Set long TTLs for immutable, versioned assets and use cache policies to control which headers, cookies, and query strings are included in the key. Including too many of them fragments the cache and reduces hit ratio.

• Use versioned object names for static assets rather than relying heavily on invalidations, which are slower and cost more.

• Follow least privilege: If CloudFront needs to read from S3, use OAC with a bucket policy that grants access only to your distribution’s ARN, rather than making the bucket public.

• Force HTTPS and use the latest TLS versions in both viewer and origin protocol policies.

• Watch for caching dynamic content by accident: API responses that vary per user usually need Cache-Control: no-cache or a cache policy scoped to the relevant authorisation header.

• Pricing varies by region and over time and depends on data transfer out and request counts; check the CloudFront pricing and quota pages in the AWS documentation for current figures instead of assuming fixed numbers.

Core Functionality and Health Checks

ELB sits between clients and compute resources, accepting incoming traffic and spreading it across targets registered in a target group. This prevents any single resource from becoming a bottleneck or a single point of failure. The service continuously monitors target health; if a target fails health checks, the load balancer stops sending traffic to it until it recovers. By integrating with Auto Scaling groups, ELB ensures capacity adjusts based on demand without manual intervention, maintaining performance during variable load.

Types of Load Balancers

AWS offers three main types suited for different layers of the OSI model: - Application Load Balancer (ALB): Operates at Layer 7 (HTTP/HTTPS). Supports host-based and path-based routing, making it ideal for microservices and containerized applications. - Network Load Balancer (NLB): Operates at Layer 4 (TCP/UDP/TLS). Designed for extreme performance and static IP addresses, suitable for latency-sensitive traffic. - Gateway Load Balancer (GWLB): Operates at Layer 3. Used to deploy, scale, and manage third-party virtual appliances like firewalls. - Classic Load Balancer (CLB): Legacy generation. Avoid using CLB for new architectures; use ALB or NLB instead.

Security and IAM Considerations

Access to manage ELB resources should follow least privilege principles. Instead of long-lived access keys, assign IAM roles to EC2 instances or Lambda functions that need to interact with ELB APIs. Security groups act as virtual firewalls for the load balancer, controlling inbound and outbound traffic. For ALBs, HTTPS listeners should use ACM-managed certificates to encrypt data in transit, ensuring compliance with security standards.

Best Practices

• Distribute Across AZs: Enable at least two Availability Zones for redundancy to survive zone failures.

• Use Access Logs: Enable access logs to analyze traffic patterns and troubleshoot issues without impacting performance.

• Stickiness Only When Necessary: Enable session stickiness only for stateful applications, as it can reduce load balancing efficiency.

• Monitor Metrics: Use Amazon CloudWatch to track metrics like RequestCount, TargetResponseTime, and HTTPCode_ELB_5XX.

• Avoid Root Account: Never use the root account for configuring load balancers; use IAM users or roles with specific permissions.

Core Concept and Architecture

AWS Direct Connect bypasses the public internet to link your data center or colocation environment directly to the AWS global network. You provision a physical Ethernet port at a Direct Connect location, which supports virtual interfaces (VIFs) for accessing public AWS services or private resources within a VPC. This dedicated path ensures that traffic does not traverse unpredictable internet routes, providing stable throughput and reduced jitter.

When to Use Direct Connect

Choose Direct Connect for hybrid cloud architectures where on-premises systems interact frequently with AWS resources. It is ideal for large-scale data migration, real-time replication, or workloads sensitive to latency variability, such as financial trading platforms or media processing pipelines. Organizations subject to strict compliance requirements also benefit because traffic remains off the public internet, reducing exposure to common network threats. If your bandwidth needs are consistently high, the cost model often becomes favorable compared to data transfer over the internet.

Comparison with Internet-Based VPN

While AWS Site-to-Site VPN runs over the public internet using IPsec tunnels, Direct Connect offers a private physical connection. The table below highlights key differences:

Use VPN for quick setup or lower bandwidth needs; use Direct Connect for stability and scale.

Implementation Best Practices

Always design for redundancy by provisioning connections across multiple locations or devices to avoid single points of failure. Apply least privilege IAM policies when managing connections via API or CLI, restricting actions like directconnect:CreateConnection to specific roles. Enable CloudWatch metrics to monitor connection state and BGP status. For security, associate connections with Direct Connect Gateways to access multiple VPCs across regions without traversing the internet. Avoid hardcoding credentials in automation scripts; use IAM roles attached to EC2 instances or CI/CD runners instead.

How Connectivity Is Established

A peering connection is a networking link between two VPCs. The owner of one VPC (the requester) sends a peering request to the owner of the other VPC (the accepter); the connection stays in the pending-acceptance state until the accepter accepts it, after which it becomes active. Once accepted, AWS infrastructure routes traffic between the VPCs internally. Neither VPC requires a virtual private gateway, NAT device, or separate VPN connection. The traffic stays within the AWS network, giving lower latency and higher security compared to internet-based links. Both intra-region and inter-region peering are supported, though inter-region connections may incur data transfer costs that vary by region and change over time.

Routing and DNS Configuration

Creating the connection alone does not enable traffic flow. You must manually update route tables in both VPCs to direct traffic destined for the peer’s CIDR block to the peering connection ID. Additionally, private DNS resolution requires specific attributes enabled on the connection. By default, instances in the accepter VPC cannot resolve the public DNS hostnames of resources in the requester VPC unless AllowDnsResolutionFromRemoteVpc is set to true on the requester side. Both VPCs must have EnableDnsSupport enabled, and the requester VPC must also have EnableDnsHostnames enabled.

Limitations and Design Constraints

VPC peering has strict constraints that impact architecture. The CIDR blocks of the peered VPCs cannot overlap; if they do, the connection cannot be established. Peering relationships are not transitive: if VPC A peers with VPC B, and VPC B peers with VPC C, VPC A cannot communicate with VPC C through VPC B. There is a limit on the number of active peering connections per VPC; check the VPC documentation for the current value and whether it can be raised.

Security and IAM Considerations

Security groups associated with instances act as a firewall even across peered connections. You must explicitly allow inbound traffic from the peer VPC’s CIDR range or reference security groups in the peer VPC, including across accounts. Network ACLs also apply at the subnet level. For IAM, users creating or accepting peering requests need permissions for ec2:CreateVpcPeeringConnection and ec2:AcceptVpcPeeringConnection. Following least privilege, these actions should be restricted to specific resources and granted to network administrators rather than general developers, with roles used instead of long-lived access keys.

Code Example

The following AWS CLI commands create a peering request in one account and accept it in another. Ensure credentials are configured securely via IAM roles or environment variables.

# Requester account: Create the peering connection
aws ec2 create-vpc-peering-connection \
    --vpc-id vpc-11223344 \
    --peer-vpc-id vpc-55667788 \
    --peer-region us-west-2

# Owner account: Accept the peering connection request
aws ec2 accept-vpc-peering-connection \
    --vpc-peering-connection-id pcx-abcdef12

# Requester account: Let the remote VPC resolve this VPC's DNS hostnames
aws ec2 modify-vpc-peering-connection-options \
    --vpc-peering-connection-id pcx-abcdef12 \
    --requester-peering-connection-options '{"AllowDnsResolutionFromRemoteVpc": true}'

Best Practices

For simple one-to-one connectivity, VPC peering is cost-effective and low latency. However, for hub-and-spoke architectures involving many VPCs, AWS Transit Gateway is preferred because it supports transitive routing and simplifies management. Avoid peering VPCs with overlapping CIDRs by planning IP address allocation centrally. Always verify security group rules to prevent unintended exposure of resources across the peering boundary.

API Types

• REST APIs: The original feature set, with the richest options for request/response mapping, validation, API keys, usage plans, per-client throttling, and AWS SigV4 signing of downstream requests. Best when you need fine-grained control or are exposing existing HTTP/REST backends.

• HTTP APIs: A newer, lighter option for serverless workloads. Lower latency and cost, simpler configuration, built-in JWT authorizer support, but fewer features than REST APIs (for example, no API keys or usage plans). This is the default choice for Lambda-first APIs that don’t need those features.

• WebSocket APIs: Persistent bidirectional connections for chat, notifications, dashboards, and real-time collaboration, with messages routed to Lambda, AWS services, or HTTP endpoints via route selection expressions.

Core Capabilities

• Traffic management: Throttling and quotas per stage or per API key protect backends from traffic spikes; this alone makes API Gateway a common rate-limiting layer in front of services.

• Authentication and authorization: IAM authorizers, Amazon Cognito user pools, Lambda authorizers (custom token or request based), and JWT authorizers on HTTP APIs. You can also keep APIs private so they are reachable only from your VPC.

• Transformation and validation: Request body and parameter validation, request/response mapping templates (REST APIs), and CORS handling without changing backend code.

• Observability and operations: Amazon CloudWatch metrics and logs, AWS X-Ray tracing, deployment stages with rollback, canary releases, and custom domain names with AWS Certificate Manager TLS certificates.

• Security posture: Integration with AWS WAF and AWS Shield, plus IAM policies that follow least privilege. Prefer temporary credentials assumed via IAM roles for clients inside AWS rather than long-lived access keys.

Typical Use Cases

• Serverless application APIs: The most common pattern — API Gateway routes requests to Lambda functions, so you scale without managing servers. This pairs naturally with Amazon DynamoDB, Amazon S3, and Amazon SQS.

• Modernizing legacy services: Expose existing on-premises or VPC-hosted services through REST or HTTP APIs using a VPC link, giving them a stable public contract while you migrate backends.

• Internal and partner APIs: Private APIs plus per-consumer usage plans and throttling let you share APIs with partners or internal teams under controlled quotas.

• Real-time applications: WebSocket APIs for chat, live dashboards, and push notifications, often with Lambda and Amazon EventBridge or Amazon SNS fan-out.

• Event-driven and hybrid architectures: API Gateway can push messages directly to Amazon SQS, Amazon SNS, EventBridge, or AWS Step Functions, so a synchronous client call can enter an asynchronous pipeline without a custom intermediary.

• BFF (backend-for-frontend) and aggregation layers: A single API endpoint can front several microservices, applying shared auth, logging, and rate limiting consistently.

Choosing Between Similar AWS Services

Best Practices

• Use HTTP APIs unless you specifically need REST API features such as usage plans or mapping templates.

• Secure endpoints with JWT or Lambda authorizers and avoid embedding credentials in clients; mobile and browser clients should use Amazon Cognito or an OIDC provider.

• Apply least-privilege IAM policies to any integration role, granting only the specific actions and resources the integration needs.

• Enable throttling and quotas at the stage level so one slow or misbehaving client cannot exhaust your backends.

• Use stages and canary deployments to release safely, and monitor with CloudWatch alarms on 4XX/5XX errors and integration latency.

• Front public APIs with AWS WAF and use TLS via custom domain names rather than exposing plain HTTP.

Core Architecture and Components

The service operates on a hub-and-spoke model. The central component is the transit gateway itself, which exists within a specific AWS Region. Networks connect to it via attachments: - VPC Attachments: Connects a VPC to the gateway using subnets specified in each Availability Zone. - VPN Attachments: Establishes IPsec tunnels from on-premises routers or third-party appliances. - Direct Connect Attachments: Links AWS Direct Connect gateways for high-bandwidth private connectivity. - Peering Attachments: Connects transit gateways across different AWS Regions or accounts.

Each attachment associates with a transit gateway route table. These tables determine where traffic flows next, similar to VPC route tables but scoped to the gateway. You can create multiple route tables to segment traffic, such as separating production from development environments. Gateways can also be shared securely across accounts in an AWS Organization using AWS RAM, avoiding duplicate hubs per account.

Advantages Over VPC Peering

Before transit gateways, organizations relied on VPC peering. While valid for small setups, peering does not support transitive routing. If VPC A peers with VPC B, and VPC B peers with VPC C, traffic cannot flow from A to C without additional configuration. Transit gateway solves this centrally.

Security and IAM Considerations

Access to manage the gateway should follow least privilege principles. Instead of using root account credentials or long-lived access keys, assign IAM roles to users and services. Policies should specify exact actions like ec2:CreateTransitGateway and specific resource ARNs. You can integrate with IAM Identity Center for centralized workforce access management across AWS accounts. Network security is enforced through security groups and network ACLs on the attached VPCs, as the gateway itself operates at the network layer without inspecting payload content.

Best Practices for Implementation

• Segmentation: Use multiple route tables to isolate traffic domains rather than relying on a single default table.

• Shared Services: Deploy a shared services VPC attached to the gateway for common resources like DNS resolvers or logging tools.

• Monitoring: Enable CloudWatch metrics to monitor packet drop rates and attachment state changes.

• Automation: Use Infrastructure as Code tools like AWS CloudFormation or Terraform to deploy gateway configurations consistently across accounts.

• Quota Management: Service limits vary by Region and change over time; request increases via the Service Quotas console if needed rather than assuming fixed defaults.

When to Use This Service

Use AWS Transit Gateway when you have multiple VPCs across several accounts or Regions that need to communicate frequently. It is ideal for enterprise migrations involving hybrid cloud setups where on-premises data centers must access cloud resources securely. For simple two-VPC connections, VPC peering might suffice, but any architecture expecting growth should adopt the gateway model early to avoid refactoring later.

Core Components

• KMS keys: The primary resources in KMS. A KMS key contains the key material (or references key material imported from your own HSM) plus metadata such as the key ID, ARN, policy and rotation settings. Symmetric KMS keys (AES-256) are the default and are used by most integrated AWS services; asymmetric KMS keys (RSA or ECC key pairs) and HMAC keys are available for signing and verification use cases.

• Key policies: A resource-based policy attached directly to a KMS key that defines who and what may use it. IAM policies alone are usually insufficient: for most operations both the key policy and the IAM policy must allow the action.

• Hardware Security Modules (HSMs): KMS uses FIPS 140-2 Level 2 validated HSMs to protect the confidentiality and integrity of key material. Plaintext key material never leaves the HSMs.

AWS Managed vs. Customer Managed Keys

How Envelope Encryption Works

KMS typically employs envelope encryption to protect large amounts of data efficiently. 1. The application requests a data key from KMS, referencing a KMS key. 2. KMS generates a unique data key and returns both a plaintext copy and a copy encrypted under the KMS key. 3. The application encrypts the data locally using the plaintext data key. 4. The plaintext data key is discarded from memory, and the encrypted data key is stored alongside the ciphertext. 5. To decrypt, the application sends the encrypted data key to KMS, which returns the plaintext data key for local decryption.

This pattern minimizes use of the KMS key itself, keeps the plaintext data key short-lived, and lets you encrypt large payloads without sending all of the data to KMS.

Security and IAM

Follow least privilege when configuring access. Grant specific actions such as kms:Encrypt, kms:Decrypt, kms:GenerateDataKey or kms:CreateGrant only to the roles that require them, and scope resources to specific key ARNs rather than *. Avoid granting kms:* on production keys. Ensure the key policy explicitly allows the IAM role to use the key, because KMS evaluates both the IAM policy and the key policy. Prefer roles assumed by your workloads over long-lived access keys, and never use root account credentials.

Best Practices

• Enable automatic rotation: Configure customer managed keys to rotate annually; AWS managed keys rotate automatically.

• Use grants for temporary access: For short-lived access (for example, a Lambda function or an AWS service acting on your behalf), use grants instead of repeatedly editing policies.

• Monitor usage: Enable AWS CloudTrail to log all KMS API calls for auditing and anomaly detection; KMS also emits CloudWatch metrics and events for key state changes.

• Avoid hardcoding identifiers: Keep key IDs and ARNs out of source code; reference key aliases, environment variables or AWS Systems Manager Parameter Store instead.

• Plan for deletion: KMS keys are irreversible after the mandatory waiting period, so use aliases to rotate keys and test deletion on non-production keys first.

Core Functionality

CloudTrail operates by capturing management events (control-plane API calls such as creating or deleting resources) and optionally data events that track resource operations like Amazon S3 object access or Lambda function invocations. These events are delivered as log files to an Amazon S3 bucket you specify, where they can be stored indefinitely or integrated with Amazon CloudWatch Logs and Amazon EventBridge for alerting. Each log entry contains critical context including the identity of the caller, the source IP address, the timestamp, and the specific request parameters used during the API call. As a current alternative to file-based delivery, CloudTrail Lake lets you ingest events into a managed data store and query them directly with SQL-style filters, without managing log files.

Key Benefits

• Security Analysis: Enables detection of unauthorized activity by reviewing user actions and identifying anomalies in access patterns.

• Compliance Auditing: Helps meet regulatory requirements by maintaining a tamper-evident record of configuration changes and access events, with optional log file validation to prove files were not altered after delivery.

• Operational Troubleshooting: Allows engineers to trace the sequence of events leading to an operational issue or an unexpected resource state change.

• Non-Repudiation: Records the identity, source, and parameters of each call so actions can be attributed to a specific principal, which supports accountability across teams.

Service Distinctions

It is important to distinguish CloudTrail from other monitoring services to use the right tool for each scenario.

While CloudWatch alerts you on metric thresholds and Config tracks resource configuration history and compliance rules, CloudTrail is the definitive source for identity and access auditing within the account.

Best Practices

• Enable Multi-Region Trails: Create a trail that applies to all AWS Regions so global activity is captured, including Regions you may start using later.

• Encrypt Log Files: Use an AWS KMS key (customer managed key) to encrypt log files in S3, so only authorized roles can decrypt and read audit data.

• Apply Least Privilege: Restrict access to CloudTrail logs and configuration with IAM policies that grant only specific actions, such as cloudtrail:LookupEvents for security auditors, rather than broad administrator access.

• Protect Trail Configuration: Prevent users from stopping or deleting trails by combining S3 bucket policies with IAM conditions that deny modifications to the trail itself, and prefer assuming a short-lived role over using long-lived access keys or the root account.

How It Works

The core unit of configuration is a web ACL (web access control list), which you attach to the resource you want to protect. Inside the web ACL you add rules, and each rule matches requests against conditions such as the request path, query string, headers, body, source IP address, or labels added by previous rules. Every web ACL has a default action that applies when no rule matches.

• Web ACL: the container for rules and the default action; associated with one or more protected resources.

• Rules: statements that inspect request components. A rule can combine multiple conditions with AND, OR, and NOT logic.

• Rule groups: reusable collections of rules that you can share across web ACLs.

• Managed rule groups: pre-built rule sets maintained by AWS or AWS Marketplace sellers, subscribed to rather than written by you.

• Labels: metadata that rules attach to matching requests, letting later rules act on what earlier rules detected.

Rule Actions

Each rule can apply one of the following actions to a matching request:

Common Use Cases

• Protection against common web exploits: the AWS Managed Rules rule groups include curated protections for the OWASP Top 10, covering attacks such as SQL injection, cross-site scripting (XSS), and cross-site request forgery. This is the most common starting point.

• Rate limiting: rate-based rules count requests from a single client IP address or forwarded IP over a five-minute window and block clients that exceed a threshold you define, mitigating brute-force login attempts, credential stuffing, and simple volumetric floods.

• Bot and scraper control: the AWS Bot Control managed rule group categorizes request traffic and helps block or throttle automated bots, while the CAPTCHA and Challenge actions let you interpose a check for suspicious clients.

• Geo and IP filtering: you can allow or block requests by country of origin using the GeoMatchStatement, or maintain explicit allow and deny lists of IP addresses and CIDR ranges.

• Application-specific rules: custom rules enforce your own business logic, for example allowing requests to /api/* only with a required header, blocking requests whose user agent matches a known abusive client, or limiting the size of request bodies.

• API protection: when attached to Amazon API Gateway or AppSync, AWS WAF filters malicious or malformed calls before they consume your backend, complementing but not replacing authorization inside your application.

Integrations and Best Practices

AWS WAF integrates closely with AWS Shield Advanced for expanded DDoS protection and with AWS Firewall Manager, which lets you centrally deploy and manage WAF policies across many accounts and resources in an AWS Organizations organization. Request sampling, CloudWatch metrics, and full logging to Amazon CloudWatch Logs, Amazon S3, or Amazon Data Firehose support monitoring and rule tuning.

Recommended practices:

• Start in Count mode: run new rules with the Count action and review the sampled requests before switching to Block, so you avoid blocking legitimate users.

• Layer managed and custom rules: place AWS Managed Rules groups first for broad coverage, then add custom rules for application-specific requirements.

• Order rules deliberately: rules are evaluated in the order you configure them, so place high-priority exceptions and rate-based rules where they take effect before broader blocks.

• Follow least privilege for the WAF configuration role: grant only the WAF actions and resources needed to manage web ACLs, and prefer roles over long-lived access keys.

• Test with real traffic patterns: use CloudWatch metrics and access logs to tune thresholds, because overly broad rules are a common cause of false positives.

Capacity in a web ACL is measured in Web ACL capacity units (WCU), and limits vary by resource type and region; check the AWS WAF documentation for current values.

Purpose

The goal of AWS Shield is to keep applications available when attackers attempt to overwhelm or disrupt them. DDoS attacks commonly flood a target with traffic (volumetric attacks), exhaust resources with malformed or slow requests (protocol attacks), or abuse application logic at layer 7. Shield integrates with AWS network services such as Amazon CloudFront, Route 53, Elastic Load Balancing (Application, Network and Gateway Load Balancers), AWS Global Accelerator, and Amazon API Gateway, so protection is applied close to where traffic enters AWS without changing application code.

Tiers of Protection

• AWS Shield Standard: automatically enabled at no additional charge for all AWS accounts and customers. It defends against the most common, frequently occurring network and transport layer (layer 3 and 4) attacks, such as SYN floods and UDP reflection attacks, targeting CloudFront, Route 53, and Elastic Load Balancing resources.

• AWS Shield Advanced: an optional, paid add-on that provides enhanced protection for higher-value or higher-risk workloads. It adds application-layer (layer 7) DDoS protection, more sophisticated detection and real-time metrics, dedicated support from the Shield Response Team (SRT) during suspected events, and financial protection through DDoS cost protection for qualifying charges caused by a DDoS attack. Shield Advanced also integrates with AWS WAF and AWS Firewall Manager for centralized rule management.

Key Features

• Always-on detection and automatic inline mitigation: traffic is continuously inspected, and identified attack traffic is mitigated inline without manual intervention.

• Layer 3/4 protection: handles volumetric and protocol attacks at the network and transport layers.

• Layer 7 protection: Shield Advanced, together with AWS WAF, can mitigate HTTP-level attacks such as HTTP floods; Shield Advanced can automatically apply mitigating WAF rules in response to detected events.

• Visibility and diagnostics: near-real-time metrics, event dashboards, and Amazon CloudWatch integration help teams understand what is happening during an attack. Shield Advanced provides additional attack detail and health-based detection.

• Expert support: Shield Advanced customers get 24/7 access to the Shield Response Team, which can help analyze and mitigate sophisticated or novel attacks.

• Integration with AWS WAF and AWS Firewall Manager: WAF provides custom layer 7 rules (rate limiting, IP matching, bot control), while Firewall Manager can roll out WAF and Shield Advanced protections across many accounts and resources.

Shield Standard vs Shield Advanced

Best Practices

• Use Shield Standard as a baseline, but treat it as one layer of defense, not a complete strategy.

• Put AWS WAF in front of public web endpoints to add custom layer 7 rules, such as rate-based statements and geographic or IP-based filtering.

• Consider Shield Advanced for high-value workloads where downtime or scaling costs during an attack would be unacceptable; pricing and coverage details change over time, so confirm current terms in the AWS Shield documentation.

• Follow least privilege when granting IAM permissions to view or manage Shield resources, and restrict shield:AssociateDRTLogBucket and similar administrative actions to a small set of operators.

• Combine Shield with resilient architecture: CloudFront or Global Accelerator for edge absorption, multiple Availability Zones, autoscaling, and Route 53 health checks so that mitigation and recovery work together.

• Test and monitor: review Shield findings and CloudWatch metrics regularly, and use AWS WAF logging to tune rules before a real attack rather than during one.

How It Works

• Secret storage: a secret is a named object that holds secret text (a simple string or structured JSON key/value pairs) plus metadata: an Amazon Resource Name (ARN), description, encryption key reference, tags and one or more versions.

• Encryption at rest: every secret is encrypted with an AWS KMS key, either the AWS managed key for the service (aws/secretsmanager) or a customer managed key that you control. Retrieval always requires secretsmanager:GetSecretValue; when a customer managed key is used, the caller must also have kms:Decrypt on that key, whereas the AWS managed key is decrypted by the service on the caller’s behalf.

• Access from code: applications call GetSecretValue using an AWS SDK (AWS SDK for JavaScript v3, boto3, AWS SDK for .NET, and others) and pass the secret identifier or ARN. The SDK returns the secret payload over TLS; nothing is logged in plaintext by the service.

• Identity-based access: retrieval is authorised by IAM policies attached to the workload’s execution role (for example an Amazon ECS task role or a Lambda execution role), which is the preferred pattern for workloads running on AWS. IAM Identity Center or federated identities are the recommended way for human users, and temporary credentials are used instead of long-lived access keys.

Key Features

• Automatic rotation: Secrets Manager can rotate a secret on a schedule by invoking an AWS Lambda rotation function. Managed rotation is built in for several Amazon RDS engines (MySQL, PostgreSQL, MariaDB, Oracle, Microsoft SQL Server), Amazon Aurora, Amazon Redshift, Amazon DocumentDB and Amazon ElastiCache for Redis, with no rotation code required.

• Versioning and staging labels: each rotation produces a new version labelled AWSCURRENT, while the previous version moves to AWSPREVIOUS, so you can roll back quickly if a rotation breaks an application.

• Cross-account and cross-region access: you can share a secret with another AWS account by attaching a resource-based policy to the secret, and replicate secrets into other regions for low-latency reads and disaster recovery.

• Auditability: every API call is recorded in AWS CloudTrail, and you can validate configuration with AWS Config rules or detect secrets committed to source control with Amazon CodeGuru Security.

• Integration: secrets are consumed natively by Amazon RDS, Aurora, Redshift, ECS, EKS, Lambda, AWS CloudFormation, AWS CDK and AWS AppConfig, and can be mounted into Amazon EKS pods via the AWS Secrets and Config Provider (ASCP) for the CSI driver.

Secrets Manager Compared with Alternatives

Choose Secrets Manager when rotation, versioning and database integration matter; use Parameter Store when you mainly need encrypted configuration values at lower cost, and keep the two distinct rather than mixing them for the same secret.

Best Practices

• Follow least privilege: grant secretsmanager:GetSecretValue on a specific secret ARN, not secretsmanager:* on all resources, and scope the KMS key policy to the same callers.

• Prefer roles over keys: let workloads assume a role through their runtime identity instead of embedding long-lived access keys.

• Reference by ARN or name in code, and load the secret at startup or cache it with a sensible refresh interval rather than calling GetSecretValue on every request.

• Enable rotation for database credentials and set a schedule appropriate to the application; test the rotation function in a staging environment first.

• Tag secrets and use tag-based IAM conditions so access scales with your environment without broad wildcards.

• Never log secret values: mask or omit the SecretString field in application logs, and avoid returning it in error messages or traces.

What They Cover

AWS Compliance Programs validate the security, availability, confidentiality, and processing-integrity controls of the AWS cloud infrastructure, not the customer’s own applications or data. Audits are performed by independent third parties (for example, AICPA-affiliated CPAs for SOC reports, BSI for ISO certifications, accredited assessors for FedRAMP and PCI DSS). Each program has a defined scope: which services, which Regions, and which control framework were assessed. Scope expands over time, so the authoritative source is the “AWS Services in Scope by Compliance Program” page in the AWS Compliance documentation.

Key Programs and Frameworks

• SOC 1 / SOC 2 / SOC 3: AICPA-based attestations. SOC 2 covers the Trust Services Criteria (security, availability, confidentiality, processing integrity, privacy) and is the most commonly requested report by AWS customers.

• ISO/IEC certifications: including ISO/IEC 27001 (information security management), ISO/IEC 27017 (cloud security controls), ISO/IEC 27018 (protection of personal data in public clouds), and ISO/IEC 22301 (business continuity).

• PCI DSS: AWS is a validated Level 1 service provider; the Attestation of Compliance (AOC) supports customers building PCI-compliant workloads.

• FedRAMP: US government program with authorizations at FedRAMP Moderate and High impact levels; relevant to US public-sector workloads.

• HIPAA / HITECH: AWS offers Business Associate Agreements (BAAs) and publishes a list of HIPAA-eligible services; signing a BAA is required for workloads processing protected health information.

• Regional and national programs: IRAP (Australia), C5 (Germany), ENS (Spain), MTCS (Singapore), K-ISMS (South Korea), CS Mark (Japan), and Cyber Essentials Plus (UK), among others.

How Customers Use Them

1. Identify which services and Regions their workload needs, then confirm those are in scope for the required program.

2. Download the relevant reports and certificates from AWS Artifact (SOC reports, ISO certificates, PCI AOC, and others), accepting any applicable non-disclosure terms.

3. Sign required agreements, such as a HIPAA BAA obtained through AWS Artifact, before processing regulated data.

4. Map the AWS-side controls described in the reports to their own control framework (for example, NIST CSF, CIS Benchmarks, ISO 27001) instead of re-auditing AWS themselves.

5. Implement and evidence the customer-side controls required by the shared responsibility model.

Supporting AWS Services

• AWS Artifact: central portal for compliance reports, certificates, and agreements.

• AWS Audit Manager: continuously collects evidence from AWS services to prepare for audits, mapping it to frameworks such as PCI DSS, HIPAA, CIS AWS Foundations Benchmark, and NIST standards.

• AWS Config: records resource configuration history and evaluates resources against conformance packs and rules.

• AWS Security Hub: aggregates security findings and checks against compliance standards.

• AWS CloudTrail and Amazon GuardDuty: provide audit logging and threat detection that support customer-side control evidence.

Shared Responsibility and Best Practices

Compliance is shared: AWS is responsible for the security of the cloud (facilities, hardware, hypervisor, managed service controls), while the customer is responsible for security in the cloud (data classification, encryption settings, IAM, network configuration, patching of guest OS and applications). A common pitfall is assuming that using a HIPAA-eligible or PCI-scoped service automatically makes a workload compliant — it does not; the customer’s own configuration must also meet the standard. Best practice is to check service and Region scope first, apply least-privilege IAM, encrypt data at rest and in transit, enable CloudTrail and Security Hub, and use Audit Manager to keep evidence current rather than assembling it at audit time.

Core Functionality: Inventory and Change Tracking

AWS Config uses a configuration recorder to capture changes to supported resources within a region. Every change results in a configuration item stored in an Amazon S3 bucket, creating a timeline of who changed what and when. This inventory is critical for forensic analysis during security incidents or operational troubleshooting. You can stream these changes to Amazon EventBridge for real-time processing or query them via the AWS Config console and API.

Compliance Evaluation with Managed and Custom Rules

Compliance is enforced through Config Rules, which are either managed by AWS or custom-built using AWS Lambda. Managed rules cover common best practices, such as ensuring S3 buckets are not publicly accessible or checking if encryption is enabled on EBS volumes. Custom rules allow you to encode specific organizational policies. Each rule evaluates resources periodically or in response to configuration changes, marking them as COMPLIANT or NON_COMPLIANT. This status is visible in dashboards and can trigger automated responses.

Integration for Automated Governance

To move from detection to remediation, AWS Config integrates with other services. Non-compliant events can trigger AWS Lambda functions via Amazon EventBridge to automatically fix issues, such as closing a security group port. Additionally, findings can be sent to AWS Security Hub for a consolidated view of security standards across accounts. AWS Systems Manager Automation can also be invoked to execute runbooks based on Config alerts, ensuring governance policies are enforced without manual intervention.

Code Example: Checking Compliance Status

You can programmatically retrieve compliance data using the AWS SDK for Python (boto3). The following snippet demonstrates how to check the compliance status of a specific Config Rule, assuming proper IAM permissions are configured.

import boto3

def check_config_compliance(rule_name):
    client = boto3.client('config')
    response = client.describe_compliance_by_config_rule(
        ConfigRuleNames=[rule_name]
    )
    for compliance in response['ComplianceByConfigRules']:
        print(f"Rule: {compliance['ConfigRuleName']}")
        print(f"Compliance Type: {compliance['Compliance']['ComplianceType']}")
        print(f"Timestamp: {compliance['Compliance']['ComplianceTimestamp']}")

# Ensure IAM policy allows config:DescribeComplianceByConfigRule
check_config_compliance('restricted-ssh-rule')

Best Practices and Security Considerations

Implement least privilege by granting only necessary IAM actions, such as config:DescribeComplianceByConfigRule, rather than full administrative access. Enable AWS Config in all regions where resources exist, as configuration data is regional. Use AWS Organizations to aggregate data from multiple accounts into a central account for unified governance reporting. Avoid storing sensitive data in configuration items by ensuring resources themselves are configured securely. Regularly review Config Rules to ensure they align with evolving compliance requirements and remove unused rules to reduce noise.

Core Capabilities

GuardDuty operates as an agentless service, meaning no software needs to be installed on EC2 instances or containers. It analyzes billions of events across multiple AWS data sources to generate security findings. Each finding includes a numeric severity score from 0 to 10, categorized as Low (1–3), Medium (4–6), or High (7–10), along with detailed context such as affected resources, actor IP addresses, and recommended remediation steps. This allows security teams to prioritize responses based on potential impact rather than manually correlating logs.

Analyzed Data Sources

The service ingests and analyzes specific log types to detect various threat vectors. Enabling GuardDuty automatically activates analysis for core logs, while additional protections can be turned on for specific workloads.

• VPC Flow Logs: Detects unauthorized network access, port scanning, and communication with known malicious IPs.

• AWS CloudTrail Management Events: Identifies unauthorized API calls, privilege escalation, and changes to security configurations.

• DNS Logs: Reveals attempts to communicate with command-and-control servers or domains associated with malware.

• Amazon S3 Data Events (via CloudTrail): Monitors for unusual access patterns indicating data exfiltration or unauthorized bucket access.

• EKS Audit Logs: Detects suspicious activity within Kubernetes clusters, such as unauthorized pod execution.

Integration and Automated Response

GuardDuty findings are not isolated; they integrate deeply with the AWS security ecosystem. Findings can be sent to Amazon EventBridge to trigger automated remediation workflows, such as isolating an instance via Lambda or updating a security group rule. Additionally, findings aggregate into AWS Security Hub, providing a unified view of security alerts across multiple AWS accounts and services. This integration supports centralized security operations centers (SOCs) managing multi-account AWS Organizations structures.

Best Practices for Deployment

To maximize effectiveness, organizations should follow specific deployment strategies aligned with least privilege principles.

• Enable Across All Regions: Threats often target unused regions; enable GuardDuty in every region where you operate.

• Use AWS Organizations: Designate a delegated administrator account to manage GuardDuty centrally rather than configuring each member account individually.

• Apply Least Privilege: IAM roles used to access GuardDuty findings or configure the service should have specific actions like guardduty:ListFindings rather than full administrative access.

• Suppress Known Noise: Use finding suppression rules to reduce alert fatigue from known benign activities, such as approved penetration testing IPs.

• Combine with Other Services: Use GuardDuty for threat detection alongside AWS Inspector for vulnerability scanning and AWS Config for compliance monitoring, as these services address different security layers.

Common Pitfalls

A frequent mistake is enabling GuardDuty but ignoring findings due to volume. Without integration into a ticketing system or Security Hub, critical alerts may be overlooked. Another pitfall is relying solely on GuardDuty for compliance; it detects threats but does not enforce configuration compliance like AWS Config does. Finally, ensure CloudTrail logging is enabled globally, as GuardDuty depends on CloudTrail management events to detect identity-based threats.

Core Purpose

Security Hub acts as a centralized pane of glass for security alerts and compliance monitoring. Instead of logging into multiple services to check for vulnerabilities or compliance deviations, security teams can view all findings in one place. It continuously monitors your environment and automatically runs compliance checks against enabled standards, reducing the operational overhead of managing security across a cloud infrastructure.

Key Features

• Standards Management: You can enable standards such as AWS Foundational Security Best Practices, CIS AWS Foundations Benchmark, and PCI DSS. Each standard consists of controls that are checked against your resources to generate compliance findings.

• Finding Aggregation: Findings are normalized into the AWS Security Finding Format (ASFF). This allows consistent filtering, sorting, and analysis regardless of the source service.

• Automation and Remediation: Security Hub supports automated remediation using AWS Systems Manager Automation documents or AWS Lambda functions via Amazon EventBridge rules triggered by specific findings.

• Multi-Account Support: Using AWS Organizations, you can designate an administrator account to aggregate findings from member accounts, enabling centralized governance and access via IAM Identity Center.

Service Integrations

Security Hub integrates natively with several AWS security services. Amazon GuardDuty provides threat detection, Amazon Inspector assesses application vulnerabilities, and Amazon Macie discovers sensitive data. Additionally, AWS Partner Network (APN) partners can send findings to Security Hub, allowing third-party tools to feed into the same workflow. It is distinct from AWS Config, which tracks resource configuration history, though Config rules can feed findings into Security Hub.

Best Practices

• Least Privilege: Grant IAM principals only the specific actions they need, such as securityhub:GetFindings for analysts and securityhub:EnableSecurityHub for administrators. Avoid using root account credentials.

• Enable Standards Early: Enable standards like AWS Foundational Security Best Practices immediately upon setup to establish a baseline posture.

• Automate Remediation: Configure EventBridge rules to trigger Lambda functions for auto-remediation of common issues, such as closing public S3 buckets.

• Regular Review: Review findings regularly and suppress false positives using finding fields rather than disabling controls, to maintain audit integrity.

Remember that specific standards and integrations may vary by Region, so consult the AWS documentation for the latest availability.

What AWS Artifact Provides

• Compliance reports: AWS Artifact provides access to third-party audit reports and certifications for AWS services, commonly including frameworks such as SOC reports, ISO certifications, PCI DSS, and FedRAMP authorizations. The exact reports available depend on the AWS service, Region, and audit cycle.

• Agreements: Customers can review and accept select online agreements through AWS Artifact, such as nondisclosure agreements and industry-specific agreements, for example a Business Associate Agreement used in HIPAA-related workloads.

• Service and Region scoping: Reports identify which AWS services and Regions were in scope for a particular audit, which helps customers determine whether the AWS services they use are covered by the certification they need.

• Auditable evidence: The reports are produced by independent auditors and can be shared with a customer’s internal compliance team or external auditors, subject to the terms of the applicable agreement.

Why It Matters

• Supports due diligence: Organizations can verify AWS security and compliance claims using auditor-produced documents instead of relying only on marketing statements.

• Reduces audit friction: Compliance teams can download current evidence directly, rather than requesting it through separate channels.

• Strengthens compliance programs: Customers can map AWS-operated controls to their own control framework and focus their assessment effort on the controls they are responsible for.

• Clarifies shared responsibility: AWS Artifact evidence covers AWS infrastructure and services, while customers remain responsible for configuring their resources, protecting their data, and meeting obligations applicable to their applications.

How Customers Use It

1. Identify the compliance framework and the AWS services and Regions used by the workload.

2. Open AWS Artifact and locate the report that matches that framework and audit period.

3. Accept any required terms or nondisclosure agreement before accessing the report.

4. Download the report and review the scope to confirm it covers the relevant AWS services.

5. Share the report with authorized auditors or compliance stakeholders, and retain it as evidence.

6. Combine AWS-side evidence with customer-side evidence from services such as AWS Audit Manager, AWS Security Hub, and AWS Config.

AWS Artifact Compared With Related Services

Access Control and Best Practices

• Use federation, not long-lived keys: Give compliance and security team members access through AWS IAM Identity Center or another federated identity provider instead of creating long-lived IAM access keys.

• Apply least privilege: Grant AWS Artifact access only to users who need to view or accept compliance documents, and prefer roles over standing credentials.

• Protect downloaded reports: Many reports are confidential and governed by nondisclosure terms; store them in controlled internal systems and avoid public sharing.

• Verify scope before relying on a report: Confirm the report covers the AWS service, Region, and time period relevant to the workload.

• Keep AWS and customer evidence separate: Use AWS Artifact for AWS-operated controls and AWS Audit Manager or Security Hub for controls the customer must implement and monitor.

Core Purpose

CloudWatch provides a single place to observe AWS workloads without operating your own monitoring infrastructure. Many AWS services automatically send metrics to CloudWatch, while the CloudWatch agent and embedded SDK calls add custom data from EC2 instances, on-premises servers, containers, Lambda functions, and applications. CloudWatch is operational telemetry: “what is happening and how healthy is it.” It is distinct from AWS CloudTrail, which records API activity and account events (“who did what”), and from AWS X-Ray, which traces requests through distributed services, although CloudWatch ServiceLens integrates X-Ray traces with CloudWatch metrics and logs.

Key Components

• Metrics: Time-ordered data points stored under a namespace with a metric name, dimensions, timestamp, value, and unit. AWS services publish many metrics by default; custom metrics can be published through the CloudWatch API or agent. Standard-resolution metrics are typically one minute, while high-resolution custom metrics can be as frequent as one second.

• Alarms: Rules that evaluate a metric over time and move between OK, ALARM, and INSUFFICIENT_DATA states. Alarms can send Amazon SNS notifications, trigger Auto Scaling policies, or run EC2 recovery actions, making them the main bridge between monitoring and automated remediation.

• CloudWatch Logs: A log ingestion and storage service organized into log groups and log streams. It supports retention settings, metric filters to derive metrics from log text, subscription filters to forward log events to destinations such as Kinesis, Lambda, or Amazon OpenSearch Service, and CloudWatch Logs Insights for ad hoc querying of log data.

• Dashboards: Customizable web pages containing graphs, text, alarm status, and other widgets. Dashboards can display metrics across accounts and Regions when CloudWatch cross-account observability is configured, and AWS also provides automatic dashboards for many services.

• Amazon EventBridge integration: The functionality previously known as CloudWatch Events is now delivered through Amazon EventBridge, which uses the same event stream and remains the service to use for event-driven routing of AWS API activity and scheduled rules.

• Extended observability features: CloudWatch Synthetics runs canaries that periodically test endpoints and APIs from a browser-like runtime; CloudWatch RUM collects client-side telemetry from web applications; ServiceLens combines metrics, logs, traces, and service maps; Container Insights, Lambda Insights, and Contributor Insights add specialized views for common workloads.

How the Components Fit Together

A typical flow starts with data collection: resources and applications emit metrics and logs. Those metrics are visualized on dashboards and evaluated by alarms. When an alarm changes state, it notifies operators through SNS or invokes an automated action. Logs provide the detailed context behind a metric change, and Logs Insights helps find the relevant lines after an incident. Synthetics and RUM extend visibility beyond the AWS network by testing what end users actually experience.

Choosing CloudWatch Appropriately

Use CloudWatch when you need AWS-native monitoring, tight integration with AWS services, and simple alerting on metrics and logs. Choose Amazon Managed Grafana or third-party tools when you want to correlate AWS data with broad multi-cloud dashboards. Choose Amazon OpenSearch Service when log analytics requires heavier full-text search and long-term analysis at scale. CloudWatch is often the starting point for AWS observability because it requires minimal setup and is already receiving data from most AWS services.

Core Purpose

Amazon CloudWatch functions as an observability service that collects metrics, logs, and events to help you visualize and alarm on the health of your AWS resources and applications. It enables automated responses, such as scaling EC2 instances when CPU usage spikes. In contrast, AWS CloudTrail is a governance service that records API calls made across your AWS account, providing a history of who did what, when, and from where. CloudWatch answers “Is my system slow?” while CloudTrail answers “Who changed my security group?” or “Was this deletion authorized?”

Data Types and Retention

• CloudWatch Metrics: Numerical data points collected over time, such as CPU utilization or request latency, stored based on retention policies you configure.

• CloudWatch Logs: Timestamped records from applications or AWS services, useful for debugging errors or analyzing application behavior.

• CloudTrail Events: JSON-formatted records of API activity, categorized into management events (control plane operations) and data events (resource operations like S3 object access).

• Retention: CloudWatch log retention is configurable indefinitely. CloudTrail retains event history in the console for a defined period, while long-term storage requires exporting trails to Amazon S3, where retention depends on your bucket lifecycle policies.

Comparison Overview

Best Practices and Security

• Enable CloudTrail Everywhere: Configure trails to apply to all AWS Regions to ensure comprehensive audit coverage, including global services like IAM and STS.

• Least Privilege for Access: Grant IAM roles only the specific permissions needed to read CloudWatch metrics or CloudTrail events, avoiding long-lived access keys.

• Encrypt Audit Logs: Enable AWS KMS encryption for CloudTrail log files to protect sensitive audit data at rest in Amazon S3.

• Integrate for Security: Use CloudWatch Logs Insights to analyze CloudTrail logs exported to CloudWatch, combining auditing with real-time alerting on suspicious API activity.

• Avoid Root Usage: Never use the root account for programmatic access; ensure CloudTrail captures root activity to monitor for unauthorized privileged use.

How It Works

• Managed instances: Any Amazon EC2 instance, on-premises server, edge device, or virtual machine running the SSM Agent and registered to your AWS account. On-premises servers are enrolled through hybrid activation, so they are managed with the same tooling as EC2.

• IAM roles: Each managed instance assumes an instance profile that grants the permissions the agent needs (commonly the AWS-managed AmazonSSMManagedInstanceCore policy, narrowed further for least privilege).

• Documents: Operations are defined by SSM documents — JSON/YAML runbooks such as AWS-RunShellScript, AWS-RunPatchBaseline, or your own custom Automation runbooks — that the service executes on targets.

• Targeting: You select instances by instance IDs, resource groups, or tags, so a single operation can touch one server or thousands.

Key Capabilities

• Fleet Manager: A browser-based console for viewing instances, drilling into file systems, processes, and logs, and starting interactive sessions.

• Run Command: Executes commands or scripts across many managed instances at once, with output streamed back to the console, Amazon S3, or CloudWatch Logs.

• Session Manager: Browser- or CLI-based interactive shell access without opening inbound SSH/RDP ports, without a bastion host, and with IAM-based authorization plus optional session logging to S3 and CloudWatch Logs.

• Patch Manager: Applies OS and application patches on a schedule or on demand using patch baselines and maintenance windows, and reports patch compliance.

• Inventory and Compliance: Collects software, hardware, and network configuration data from instances and reports compliance against desired states.

• State Manager and Automation: State Manager keeps instances in a defined configuration (for example, ensuring an agent is always installed); Automation runs multi-step runbooks for provisioning, patching, snapshotting, or remediating drift.

• Parameter Store: A hierarchical store for plain-text and SecureString configuration data, encrypted with AWS KMS, referenced by name from applications, CloudFormation, Lambda, and containers.

• Explorer, OpsCenter, and Incident Manager: Aggregated operational dashboards, a central place to investigate and resolve operational issues (OpsItems), and guided incident response with escalation plans and chat integration.

Common Use Cases

• Mass operations without SSH: Replacing ad hoc SSH loops with auditable, IAM-authorized command execution — for example, installing a package or rotating a local config file across every web-tier instance tagged Role=web.

• Patch compliance programs: Scheduling monthly patching through maintenance windows, with pre- and post-tasks (snapshots, validation) defined in an Automation runbook.

• Secure access and auditability: Replacing bastion hosts and long-lived SSH keys with Session Manager, where every session is authorized by IAM and logged to encrypted S3 buckets.

• Configuration and secrets distribution: Storing connection strings, feature flags, and license keys in Parameter Store SecureString values instead of baking them into AMIs or source code. For credentials that must rotate automatically, use AWS Secrets Manager; Parameter Store does not perform rotation.

• Hybrid and multicloud operations: Managing on-premises VMware vSphere VMs, bare-metal servers, and edge devices with the same runbooks used for EC2.

• Remediation automation: Triggering Automation runbooks from Amazon EventBridge or Amazon Config to fix drift, snapshot volumes, or restart unhealthy services.

Choosing Between Similar Tools

Best Practices

• Follow least privilege: Grant the instance profile only the SSM actions it needs, and restrict ssm:SendCommand and ssm:StartSession in user policies to specific document ARNs and specific instance resources rather than allowing "Resource": "*".

• Prefer roles over long-lived keys: Instances authenticate through their instance profile; hybrid activations use activation codes only for enrollment.

• Prefer Session Manager over SSH: It removes inbound exposure and ties access to IAM identity plus logging.

• Encrypt and centralize output: Send command output and session logs to an encrypted S3 bucket and CloudWatch Logs, and protect them with restrictive bucket policies.

• Test runbooks in a non-production account or resource group before fleet-wide rollout, and pin document versions rather than always using $DEFAULT.

• Combine services deliberately: Use Systems Manager for operations and configuration, Amazon EventBridge for event routing, and AWS Config for compliance evaluation — they integrate, but each serves a distinct purpose.

Core Architecture

EventBridge operates on four main components: - Events: JSON objects representing a state change or occurrence, containing a source, detail-type, and detail payload. - Event Buses: The conduit for events. Every AWS account has a default bus, but you can create custom buses to isolate workloads or partner events. - Rules: Logic that evaluates incoming events against an event pattern. If matched, the rule triggers one or more targets. - Targets: The destination services that process the event, such as AWS Lambda, Step Functions, Kinesis, or HTTP endpoints via API Destinations.

Evolution from CloudWatch Events

Originally launched as CloudWatch Events, the service was rebranded to Amazon EventBridge to reflect expanded capabilities. The legacy CloudWatch Events functionality is now part of EventBridge, specifically focused on AWS service events. Key enhancements include: - SaaS Integration: Direct ingestion of events from partners like Datadog, PagerDuty, or Shopify without custom middleware. - Custom Event Buses: Allowing multi-tenant architectures where different applications use separate buses rather than sharing the default bus. - Event Schema Discovery: Automatically inferring JSON schemas from incoming events to assist with code generation and validation.

Distinguishing from SNS and SQS

While often used together, these services solve different problems: - Amazon SNS: A pub/sub messaging service optimized for fan-out to many subscribers (email, SMS, HTTP). It pushes messages immediately. - Amazon SQS: A queue service designed for decoupling producers and consumers with buffering and retry logic. It pulls messages. - Amazon EventBridge: An event bus focused on routing based on content patterns and integrating disparate systems via a standardized schema. It is ideal for reacting to state changes rather than simple message delivery.

Facilitating Event-Driven Applications

EventBridge enables event-driven architectures by: - Decoupling producers and consumers: Producers emit events to a bus without knowing which targets will process them. - Centralized routing: A single rule can fan out to multiple targets, or multiple rules can process the same event differently. - Standardized schema: The schema registry ensures consumers understand the event shape, reducing integration errors. - Serverless scaling: Targets such as Lambda scale automatically, so the event bus can handle bursts without capacity planning.

IAM and Security Best Practices

Access to EventBridge should follow least privilege principles. Instead of using root account credentials or long-lived access keys, assign IAM roles to services that need to put events or manage rules. - PutEvents: Grant events:PutEvents only on specific event buses. - Rule Management: Restrict events:PutRule, events:PutTargets, and events:RemoveTargets to administrative roles. - Resource Policies: Use event bus resource policies to allow external AWS accounts or SaaS partners to write to your custom buses securely.

Common Pitfalls

• Overly Broad Patterns: Matching too many events can increase costs and trigger unnecessary invocations. Specificity in detail-type is recommended.

• Retry Logic: EventBridge retries failed targets automatically, but ensure targets are idempotent to handle duplicate deliveries.

• Dead Letter Queues: Configure an SQS dead-letter queue for rule targets to capture events that fail processing after all retries.

What It Was

OpsWorks provided a higher-level abstraction than launching EC2 instances individually. You modelled an application as a stack, and inside the stack you defined layers such as load balancer, web/app server, database and caching layers. Each layer carried its own recipes or manifests, instance settings, scaling rules and deployment logic, so the same blueprint could be applied across many instances.

There were three offerings:

• OpsWorks Stacks: the original, fully managed service with built-in integration for EC2 (and historically Elastic Load Balancing, RDS, MySQL, Memcached, Ganglia and HAProxy). You could also register on-premises instances.

• OpsWorks for Chef Automate: a managed Chef Automate server (Chef Infra plus workflow/compliance tooling) in your VPC, with the underlying server maintained by AWS.

• OpsWorks for Puppet Enterprise: an equivalent managed Puppet Enterprise master.

How It Helped

• Configuration as code: infrastructure and application configuration lived in Chef cookbooks or Puppet manifests under version control, reviewed and tested like application code.

• Layer-based modelling: the stack/layer hierarchy expressed real application architecture, making the topology visible and reusable instead of hidden in runbooks.

• Lifecycle automation: OpsWorks ran the appropriate recipes or manifests at setup, configuration, deployment, scaling, shutdown and other lifecycle events, so new instances converged to a known state automatically.

• Deployment orchestration: you could deploy an app revision to a whole layer or stack with a single action, and roll back to a previous revision.

• Elasticity and operations: integration with Auto Scaling and CloudWatch, plus CloudTrail logging and IAM-based access control, supported day-to-day operation of the fleet.

• Hybrid support: registering existing on-premises or EC2 instances let a single configuration baseline span environments.

When Each Offering Fit

Current Status and Replacements

AWS ended support for all three offerings — OpsWorks Stacks and OpsWorks for Puppet Enterprise reached end of support in March 2024, and OpsWorks for Chef Automate in May 2024 — so it should not be chosen for new projects. Modern equivalents, chosen by workload shape:

• AWS Systems Manager: the primary successor for fleet operations and configuration at scale — patching, inventory, runbooks, Session Manager and State Manager for defining and enforcing desired state.

• AWS Elastic Beanstalk or AWS App Runner: when you want a managed platform that handles deployment and scaling for a web application.

• Amazon ECS, Amazon EKS or AWS Lambda: for containerised or serverless workloads, where immutable deployment replaces in-place configuration management.

• AWS CloudFormation, AWS CDK or Terraform: for provisioning infrastructure as code, typically combined with a user-data script, SSM State Manager or a container image instead of a configuration-management agent.

• If you still need Chef or Puppet, run it yourself on EC2 or use the vendors’ SaaS offerings.

Choosing Today

Use configuration management when servers are long-lived and their drift must be corrected continuously; use immutable infrastructure — containers, Lambda or auto-replacing instances — when you can replace a server instead of repairing it. In practice, most teams now combine IaC for provisioning with SSM or container platforms for runtime state, which is why OpsWorks has been retired rather than extended.

How It Works

AWS Config first records the configuration state of supported resources as configuration items. A rule is then evaluated against those items:

1. You create or activate a rule, choosing a managed rule (maintained by AWS) or a custom rule backed by a Lambda function you own.

2. AWS Config evaluates the rule when a resource’s configuration changes (event-based evaluation) and on a schedule you define (periodic evaluation), with allowed intervals of 1, 3, 6, 12, or 24 hours.

3. For each resource it checks, AWS Config records a compliance result of COMPLIANT, NONCOMPLIANT, or NOT_APPLICABLE.

4. Results are available in the AWS Config console and API, and can be delivered to an S3 bucket and to Amazon SNS or EventBridge for alerting.

5. You can attach a remediation action, usually an AWS Systems Manager Automation runbook, to automatically fix noncompliant resources.

Key Features

• Managed rules: AWS supplies and maintains rules for common checks, such as whether S3 buckets are encrypted, whether EBS volumes are encrypted, whether security groups do not allow unrestricted SSH access, or whether CloudTrail logging is enabled.

• Custom rules: when no managed rule fits, you write a Lambda function that receives the configuration item and returns the compliance status, which lets you encode organisation-specific standards.

• Remediation: you can trigger SSM Automation documents automatically or on demand, for example to detach a noncompliant security group or enable encryption settings where the service supports remediation.

• Conformance packs: a collection of rules and remediation actions, packaged as a YAML template, that you can deploy across accounts and Regions to standardise a control set.

• Multi-account and multi-Region aggregation: an aggregator lets a central account view compliance data from many accounts and Regions, which is the usual pattern for a landing zone or control tower environment.

AWS Config Rules Compared With Related Controls

Config rules are detective and corrective, not preventive: they report and can remediate drift, but they do not block the change itself.

Advantages and Limitations

• Advantages: continuous, automated auditing without manual sampling; a single compliance view across resource types; deep integration with SSM Automation, Security Hub, and EventBridge; and a large library of managed rules that require no code.

• Limitations: rules only evaluate resources that AWS Config records and supports; custom rules require you to write, test, and maintain Lambda code; remediation is not possible for every resource setting; and evaluation frequency means a short-lived noncompliant state may not be captured by periodic rules.

Best Practices

• Prefer managed rules and only build custom rules when the check is specific to your organisation.

• Enable AWS Config through an infrastructure-as-code tool (CloudFormation, CDK, or Terraform) so rules, recorders, and delivery channels are versioned and repeatable.

• Follow least privilege: grant the Config role only the read and write actions it needs, and use Lambda execution roles scoped to the specific API calls the custom rule performs.

• Send results to Security Hub or EventBridge so alerts reach the right team, and attach remediation runbooks for the most common drift cases.

• Use conformance packs with aggregation for multi-account estates, so every account is measured against the same baseline.

Core Components

• Templates: JSON or YAML files that define the desired state of resources, including configuration details, metadata, and dependencies between resources.

• Stacks: A single unit of deployment created from a template; all resources in a stack are managed together as a group.

• Change Sets: A preview of how proposed template changes will affect the running stack before execution, allowing safe updates without immediate impact.

• Drift Detection: A feature that identifies configuration differences between the template definition and the actual live resources, highlighting unauthorized manual changes.

• StackSets: Extends stacks to deploy across multiple AWS accounts and Regions simultaneously, useful for organization-wide governance.

Key Benefits

• Automation and Speed: Entire environments can be provisioned in minutes rather than days, enabling rapid iteration, testing, and continuous integration pipelines.

• Consistency: Using the same template across development, staging, and production eliminates configuration drift caused by manual steps or human error.

• Safe Rollbacks: If a resource creation fails during an update, CloudFormation can automatically roll back the stack to its previous stable state to prevent partial deployments.

• Dependency Management: The service automatically resolves resource dependencies (e.g., creating an EC2 instance only after its VPC and security groups are ready) based on template declarations.

• Cost Efficiency: You pay only for the AWS resources created; there is no additional charge for using CloudFormation itself, though pricing varies by region and usage.

CloudFormation vs Alternatives

CloudFormation is ideal for native AWS workflows and strict compliance requirements, while AWS CDK is recommended for developers preferring imperative programming languages to generate templates. Terraform suits organizations requiring a single tool across multiple cloud providers.

Best Practices and Security

• IAM Roles: Use service roles with least privilege permissions for CloudFormation operations instead of long-lived access keys or root account credentials. The role should allow only specific actions required by the template resources.

• Parameterization: Use template parameters and AWS Systems Manager Parameter Store to inject environment-specific values without hardcoding secrets or passwords in templates.

• Nested Stacks: Break complex architectures into nested stacks to improve reusability, simplify templates, and limit update blast radii during changes.

• Drift Detection: Run drift detection regularly to identify unauthorized manual changes that violate the declared template state and compromise security posture.

• Protection: Enable stack termination protection on production environments to prevent accidental deletion via console or CLI.

When designing templates, always specify explicit resource policies and avoid wildcard permissions to adhere to least privilege principles. Remember that quotas and limits vary by region and change over time, so consult the latest AWS documentation for current constraints before architecting large-scale deployments.

Core Components

• Products: Approved service offerings, typically defined by one or more provisioning artifacts such as AWS CloudFormation templates.

• Portfolios: Collections of products that are shared with specific principals, AWS Organizations organizational units, or accounts.

• Constraints: Policies attached to portfolios or products that control launch behavior, such as which IAM role is assumed, which parameters are allowed, or whether updates are permitted.

• Provisioning artifacts: Versioned representations of a product; a new artifact is created when the underlying template or configuration changes.

Operational Model

1. Administrators register products and upload provisioning artifacts, often from a secure artifact store.

2. They build portfolios and share them with target accounts or users, optionally using AWS Organizations for centralized distribution.

3. Constraints are applied: a launch constraint delegates provisioning to a dedicated IAM role, while a template constraint restricts parameter values or instance types.

4. End users discover approved products in the Service Catalog console, CLI, SDK, or an integrated portal and launch them.

5. Service Catalog invokes the underlying engine—usually CloudFormation—using the launch role, so resources are created with governed permissions rather than the user’s personal credentials.

6. Administrators publish new provisioning artifacts for patches or architecture changes and can retire older ones, guiding users toward current versions without automatically modifying already deployed stacks.

Key Benefits

• Governance and Compliance: Only pre-reviewed architectures can be deployed, reducing shadow IT and configuration drift.

• Self-Service Provisioning: Developers obtain standardized infrastructure in minutes instead of waiting for manual operations requests.

• Lifecycle Control: Versioned provisioning artifacts let teams distribute updates and deprecate outdated configurations.

• Cost and Tag Governance: Tag options applied at the portfolio level automatically attach cost-center, environment, or owner tags to launched resources.

Service Catalog vs AWS CloudFormation

Best Practices and IAM

• Least Privilege IAM: Allow users to access only the portfolios they need; avoid granting broad servicecatalog:* permissions.

• Launch Roles: Use launch constraints so provisioning assumes a dedicated role with narrowly scoped permissions, never the end user’s credentials.

• Tag Options: Define tag options at the portfolio level to enforce consistent cost allocation and ownership metadata.

• Avoid Root Account: Manage Service Catalog with administrative roles that have multi-factor authentication enabled, not the AWS root account.

• Secure Artifact Storage: Keep templates in encrypted, access-controlled buckets or repositories and reference them through private URLs.

What It Does

Trusted Advisor runs automated checks against your account’s configuration and resource inventory. Each check produces a result with a status — for example, a check that flags an Amazon S3 bucket with open access — plus a description of the risk and recommended remediation steps. Key characteristics:

• Agentless and continuous: it evaluates resources in enabled AWS Regions using data AWS already collects, so there is nothing to deploy.

• Advisory by design: it identifies issues and links to remediation guidance, but it does not change your resources for you.

• Integrated with AWS Health: recommendations appear in the AWS Health Dashboard, and accounts on qualifying support plans can use the AWS Support API to retrieve check results programmatically.

• Account- and organization-scoped: you can review findings for a single account or, in AWS Organizations, get an aggregated view across member accounts.

The Five Check Categories

• Cost Optimization: identifies idle or underused resources, such as Amazon EC2 instances with low CPU utilization, unattached Amazon EBS volumes, and reservations or savings plans that could reduce spend.

• Performance: flags configurations that can limit throughput, such as high-load EC2 instances or database instances approaching capacity.

• Security: detects common misconfigurations, including publicly accessible S3 buckets, overly permissive IAM access keys or passwords, and missing MFA on the root account.

• Fault Tolerance: checks for resilience gaps, such as Amazon EBS snapshots missing for volumes, Auto Scaling groups without multiple Availability Zones, or no RDS backups.

• Service Limits: warns when you are approaching account-level service quotas, such as VPC or IAM limits, so you can request increases before deployment is blocked.

How You Use It

1. Open Trusted Advisor in the AWS Management Console and browse by category or severity.

2. Review each check’s affected resources, explanation, and recommended action.

3. Remediate directly in the relevant service console, or exclude resources you have intentionally configured that way.

4. Monitor the AWS Health Dashboard or retrieve results through the AWS Support API for automation and alerting.

5. Re-run or wait for scheduled refreshes to confirm the check returns to a healthy status.

Trusted Advisor Compared With Related Tools

Best Practices and Limitations

• Prioritize by severity: fix high-severity security findings first, since those carry the most immediate risk.

• Automate the follow-up: use AWS Health plus Amazon EventBridge to route Trusted Advisor findings to ticketing, chat, or remediation pipelines instead of relying on manual console reviews.

• Use least privilege: grant only the permissions needed to read findings, for example by attaching the AWS-managed AWSSupportAccess policy to a role rather than to individual users, and avoid long-lived access keys.

• Aggregate at scale: with AWS Organizations, enable the organization view so a central security or FinOps team can see findings across all member accounts.

• Remember it is advisory: Trusted Advisor does not remediate and does not replace AWS Config rules, Security Hub, or the Well-Architected Tool; use it as a continuous, broad health check alongside those tools.

• Coverage varies by support plan: all accounts receive a set of core checks at no additional charge, while the full check set and API access depend on your AWS Support plan; check the AWS Trusted Advisor documentation for current coverage details, since quotas, Regions, and check availability change over time.

How It Differs from the Public Service Health Dashboard

The public AWS Service Health Dashboard shows the general status of each service by Region for all customers. The Personal Health Dashboard adds account context: - Scope: Public dashboard = service and Region level; Personal Health Dashboard = account and resource level. - Visibility: An underlying host maintenance event appears in the Personal Health Dashboard only if your EC2 instance runs on that host; the public dashboard may show no issue at all. - Actionability: Personal events include affected-resource lists and recommended next steps, whereas public events are primarily informational.

How Personalized Guidance Is Delivered

• Event metadata: Each event carries a start time, end time, status, service, Region, and a description of the type of impact.

• Affected resources: The dashboard uses the AWS Health API to list the specific resources tied to an event, such as instance IDs or volume IDs, so you do not have to guess which workloads are at risk.

• Prescriptive guidance: Event details include recommended actions, such as rebooting an instance before a scheduled retirement, migrating to a healthy host, or updating a configuration. This guidance is curated by AWS and updated as the event evolves.

• Proactive delivery: Scheduled-change events are published to Amazon EventBridge. From there, rules can route them to Amazon SNS, chat tools, or ticketing systems before the maintenance window starts.

Integration and Automation

• EventBridge rules: Match health events by service, Region, or event type code and trigger AWS Lambda, Step Functions, or SNS topics. A common pattern is to snapshot EBS volumes or fail over a database when a scheduled maintenance event arrives.

• AWS Organizations: A delegated administrator can view health events across member accounts, giving security and operations teams a central view without granting broad account access.

• Custom tooling: The AWS Health API operations DescribeEvents, DescribeEventDetails, and DescribeAffectedEntities let you embed health data into internal dashboards or runbooks.

Best Practices

• Route events automatically: Do not rely on engineers opening the dashboard manually; push health events into your incident-management pipeline through EventBridge.

• Use least-privilege IAM: Grant read-only health permissions such as health:DescribeEvents, health:DescribeEventDetails, and health:DescribeAffectedEntities on the required resources, and use roles rather than long-lived access keys.

• Validate runbooks: Test automated responses in a non-production account so a health event does not trigger an untested failover.

• Avoid root credentials: Monitor and respond with IAM roles assigned to operators or automation, never the root account.

• Combine with other signals: Health events explain AWS-side impact; pair them with CloudWatch alarms and application logs to distinguish service issues from application faults.

How It Works

• Queues as the integration point: A producer (a web application, an AWS Lambda function, an EC2 instance, or any HTTP client using the AWS SDK) calls SendMessage to add a message. A consumer calls ReceiveMessage to retrieve it and DeleteMessage once it has been handled successfully.

• Pull-based consumption: Consumers poll the queue rather than receiving pushed notifications, which lets each consumer control its own throughput and retry behaviour.

• Visibility timeout: When a message is received, it becomes invisible to other consumers for that duration. If the consumer fails or crashes and never deletes it, the message reappears so another consumer can retry it. This is the core of SQS reliability.

• At-least-once delivery: Standard queues may occasionally deliver a message more than once, so consumers should be idempotent. FIFO queues provide exactly-once processing and preserve message order.

Queue Types

Throughput and message-size quotas vary by region and change over time; check the Amazon SQS documentation for the current limits, including the maximum message size (256 KiB by default, extendable to 2 GiB using the SQS Extended Client Library for Java or the equivalent message-payload offloading pattern).

Common Use Cases

• Decoupling producers from consumers: An e-commerce checkout service accepts an order, enqueues a message, and returns immediately, while separate workers handle payment, inventory, and email notifications.

• Smoothing traffic bursts: A flash sale generates a surge of requests; the queue holds them while a fixed number of workers drains it at a steady rate.

• Batching and background processing: Workers pull up to 10 messages per ReceiveMessage call and process them together, reducing per-request overhead.

• Reliable retries: A failing downstream dependency no longer causes request loss — the message simply returns to the queue after the visibility timeout expires.

• Event-driven architectures: SQS is frequently placed between an event source and Lambda, or used as a target for Amazon EventBridge rules and SNS topic subscriptions, so each consumer has its own retry logic and dead-letter handling.

SQS vs Related Services

SQS is often confused with Amazon SNS and Amazon EventBridge, but they serve different roles:

• Amazon SQS: message queue — one consumer group pulls messages; no push notifications.

• Amazon SNS: pub/sub notification — messages are pushed to subscribers such as SQS queues, Lambda, HTTP endpoints, or emails. A common pattern is SNS fanning out to multiple SQS queues when the same event must be processed independently by several systems.

• Amazon EventBridge: an event bus with content-based routing, schema management, and native integration with AWS services and third-party SaaS partners. Choose EventBridge when you need to route events by their content to many targets; choose SQS when you simply need durable, decoupled buffering between two components.

Best Practices

• Use dead-letter queues: Move messages that fail processing after a configured number of receives to a DLQ for inspection instead of retrying them forever.

• Enable long polling: Set a non-zero WaitTimeSeconds (up to 20) so consumers wait for messages instead of issuing repeated empty ReceiveMessage calls, which reduces cost and latency.

• Design idempotent consumers: Duplicate delivery can happen even with FIFO queues if a DeleteMessage call fails, so use a deduplication ID or a business-level unique key.

• Secure the queue: Enable server-side encryption with an AWS KMS key, use HTTPS endpoints, and grant IAM permissions scoped to specific queue ARNs and actions (for example, sqs:SendMessage restricted to the queue a producer needs) rather than granting sqs:* on all resources.

• Prefer roles over long-lived access keys: Applications running on EC2, ECS, EKS, or Lambda should assume an IAM role attached to the compute resource instead of embedding credentials.

• Monitor with CloudWatch: Track ApproximateNumberOfMessagesVisible, ApproximateAgeOfOldestMessage, and NumberOfMessagesSent/NumberOfMessagesReceived to detect consumer lag and scale workers accordingly.

Delivery Model

• Amazon SQS (Simple Queue Service): Producers send messages to a queue; consumers poll the queue, process a message, and delete it. Messages persist in the queue until consumed or until the configurable retention period expires, which provides buffering and resilience against consumer downtime.

• Amazon SNS (Simple Notification Service): Producers publish messages to a topic, and SNS pushes each message to every subscription attached to that topic. Supported subscription endpoints include AWS Lambda, Amazon SQS, Amazon Kinesis Data Firehose, HTTP/HTTPS webhooks, email, SMS, and mobile push notifications.

Message Ordering and Delivery Semantics

• SQS Standard queues offer at-least-once delivery with best-effort ordering and nearly unlimited throughput per API action; SQS FIFO queues provide first-in-first-out ordering, message deduplication, and exactly-once processing within a message group.

• SNS Standard topics provide at-least-once delivery and best-effort ordering per subscription; SNS FIFO topics preserve message ordering and support deduplication, and they pair naturally with SQS FIFO queues for ordered fan-out.

When to Use Each

• Choose SQS when you need to buffer work between a producer and one or more workers, absorb traffic spikes, retry failed processing, or guarantee a message is handled by a single consumer — for example, placing an order-processing task on a queue so an ECS or Lambda consumer handles it independently of the frontend.

• Choose SNS when the same event must reach multiple interested systems simultaneously — for example, notifying a monitoring system, sending a user an SMS, and triggering a Lambda function from one published message.

• Combine them for fan-out with durability: publish once to an SNS topic and let it deliver to multiple SQS queues, giving each downstream consumer its own queue with independent retry logic and dead-letter handling.

Related Service Distinction

Amazon EventBridge is often compared with SNS. EventBridge is a serverless event bus suited to routing structured events between AWS services, custom applications, and third-party SaaS partners, with content filtering, schema management, and targeted rule-based routing. SNS is optimized for high-throughput notifications and direct delivery to endpoints such as mobile push, SMS, and email. Keep SQS for queueing, SNS for broadcast notifications, and EventBridge for event-driven routing across producers and consumers.

Best Practices and Common Pitfalls

• Design for idempotency: both SQS and SNS Standard offer at-least-once delivery, so consumers must tolerate duplicate messages safely.

• Use dead-letter queues: SQS supports dead-letter queues through a redrive policy; SNS supports them per subscription for failed deliveries. Message-size limits, throughput quotas, and retention bounds vary by Region and configuration, so check the service documentation for current values.

• Apply least privilege: grant only the needed actions on specific resource ARNs — for example, sqs:SendMessage on a queue ARN or sns:Publish on a topic ARN — and prefer short-lived credentials from IAM roles over long-lived access keys.

• Common pitfall: treating SNS as a durable store. SNS delivers messages to subscribers immediately and does not retain them for later replay in the way SQS does, so pair it with SQS whenever a durable backlog is required.

Core Architecture

Lambda runs code in ephemeral compute environments triggered by events from AWS services or custom applications. You upload your code as a function, and the service handles capacity provisioning, patching, and scaling. Functions are stateless, meaning any persistent data must be stored externally in services like Amazon DynamoDB or Amazon S3. Billing is calculated based on the number of requests and the duration your code runs, rather than pre-purchased capacity.

Key Features

• Automatic Scaling: The service scales your application by running code in response to each trigger, managing concurrency automatically.

• Event-Driven Integration: Native integration with over 200 AWS services and SaaS providers, such as Amazon S3, Amazon API Gateway, and Amazon EventBridge.

• Flexible Runtimes: Supports multiple programming languages including Python, Node.js, Java, and .NET, along with custom runtimes via the Lambda Runtime API.

• Lambda Layers: Allows you to manage shared code and dependencies separately from function code, simplifying updates across multiple functions.

• High Availability: Functions are distributed across multiple Availability Zones within a region to ensure fault tolerance.

Security and IAM Best Practices

Security relies on IAM execution roles rather than long-lived access keys. Each function assumes a role granting only the permissions necessary for its specific task, adhering to the principle of least privilege. Credentials are temporary and rotated automatically by the service. Resource-based policies control who can invoke the function, enabling secure cross-account access. You should also encrypt environment variables using AWS KMS and configure VPC access only when required to reduce cold start latency.

Lambda vs Traditional Compute

Ideal Use Cases

Lambda is ideal for processing data streams, automating file transformations in Amazon S3, and powering backend APIs via Amazon API Gateway. It suits workloads with unpredictable traffic patterns where paying for idle capacity is inefficient. Common scenarios include real-time image resizing, log analysis, and scheduled maintenance tasks. For long-running processes or applications requiring specific operating system configurations, container services like Amazon ECS or Amazon EKS may be more appropriate.

Core Capabilities

The service offers three API types:

• REST APIs: the original, feature-rich API type, with request validation, mapping templates for payload transformation, usage plans, API keys, and stage-level caching.

• HTTP APIs: the recommended choice for new serverless and HTTP workloads, offering lower latency and lower cost, at the price of fewer management features (no mapping templates, no usage plans, no built-in caching).

• WebSocket APIs: for stateful, bidirectional, real-time communication such as chat, notifications, and streaming telemetry.

All three support stage-based deployments and versioning, so you can promote an API from dev to staging to prod and roll back by pointing a stage at an earlier deployment.

Key Benefits

• Scalability: it absorbs traffic spikes automatically, with no servers to provision or capacity to plan; throughput limits vary by region and change over time, so check the service documentation for current quotas.

• Monitoring: integration with Amazon CloudWatch provides logs, metrics, and dashboards, and AWS X-Ray adds distributed tracing for requests flowing through API Gateway to backends.

• Caching: the built-in API Gateway cache (REST APIs) stores responses at the stage level, cutting latency and reducing backend load for repeated identical requests.

• Decoupling and protocol shielding: clients talk to stable, versioned endpoints while backend implementations, hosts, and even backend technologies change behind the scenes.

• Cost efficiency: pay-per-request and optional caching tiers mean costs track usage, though pricing and quotas vary by region and over time.

Integration Patterns

API Gateway fronts several kinds of backends:

• AWS Lambda: the most common pattern for serverless, event-driven APIs; the proxy integration passes the full request and returns the response.

• HTTP endpoints: proxying to services on Amazon EC2, Amazon ECS, Amazon EKS, or on-premises servers, including private backends reached through a VPC link.

• AWS service proxies: invoking services such as Amazon DynamoDB, Amazon SNS, Amazon SQS, or AWS Step Functions directly, with no intermediate compute.

• WebSocket backends: routing connect, disconnect, and message events to Lambda or HTTP endpoints for real-time workflows.

Security and Access Control

• Authentication and authorization: Amazon Cognito user pools, IAM authentication, Lambda authorizers (custom logic), and JWT authorizers on HTTP APIs.

• Least privilege: grant IAM principals only execute-api:Invoke on specific API ARNs and stages, scoped to the resource path and method, instead of wildcard resources and actions.

• Prefer roles over keys: use short-lived credentials from IAM roles for humans and workloads; never use root account credentials for routine or programmatic access.

• Traffic protection: AWS WAF guards against common web exploits, throttling limits protect backends from traffic spikes, and usage plans (REST APIs) apply per-key rate limits and quotas.

When to Use It vs Alternatives

A practical rule: reach for API Gateway whenever the API is a product you publish to outside clients; use an ALB or VPC Lattice for internal service-to-service traffic, and Lambda Function URLs only for the simplest single-function endpoints.

What It Is

DynamoDB Streams is an optional capability that you enable on an existing DynamoDB table. Once enabled, it records changes to that table’s items and stores them in a stream that is associated with the table for a limited retention window. It is distinct from, but often compared with, Amazon Kinesis Data Streams: DynamoDB Streams is tightly coupled to a single table, is managed entirely by AWS, and is billed and scaled with the table, whereas Kinesis Data Streams is a general-purpose, independently provisioned streaming service.

How It Works

1. You enable a stream on a table and choose a stream view type that determines what data each record contains.

2. When an application creates, updates or deletes an item, DynamoDB writes a stream record to the stream almost immediately.

3. Each stream is divided into shards, and records within a shard are ordered by the time of the change. Each record has a sequence number that reflects that ordering.

4. A consumer application reads records from the shards, processes them, and advances its position using the sequence numbers. Consumers are typically implemented with AWS Lambda (through the DynamoDB Streams event source mapping) or with the DynamoDB Streams API via the Kinesis Client Library (KCL) adapter.

5. Records remain available for retrieval for 24 hours; after that they are automatically removed. There is no way to extend this window.

6. If you disable the stream, the existing records expire per the retention window and the stream is eventually removed.

Stream View Types

The view type is set when the stream is enabled and cannot be changed without disabling and re-enabling the stream:

• KEYS_ONLY: only the key attributes of the modified item.

• NEW_IMAGE: the entire item as it appears after the modification.

• OLD_IMAGE: the entire item as it appeared before the modification.

• NEW_AND_OLD_IMAGES: both the before and after versions of the item.

NEW_AND_OLD_IMAGES is the most useful for audit and synchronisation use cases because it lets you compute the exact delta, while KEYS_ONLY minimises the data stored and processed when only the identity of the changed item matters.

Key Features

• Ordering guarantees: changes to a single item are always delivered in order within the same shard, because DynamoDB assigns records to shards based on the item’s partition key.

• At-least-once delivery with Lambda: the Lambda event source mapping checkpoints by sequence number so records are not reprocessed under normal operation, but retries after failures or shard rebalancing can still redeliver a record, so consumers must be idempotent.

• Near real time: records typically appear in the stream within fractions of a second of the table write.

• Automatic scaling: shard count grows and shrinks with the table’s write traffic; you do not provision shards yourself.

• Integration with AWS services: Lambda is the most common consumer, and streams can also be consumed through Amazon EventBridge Pipes, which can enrich and route the change events to other targets.

Common Use Cases

• Reacting to changes: send a confirmation email, update a derived cache such as Amazon ElastiCache, or invalidate a CDN entry when an item changes.

• Replication and aggregation: copy data between tables or Regions, maintain a materialised view, or feed an analytics pipeline into Amazon OpenSearch Service or a data lake.

• Audit and compliance: keep NEW_AND_OLD_IMAGES to record what changed and when, complementing but not replacing AWS CloudTrail, which logs API calls rather than item-level data.

Best Practices and Pitfalls

• Design consumers to be idempotent, because retries after function failures or shard rebalancing can deliver the same record more than once.

• Do not use the stream as a durable event bus: the 24-hour retention window means a long consumer outage can lose changes. For durable, fan-out event distribution across many subscribers, use Amazon EventBridge or Amazon SNS instead.

• Keep read capacity in mind: heavy stream processing consumes read units against the table’s provisioned or on-demand capacity, and throughput varies by Region and configuration; consult the DynamoDB documentation for current behaviour rather than assuming fixed numbers.

• Follow least privilege: grant consumers only dynamodb:DescribeStream, dynamodb:GetRecords, dynamodb:GetShardIterator and dynamodb:ListStreams on the specific stream ARN, and prefer a dedicated IAM role for the consumer rather than broad table access.

• Remember that streams are per table: there is no cross-table stream, and you cannot replay records older than the retention window, so archive them downstream if you need long-term history.

Orchestration Model

Step Functions workflows are defined with the Amazon States Language (ASL), a JSON-based structured language. A workflow is made up of states that perform work (Task), branch on conditions (Choice), run branches concurrently (Parallel), iterate over collections (Map), or pause for time (Wait). The service tracks each state’s input and output, so data flows reliably between services, and it persists progress so you do not have to build polling, retry queues, or state stores yourself.

Workflow Variants

Choose the workflow type by workload profile:

Security and IAM

Follow least privilege for the execution role: grant only the specific actions and resources the workflow needs (for example, lambda:InvokeFunction on the specific function ARN), prefer roles over long-lived access keys, and never use the root account. Step Functions assumes this role to invoke downstream services such as Lambda, ECS, SNS, or DynamoDB. Keep secrets in AWS Secrets Manager or Parameter Store and reference them at runtime rather than hardcoding them in the state machine definition.

When to Use It

• Good fit: multi-step processes with conditional logic, fan-out/fan-in (Parallel/Map), human approval steps, long-running sagas, and replacing hand-rolled retry and compensation code.

• Poor fit: single-step request/response, simple linear chains, or tight, low-latency loops where direct service-to-service calls are cheaper and simpler.

Common Pitfalls

• Overusing it: for trivial linear logic, direct Lambda-to-service invocation may be simpler and cheaper.

• Large payloads: Step Functions caps total input and output size (see the service documentation for the current limit), so pass large data by reference via S3 or DynamoDB instead of inline.

• Wrong type: choosing Express Workflows for long-running or auditable processes leads to timeouts and lost console history; choose Standard Workflows for durability and auditability.

• Unbounded retries: always set explicit Retry policies with bounded MaxAttempts and backoff so a failing downstream service cannot drive runaway costs.

Core Services

The Kinesis family consists of distinct services tailored to different streaming needs: - Amazon Kinesis Data Streams: Captures and durably stores streaming records for custom consumer applications that read and process them at their own pace, with per-shard ordering. - Amazon Data Firehose (formerly Kinesis Data Firehose): Loads streaming data into destinations such as Amazon S3, Amazon Redshift, or Amazon OpenSearch Service without managing infrastructure; the former name is deprecated. - Amazon Kinesis Video Streams: Captures, processes, and stores video and audio streams for analytics and machine learning. - Amazon Managed Service for Apache Flink (formerly Kinesis Data Analytics): Runs Apache Flink applications that transform and analyze streaming data; the older SQL-based Kinesis Data Analytics offering is deprecated and new SQL applications can no longer be created.

Common Use Cases

Kinesis fits scenarios where data arrives continuously and must be handled with low latency: - Real-time analytics: Monitoring website clicks, IoT telemetry, or financial transactions as they occur, feeding dashboards and alerting. - Log and event aggregation: Centralizing application and infrastructure logs from distributed systems for security auditing, compliance, or debugging. - Data lake ingestion: Continuously moving records into Amazon S3 for later batch processing, machine learning, or long-term retention. - Video analysis: Processing security camera or device feeds for object detection, motion tracking, and other edge intelligence.

Kinesis vs. Alternatives

Choosing the right service depends on the workload: | Service | Purpose | Retention | | :— | :— | :— | | Kinesis Data Streams | High-throughput streaming with replay and per-shard ordering | Hours to days (configurable) | | Amazon SQS | Decoupling microservices, task queues | Up to 14 days | | Amazon EventBridge | Event-driven architecture, SaaS integrations | None (push-based with retry) | | Amazon SNS | Fan-out notifications to subscribers | None |

Use Kinesis Data Streams when you need ordered, replayable records at massive scale with custom consumers. Use Amazon Data Firehose when the goal is simply to land streaming data in S3, Redshift, or OpenSearch without writing consumer code. Use SQS for task distribution between services, and EventBridge for routing events across accounts or SaaS partners.

Security and Best Practices

Follow least privilege when configuring access: assign an IAM role to your compute workload (for example an EC2 instance or Lambda function) instead of long-lived access keys, and grant only specific actions such as kinesis:PutRecord scoped to the specific stream ARN. Never embed credentials in source code; rely on the AWS SDK’s default credential provider chain. Choose partition keys carefully, since they determine shard assignment and therefore your effective parallelism, and monitor shard-level metrics to avoid hot shards.

How It Works

You start by defining a GraphQL schema: types, fields, queries, mutations, and subscriptions. For each field that must return data, you attach a resolver that maps the GraphQL request to a backend call. AppSync supports two resolver styles:

• Velocity Template Language (VTL) resolvers, the original mapping-template approach.

• JavaScript/TypeScript (JS) resolvers, the newer, recommended option for teams who prefer a full programming language over templates.

Resolvers can be single (one request/response template or function) or pipeline resolvers, which chain multiple functions in one call — useful for validating input, calling one data source, then enriching the result from another. Supported data sources include Amazon DynamoDB, AWS Lambda, HTTP endpoints (which covers REST APIs, the Aurora Serverless Data API, and other HTTP services), Amazon OpenSearch Service, and Amazon EventBridge.

Core Features

• GraphQL APIs: Clients ask for exactly the fields they need, which avoids over-fetching and under-fetching and reduces the number of round trips between app and backend.

• Real-time subscriptions: Built-in WebSocket support pushes data to clients when data changes, without you managing socket servers.

• AWS AppSync Events: A newer capability for WebSocket-based publish/subscribe messaging without writing a GraphQL schema, suited to broadcast and fan-out use cases.

• Offline data sync: The Amplify DataStore library syncs local app data with AppSync when connectivity returns, with conflict detection and resolution strategies such as optimistic concurrency and AutoMerge.

• Fine-grained authorization: Security can be applied per field, not just per API.

• Server-side caching: A managed cache tier can store resolver results globally or per-resolver, reducing load on your data sources.

• Observability and protection: Integration with Amazon CloudWatch metrics and logs and AWS X-Ray tracing, plus AWS WAF web ACLs for request filtering.

Key Benefits

• No infrastructure to manage: AppSync scales automatically with request volume; there are no servers to patch or scale.

• One API over many backends: A mobile app can read user profile data from DynamoDB, search products in OpenSearch, and invoke a Lambda microservice through one GraphQL query.

• Strong contracts: The schema is a typed contract between frontend and backend, which improves collaboration and enables tooling such as generated TypeScript or Swift clients through Amplify Codegen.

• Real-time and offline by default: These are first-class features rather than custom-built additions.

• AWS-native security and IAM integration: Temporary credentials and roles follow least privilege instead of long-lived keys embedded in apps.

Authorization Options

When to Choose AppSync

Choose AppSync when clients need flexible, client-specific queries, real-time updates, or offline sync — typical for mobile apps, dashboards, and collaboration tools. Choose Amazon API Gateway plus Lambda for straightforward REST APIs or when you need protocol features GraphQL does not provide. Choose Amazon SNS, Amazon SQS, or EventBridge for asynchronous messaging and event-driven decoupling, and AppSync Events when you mainly need WebSocket pub/sub without GraphQL semantics.

Best Practices and Caveats

• Follow least privilege: grant resolvers and data sources only the specific actions and resources they need, and prefer IAM roles over long-lived access keys.

• Use Cognito user pools or OIDC for end-user identity; treat API keys as a development convenience, not a production security control.

• Keep resolvers small and push business logic into Lambda when it becomes complex.

• Enable caching and CloudWatch/X-Ray observability from the start, and review service quotas and pricing in the AWS documentation, since they vary by Region and change over time.

Core Architecture

• Event Bus: The container for events. Custom buses allow isolation between applications, while the default bus receives events from AWS services.

• Rules: Define patterns to match incoming events based on their JSON structure, including content-based filtering on fields in the detail section.

• Targets: Destinations where matched events are sent, including Lambda, SQS, SNS, Step Functions, ECS tasks, API destinations (HTTP endpoints) and more.

• Event Sources: AWS services (e.g., S3, EC2, CodePipeline), custom applications via PutEvents, or partner SaaS providers.

Role in Serverless Architectures

EventBridge enables decoupling by allowing producers to emit events without knowing the consumers. When a state change occurs (e.g., an object uploaded to S3), an event is published. EventBridge evaluates rules and invokes the appropriate serverless compute (Lambda) asynchronously. This scales automatically with traffic spikes and simplifies error handling via retry policies on targets. It facilitates patterns like fan-out, where one event triggers multiple downstream processes independently, and it supports choreography-style workflows where each service reacts to events it cares about.

Service Comparison

Use SNS for simple fan-out notifications and SQS for buffering workloads. Use EventBridge when you need content-based routing, scheduled events, or integration with external SaaS providers.

Security and IAM

Follow least privilege when configuring permissions. The service invoking PutEvents should have permission only for events:PutEvents on the specific event bus ARN. Targets like Lambda functions must grant EventBridge permission to invoke them via resource-based policies. Avoid using root account credentials; instead, assign IAM roles to EC2 instances or use Lambda execution roles. Encryption at rest is enabled by default with an AWS managed KMS key, and you can optionally use a customer managed key.

Best Practices

• Schema Registry: Use EventBridge Schema Registry to discover and validate event structures and generate code bindings.

• Dead Letter Queues: Configure DLQs for targets to capture failed invocations for analysis.

• Input Transformation: Use input transformers in rules to send only necessary data to targets, reducing payload size.

• Monitoring: Monitor the CloudWatch metrics that EventBridge publishes automatically for rule matches, invocations and throttling to detect failures.

• Naming: Use consistent naming conventions for event buses and rules to simplify management across environments.

How It Works

A SAM template is a valid CloudFormation template that begins with Transform: AWS::Serverless-2016-10-31. When you deploy, the SAM transform rewrites your shorthand resources into native CloudFormation resources before the stack is created or updated:

• AWS::Serverless::Function expands into an AWS::Lambda::Function, an auto-generated AWS::IAM::Role (unless you supply one), plus an AWS::Lambda::EventSourceMapping or AWS::Lambda::Permission for every event you declare.

• AWS::Serverless::Api and AWS::Serverless::HttpApi expand into AWS::ApiGateway::RestApi or AWS::ApiGatewayV2::Api with their stages, methods, and integration resources.

• AWS::Serverless::StateMachine expands into an AWS Step Functions state machine with its execution role; AWS::Serverless::LayerVersion into a Lambda layer; AWS::Serverless::Application into a nested serverless application from the AWS Serverless Application Repository.

• AWS::Serverless::Connector attaches two resources and generates only the least-privilege IAM permissions needed for the source to call the target — useful for keeping permissions tight without hand-writing policy JSON.

• The Globals section defines properties inherited by all resources of a type, so Handler, Runtime, MemorySize, Timeout, Environment, and Policies can be declared once instead of on every function.

Key Features

• Event-driven configuration: inline event sources such as Api, HttpApi, Schedule, S3, SNS, SQS, DynamoDB, KinesisDataStreams, EventBridgeRule, and CloudWatchLogs wire up triggers and permissions together.

• Policy templates: pre-built scoped policies like DynamoDBCrudPolicy, S3ReadPolicy, or SQSSendMessagePolicy grant narrow permissions without writing full policy documents, which supports least-privilege IAM.

• First-class local development: the SAM CLI mirrors AWS services on your machine for fast iteration.

• Open source and IDE-supported: the template schema is published, so editors offer validation and autocompletion.

SAM CLI Workflow

1. sam init scaffolds a new project from a managed template, choosing runtime and package type (zip or container image).

2. sam build prepares deployment artifacts, including building dependencies for the target runtime; --use-container builds inside a Lambda-like environment for reproducibility.

3. sam local invoke, sam local start-api, and sam local start-lambda run functions and APIs locally, emulating event payloads and the Lambda Runtime Interface.

4. sam validate checks the template, and sam validate --lint runs deeper linting through the CloudFormation linter.

5. sam deploy --guided packages artifacts (Amazon S3 for zip packages, Amazon ECR for images), creates the CloudFormation changeset, and saves settings to samconfig.toml; it also prompts for the required stack capabilities (CAPABILITY_IAM, CAPABILITY_NAMED_IAM, CAPABILITY_AUTO_EXPAND).

6. sam sync (SAM Accelerate) pushes local changes to a development stack incrementally for faster feedback, and sam logs tails CloudWatch Logs for the deployed functions.

The SAM CLI can also deploy applications authored with the AWS CDK (using sam deploy and sam build), so teams can use SAM’s deployment workflow while defining infrastructure in TypeScript, Python, or other CDK languages; local testing commands such as sam local invoke still require a SAM template.

Choosing Between SAM, CDK, and Plain CloudFormation

Advantages, Limitations, and Best Practices

• Advantages: dramatically less YAML than equivalent CloudFormation, sensible defaults, strong local testing story, and a single sam deploy pipeline from source to stack.

• Limitations: the abstraction is serverless-centric, so heavily non-serverless architectures fit CDK or CloudFormation better; debugging the expanded template requires reading the transformed output in the CloudFormation console when behavior diverges from expectations.

• Best practices: prefer Globals for shared settings, grant permissions through scoped policy templates or AWS::Serverless::Connector rather than AdministratorAccess-style broad policies, keep functions small and single-purpose, and run sam sync only against dedicated development stacks since it bypasses normal change-set review.

How It Works

• You choose an EMR release, which bundles specific, tested versions of the open-source applications (for example, a particular Spark and Hive version), so you do not assemble the stack yourself.

• You launch a cluster (or a serverless application) that runs your processing code, typically reading input from and writing results to Amazon S3 using EMRFS, the Hadoop-compatible file system for S3.

• AWS installs and configures the selected software, manages the cluster’s health, and provides CloudWatch metrics and logging so you can observe jobs.

• Because the persistent data lives in S3, clusters can be treated as ephemeral compute: spin one up for a job, then terminate it.

Key Features

• Managed big data frameworks: first-class support for Spark, Hive, HBase, Presto/Trino, Flink and others, with AWS-optimized builds and default tuning.

• Flexible deployment models: traditional EC2-based clusters, EMR Serverless for Spark and Hive workloads with no cluster management, and EMR on EKS to run Spark jobs inside existing Amazon EKS environments.

• Elastic capacity: add or remove task nodes manually, use Managed Scaling to resize automatically based on workload, and combine On-Demand and Spot Instances in instance fleets to control cost.

• S3-native storage: EMRFS lets compute read and write directly to S3, decoupling storage from compute and enabling cheap, durable datasets that outlive any cluster.

• Security and governance: runs inside an Amazon VPC, uses IAM roles for the service and the EC2 instance profile, supports encryption in transit and at rest with AWS KMS, integrates with AWS Lake Formation for fine-grained access control in supported configurations, and can publish detailed logs to S3 or CloudWatch.

• Notebook-based development: EMR Studio provides a web-based workspace for Jupyter notebooks, SQL editors and orchestration with Apache Airflow or Amazon Managed Workflows for Apache Airflow, integrated with EMR clusters and EMR Serverless.

Choosing a Deployment Model

When EMR Fits and When It Does Not

• Use EMR for large-scale ETL, batch analytics, log processing, interactive SQL over data lakes, and Spark-based machine learning pipelines.

• For event streaming between applications, choose Amazon SNS, Amazon SQS or Amazon EventBridge; EMR is for analytics workloads, not message brokering.

• For simple scheduled transformations on data already in a warehouse, services such as AWS Glue or Amazon Redshift may be simpler; EMR shines when you need the full open-source big data stack and control over its execution.

Best Practices and Common Pitfalls

• Keep data in S3 and compute ephemeral: terminate idle clusters instead of leaving them running; this is the single biggest cost lever for EC2-based EMR.

• Prefer EMR Serverless when your workload is Spark or Hive and you do not need a persistent cluster; it removes instance-type selection and scaling tuning.

• Use Spot capacity with a checkpointing strategy: Spot Instances can be reclaimed, so design Spark jobs to recover rather than assuming nodes persist.

• Follow least privilege: grant the EMR service role and the EC2 instance profile only the actions and resources they need, and prefer temporary credentials and roles over long-lived access keys.

• Secure by default: run in a VPC with private subnets where possible, enable encryption with AWS KMS, and never embed credentials in job code or bootstrap scripts.

• Pin and test the EMR release: each release bundles specific application versions, so upgrade deliberately and validate job compatibility rather than always taking the latest release.

How Each Works

• Amazon Athena: You point Athena at a data catalog (the default is the AWS Glue Data Catalog) that defines the schema of files in an S3 bucket, then run SELECT queries through the console, CLI, SDK, or JDBC/ODBC drivers. Athena spins up the compute itself, runs the query, and tears it down. Supported formats include CSV, JSON, ORC, Parquet, and Avro. There is no cluster to size, patch, or scale.

• Amazon Redshift: You create a cluster (or a serverless workgroup with a base data-warehouse capacity measured in RPU) and load data into it from S3, DynamoDB, Kinesis, or relational databases. Redshift stores the data locally in a columnar, compressed format and uses features such as sort keys, distribution styles, and result caching to accelerate queries. It is based on PostgreSQL, so it is familiar to SQL developers, but it is not a drop-in PostgreSQL replacement.

Key Differences

When to Choose Which

• Choose Athena when your data already lives in S3, queries are infrequent or exploratory, and you want to avoid managing capacity. Classic use cases include querying CloudTrail logs, VPC Flow Logs, application logs, ad-hoc data discovery, and lightweight reporting. Cost stays low when data is partitioned and stored in columnar formats such as Parquet or ORC.

• Choose Redshift when you need a central analytics store with predictable, high performance: dashboards refreshed frequently, large multi-table joins, machine-learning scoring in-database (Redshift ML), or materialised views that many users hit concurrently. Redshift is also the better choice when you need to transform and persist data rather than just scan it.

• They are complementary, not mutually exclusive. A common pattern is to keep raw data in S3, explore it with Athena, and load the curated, business-critical subsets into Redshift for production reporting. Redshift Spectrum extends this by letting a Redshift cluster join its local tables with S3 data, so you can keep cold data in S3 and hot data in the warehouse.

Best Practices and Common Pitfalls

• Partition your S3 data for Athena by columns used in WHERE clauses (for example, dt='2025-06-01'); scanning a single partition instead of the whole bucket is the single biggest cost and performance lever.

• Use columnar formats (Parquet or ORC) with compression for Athena workloads — querying row-based CSV or JSON can multiply the amount of data scanned.

• Run MSCK REPAIR TABLE (or update partitions in the AWS Glue Data Catalog) after adding new S3 partitions, or Athena will not see the new data.

• Design for Redshift performance: choose distribution keys to align joined tables, sort keys to match filter predicates, and use COPY with a manifest for bulk loads instead of many small INSERT statements.

• Do not treat Redshift as an OLTP database; it is built for analytics, not high-frequency single-row writes or transactional workloads.

• Watch the Athena bill: queries that scan large, unpartitioned datasets can become expensive quickly. Use workgroups with per-query data-scan limits to guard against runaway queries.

• Secure both services with least privilege: grant access to specific catalog databases, tables, and S3 prefixes rather than whole buckets, and prefer S3 bucket policies plus IAM roles over long-lived credentials.

Core Architecture

A stream is composed of shards, which are the base throughput units. Each shard provides a fixed write and read capacity, and data records are ordered within a partition key. - Retention Period: Records are retained for 24 hours by default and can be extended up to 365 days, so consumers can replay recent history after an outage or a code change. - Sequence Numbers: Each record receives a unique sequence number within its shard, preserving ordering per partition key. - Scaling: Throughput scales by adding or splitting shards. Resharding is an explicit operation, so capacity planning matters; on-demand mode lets the stream adapt automatically to traffic spikes, while provisioned mode gives predictable, shard-based capacity.

Common Use Cases

• Real-Time Analytics: Feed events into dashboards, alerting, or anomaly-detection systems immediately as they occur, for example fraud detection on payment events.

• Log and Event Aggregation: Centralize logs and events from distributed microservices for downstream processing and search.

• Data Lake Ingestion: Act as a buffer that absorbs bursty traffic before records are delivered to Amazon S3 or other destinations through Amazon Data Firehose (formerly Kinesis Data Firehose).

• Change Data Capture: Stream database changes to keep secondary stores, caches, or search indexes synchronized with low latency.

Comparison with Similar Services

Distinguishing Kinesis Data Streams from related services is a common interview topic and key for architectural decisions.

Choose Kinesis Data Streams when you need custom consumers, replay, and ordered per-key processing; choose Amazon Data Firehose when the goal is simply to deliver records to a destination with no custom consumer code.

Best Practices

• IAM Least Privilege: Grant only the actions a role needs, such as kinesis:PutRecord or kinesis:GetRecords, on specific stream ARNs rather than with wildcards; prefer roles over long-lived access keys.

• Consumer Types: Use enhanced fan-out for low-latency consumers that need dedicated throughput per shard; the classic shared-throughput model suits batch-oriented consumers.

• Error Handling: Implement retries with exponential backoff for ProvisionedThroughputExceededException, which indicates a shard limit has been hit.

• Monitoring: Track IteratorAge in Amazon CloudWatch to detect consumers falling behind producers, and monitor WriteProvisionedThroughputExceeded for throttled producers.

What AWS Glue Does

AWS Glue sits between your data stores and your analytics consumers. It connects to sources such as Amazon S3, Amazon RDS, Amazon DynamoDB, Amazon Redshift, and JDBC-accessible databases, reads and classifies the data, transforms it, and writes the results to a target you choose. Because it is serverless, AWS provisions the compute needed for each job and releases it when the job finishes, so there is no cluster to size, patch, or idle.

Core Components

• AWS Glue Data Catalog: a central metadata repository that stores database and table definitions, schemas, and partition locations. It is Hive Metastore-compatible and is shared by services such as Amazon Athena, Amazon Redshift Spectrum, Amazon EMR, and AWS Lake Formation, so all of them see the same view of your data.

• Crawlers: automated jobs that scan source data, classify it using built-in or custom classifiers, infer the schema, and write the metadata into the Data Catalog.

• Classifiers: rules that recognize common formats such as JSON, CSV, Parquet, ORC, Avro, XML, and JDBC tables.

• AWS Glue jobs: Spark-based ETL scripts that you write in Python (PySpark) or Scala, generate visually, or run as Ray-based jobs. Jobs use built-in transforms such as ApplyMapping, Join, Filter, and SelectFields.

• Job bookmarks: state that records which data has already been processed, enabling incremental loads on scheduled runs.

• Triggers and workflows: schedule jobs or chain them on events, for example running a transform when a crawler finishes.

• AWS Glue Studio: a visual interface for building, running, and monitoring ETL flows with little or no code.

• AWS Glue DataBrew: a visual data preparation and cleaning tool for analysts who do not write Spark code.

• AWS Glue Data Quality: rules-based and automated data quality checks that can stop bad data from moving downstream.

• Streaming ETL: jobs that consume continuously from Amazon Kinesis Data Streams or Amazon MSK (Apache Kafka) for near-real-time transformation.

• AWS Glue Schema Registry: a central registry for schemas used by streaming producers and consumers.

Key Benefits

• No infrastructure to manage: capacity, patching, and scaling are handled by the service; you pay for the processing resources used while jobs run, billed per data processing unit-hour, with rates that vary by AWS Region and change over time (see the AWS Glue pricing documentation).

• Faster time to insight: crawlers and the Data Catalog automate schema discovery, so data becomes queryable in Athena or Redshift Spectrum without manual data definition.

• Centralized governance: one catalog plus integration with AWS Lake Formation gives fine-grained access control, and AWS KMS encryption can be applied at rest and in transit.

• Heterogeneous integration: a single job can read from S3, join it with a relational table, and write Parquet to a data lake.

• Incremental processing: job bookmarks avoid reprocessing unchanged data on repeated runs.

• Choice of interfaces: visual tools for analysts and PySpark/Scala for engineers, plus streaming ETL for near-real-time pipelines.

How It Compares to Related AWS Services

Best Practices and Common Pitfalls

• Grant each job an IAM role with least privilege: only the actions and resources it needs, and prefer roles over long-lived access keys.

• Scope crawlers to specific paths and run them on a schedule that matches how often the data changes; crawling too broadly slows catalog updates.

• Partition S3 data by keys such as date and region so downstream queries and crawlers only read what they need.

• Enable job bookmarks for incremental pipelines and reset them deliberately when you change the transformation logic.

• Use AWS Glue Data Quality rules early in the pipeline so invalid data is caught before it reaches analytics consumers.

• Choose the worker type and Glue engine version that fits the workload, since both affect runtime and cost; monitor jobs in the console and tune rather than defaulting to the largest configuration.

What It Is