🔗 Application Integration#

Learning Objectives#

• Decouple microservices using SQS, SNS, and EventBridge
• Choose between pull (SQS) and push (SNS) messaging patterns
• Design event-driven architectures
• Implement streaming data with Kinesis

1. Amazon SQS (Simple Queue Service)#

1.1 SQS Overview#

Fully managed message queuing service for decoupling application components.

Producer → SQS Queue → Consumer
             │
        ┌────┴────┐
        │ Standard │ or │ FIFO
        │ Queue    │    │ Queue
        │          │    │
        │ At-least-│    │ Exactly-once
        │ once     │    │ processing
        │ delivery │    │ (ordered)
        │ Best-effort  │
        │ ordering  │    │
        └──────────┘    └────────┘

1.2 Standard vs FIFO Queues#

⚡ Exam Tip: FIFO = exactly-once + ordering. Standard = high throughput + at-least-once. FIFO names must end with .fifo.

1.3 SQS Features#

Visibility Timeout:

Message received by Consumer A
    │
    ├── Consumer A processes (30 sec timeout)
    │       │
    │       ├── Success → Message deleted
    │       └── Fail → Message reappears in queue
    │
    └── Timeout expires (30 sec) → Message reappears
        → Consumer B picks it up

Dead Letter Queue (DLQ):

# Create DLQ
aws sqs create-queue --queue-name my-app-dlq

# Get DLQ ARN
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-app-dlq --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)

# Create main queue with DLQ + redrive policy
aws sqs create-queue \
  --queue-name my-app-queue \
  --attributes '{ "RedrivePolicy": "{\"deadLetterTargetArn\":\"'"$DLQ_ARN"'\",\"maxReceiveCount\":\"3\"}",
    "VisibilityTimeout": "60",
    "MessageRetentionPeriod": "345600"
  }'

Delay Queues & Long Polling:

# Create queue with 10 sec delay and 20 sec long polling
aws sqs create-queue \
  --queue-name delayed-queue \
  --attributes '{"DelaySeconds": "10", "ReceiveMessageWaitTimeSeconds": "20" }'

• Delay queues: Messages hidden for 0-15 minutes after sending
• Long polling: Reduces empty responses by waiting up to 20 seconds for messages
• Short polling: Returns immediately (may return empty)

1.4 SQS Extended Client Library#

Send messages up to 2 GB (SQS max is 256 KB) by storing payload in S3:

Producer → S3 (large payload) + SQS (pointer to S3) → Consumer reads from S3

2. Amazon SNS (Simple Notification Service)#

2.1 SNS Overview#

Fully managed pub/sub messaging service. Messages are pushed to subscribers.

Topic: order-notifications
    │
    ├── SQS Queue (order-processor)
    ├── Lambda (email-notification)
    ├── Lambda (sms-notification)
    ├── HTTP/HTTPS (webhook)
    └── Email (admin@example.com)

2.2 SNS + SQS Fan-Out Pattern#

Send one message → Process in multiple independent services:

# Create SNS topic
aws sns create-topic --name order-events

# Create SQS queues for different processors
aws sqs create-queue --queue-name order-inventory-queue
aws sqs create-queue --queue-name order-shipping-queue
aws sqs create-queue --queue-name order-analytics-queue

# Subscribe queues to the topic
aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:order-events \
  --protocol sqs \
  --notification-endpoint arn:aws:sqs:us-east-1:123456789012:order-inventory-queue

aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:order-events \
  --protocol sqs \
  --notification-endpoint arn:aws:sqs:us-east-1:123456789012:order-shipping-queue

# Publish message
aws sns publish \
  --topic-arn arn:aws:sns:us-east-1:123456789012:order-events \
  --message '{"orderId": "12345", "status": "confirmed"}'

2.3 Message Filtering#

SNS supports filter policies so subscribers only get relevant messages:

// Order processor: only order.created events
{"event_type": ["order.created"] }

// Complaint handler: only order.complaint events
{"event_type": ["order.complaint"] }

3. Amazon EventBridge#

3.1 EventBridge Overview#

Serverless event bus for connecting applications using events from AWS services, SaaS partners, and custom apps.

graph LR
    subgraph Sources["Event Sources"]
        AWS["AWS Services\nEC2, S3, Lambda,\nCloudWatch"]
        SaaS["SaaS Partners\nShopify, Zendesk,\nDatadog, PagerDuty"]
        Custom["Custom Apps\nYour application\nevents"]
    end

    subgraph Bus["EventBridge Event Bus"]
        Default["Default Bus\n(AWS events)"]
        CustomBus["Custom Bus\n(Custom events)"]
        PartnerBus["Partner Bus\n(SaaS events)"]
    end

    subgraph Rules["Rules Engine"]
        R1["Rule: EC2 State\nFilter: running/stopped"]
        R2["Rule: S3 Object\nFilter: prefix uploads/"]
        R3["Rule: Custom\nFilter: order.completed"]
    end

    subgraph Targets["Event Targets"]
        Lambda["Lambda Function\nProcess event"]
        SF["Step Functions\nOrchestrate workflow"]
        SQS["SQS Queue\nBuffer for later"]
        API_DEST["API Destination\nSlack webhook"]
    end

    AWS --> Default
    Custom --> CustomBus
    SaaS --> PartnerBus
    
    Default --> R1
    Default --> R2
    CustomBus --> R3
    
    R1 --> Lambda
    R2 --> SF
    R3 --> SQS
    R3 --> API_DEST

    style AWS fill:#ff9900,color:#fff
    style Lambda fill:#ff9900,color:#fff
    style SQS fill:#01ab5c,color:#fff
    style SF fill:#527fff,color:#fff

3.2 EventBridge Rules#

Example: Notify on EC2 state changes:

{"source": ["aws.ec2"], "detail-type": ["EC2 Instance State-change Notification"], "detail": { "state": ["stopped", "terminated", "running"] }
}

Example: Monitor S3 object creation:

{"source": ["aws.s3"], "detail-type": ["Object Created"], "detail": { "bucket": { "name": ["my-important-bucket"] },
    "object": { "key": [{"prefix": "uploads/"}]
    }
  }
}

3.3 Schema Registry#

Store event structure and generate code bindings:

• Code bindings for Java, Python, TypeScript
• Discover schemas from events on the bus

3.4 API Destinations#

Send events to any HTTP API (third-party or custom):

aws events create-api-destination \
  --name "slack-webhook" \
  --connection-arn "arn:aws:events:...connection/slack" \
  --http-method POST \
  --invocation-rate-limit-per-second 10 \
  --invocation-endpoint "https://hooks.slack.com/services/..."

⚡ Exam Tip: EventBridge replaces CloudWatch Events. Default event bus = events from AWS services in your account. Custom event bus = events from other accounts or custom apps.

4. Amazon Kinesis#

4.1 Kinesis Data Streams#

Real-time streaming data ingestion at scale:

Producers (1000s/sec) → Kinesis Stream → Consumers (EC2/Lambda/KDA)
     │                      │                  │
     │               ┌──────┴──────┐           │
     │               │ Shard 1     │           │
     ├───────────────│ Shard 2     │───────────┤
     │               │ Shard 3     │           │
     │               └─────────────┘           │
     │                                          │
     └── Data stored for 1-365 days (default 24h)

4.2 Kinesis Data Firehose#

Serverless streaming ingest to destinations:

• S3, Redshift, Elasticsearch, Splunk
• Transform data with Lambda before delivery
• Convert format (CSV → Parquet/ORC)

4.3 Kinesis Data Analytics#

Run real-time SQL or Apache Flink on streaming data:

• Filter, aggregate, and transform streams
• Time-windowed analytics (e.g., “top products in last 5 minutes”)

5. Real-World Use Cases#

Use Case 1: Order Processing Pipeline#

API Gateway → SQS → Lambda (validate) → SNS (order-created)
                                            │
                              ┌─────────────┼─────────────┐
                              │             │             │
                         SQS (inventory) SQS (payment) SQS (email)
                              │             │             │
                         Lambda          Lambda        Lambda
                              │             │             │
                          Update stock    Process      Send email
                                          payment

Use Case 2: Event-Driven Data Lake#

S3 (new file) → EventBridge → Lambda (validate) → Step Functions
                                                      │
                                           ┌──────────┴──────────┐
                                           │                     │
                                    Glue ETL (transform)    SNS (notify)
                                           │
                                      S3 (curated data)
                                           │
                                     Athena (query)

✅ Chapter Quiz#

1. Which SQS queue type guarantees message ordering?

   • A) Standard
   • B) FIFO
   • C) DLQ
   • D) Delay

2. What is the maximum visibility timeout for SQS?

   • A) 30 seconds
   • B) 12 hours
   • C) 24 hours
   • D) 14 days

3. Which service provides fan-out messaging to multiple subscribers?

   • A) SQS
   • B) SNS
   • C) Kinesis
   • D) EventBridge

4. How long can data be retained in Kinesis Data Streams?

   • A) 24 hours
   • B) 7 days
   • C) 365 days
   • D) Unlimited

5. Which EventBridge component stores the structure of events for code generation?

   • A) Event Bus
   • B) Rule
   • C) Schema Registry
   • D) API Destination

6. An SQS queue accumulates millions of messages during peak hours. The consumer processes slower than the producer. How should this be addressed?

   • A) Replace SQS with SNS for higher throughput
   • B) Increase the number of consumer instances for parallel processing
   • C) Switch to a FIFO queue for higher throughput
   • D) Enable SQS extended client library

7. A financial application processes payment transactions requiring strict ordering and exactly-once processing. Which queue type should be used?

   • A) Standard queue
   • B) FIFO queue
   • C) Dead Letter Queue
   • D) Delay queue

8. An SNS topic publishes order events. Order confirmation emails must be sent immediately, while backend processing can be asynchronous. Which architecture should be used?

   • A) Subscribe a Lambda function for email and an SQS queue for backend to the same SNS topic
   • B) Create two separate SNS topics
   • C) Use SQS with two separate consumer groups
   • D) Use EventBridge with two targets

9. A company needs to capture EC2 instance state changes and trigger a Lambda function. How should this be configured in EventBridge?

   • A) Create a scheduled event rule
   • B) Create an EventBridge rule with an event pattern matching EC2 state-change notifications
   • C) Configure EC2 to publish directly to Lambda
   • D) Use CloudTrail to capture EC2 events

10. An SQS consumer occasionally fails before deleting a processed message, causing it to reappear and be processed again. What is the BEST way to handle this?

    • A) Switch to a FIFO queue with deduplication
    • B) Design the consumer to be idempotent and set an adequate visibility timeout
    • C) Use a Dead Letter Queue
    • D) Enable SQS long polling

11. A company needs to send messages larger than 256 KB through SQS. What approach should be used?

    • A) Enable the SQS Extended Client Library to store payloads in S3
    • B) Increase the SQS maximum message size quota
    • C) Use SNS instead of SQS
    • D) Compress messages to fit within 256 KB

12. A data pipeline ingests data from thousands of IoT devices. Multiple consumer applications must independently process the same data with the ability to replay it. Why is Kinesis Data Streams better than SQS?

    • A) Kinesis supports multiple independent consumers with data retention of up to 365 days
    • B) Kinesis has higher throughput than SQS
    • C) Kinesis supports larger message sizes
    • D) Kinesis is cheaper than SQS

13. An e-commerce platform uses SNS for order events. Payment and inventory services need all events, but shipping should only receive events with status “shipped”. How should this be configured?

    • A) Create separate SNS topics for each service
    • B) Use SNS subscription filter policies
    • C) Implement filtering in each subscriber application
    • D) Use EventBridge with separate rules

14. An SQS queue has a backlog of 100,000 messages. The consumer reads 10 messages per batch every 30 seconds. What is the approximate time to clear the backlog?

    • A) 5 minutes
    • B) 50 minutes
    • C) 83 hours
    • D) 4 hours

15. A multi-step workflow requires human approval steps, error handling with retries, and integration with multiple AWS services. Which service should orchestrate this?

    • A) Step Functions
    • B) EventBridge
    • C) SNS
    • D) Lambda

16. An EventBridge rule triggers a Lambda when an S3 object is created. The Lambda fails due to a transient error. What happens to the event?

    • A) The event is lost permanently
    • B) EventBridge retries for up to 24 hours and can send failed events to a DLQ
    • C) S3 resends the event
    • D) The event is stored in a separate S3 bucket

17. A company needs to stream EC2 application logs to OpenSearch Service with format transformation. Which service should be used?

    • A) Kinesis Data Streams with Lambda consumers
    • B) Kinesis Data Firehose with Lambda transformation
    • C) SQS with Lambda consumers
    • D) SNS with Lambda subscription

18. An SQS FIFO queue processes 200 messages/second but now needs 2,000/second. How can this be achieved?

    • A) Switch to a Standard queue
    • B) Use message batching to reach up to 3,000/second
    • C) Create additional FIFO queues
    • D) Enable SQS extended library

19. A microservice needs to receive events from a SaaS application (Shopify) without polling. How should this be configured in EventBridge?

    • A) Create a partner event bus and configure a rule targeting the microservice
    • B) Use API Gateway to receive Shopify webhooks
    • C) Poll the Shopify API with Lambda
    • D) Use SQS to receive Shopify events

20. Which SQS configuration reduces empty responses and lowers costs by waiting for messages to become available?

    • A) Set ReceiveMessageWaitTimeSeconds between 1 and 20
    • B) Set WaitTimeSeconds to 0
    • C) Use short polling with retries
    • D) Enable SQS extended client library

21. A Kinesis Data Stream uses user_id as partition key. One user generates 10x more data than others. What impact does this have?

    • A) All data for that user goes to the same shard, potentially causing a hot shard and throttling
    • B) Kinesis automatically redistributes data across shards
    • C) There is no impact since Kinesis scales automatically
    • D) The partition key is ignored for hashing

22. A company needs a durable pub/sub pattern where messages to an unavailable subscriber are not lost. Which architecture meets this requirement?

    • A) SNS subscribed to multiple SQS queues (fan-out)
    • B) SQS with multiple consumers
    • C) EventBridge with multiple targets
    • D) Kinesis with multiple consumers

23. Real-time SQL analytics on streaming data with sub-second latency is required. Which service should be used?

    • A) Kinesis Data Streams with Lambda
    • B) Kinesis Data Analytics
    • C) Kinesis Data Firehose
    • D) SQS with Lambda

24. Messages that fail processing after 3 attempts should be isolated for investigation. How should this be configured in SQS?

    • A) Set maxReceiveCount to 3 on the DLQ
    • B) Configure a RedrivePolicy with maxReceiveCount of 3 on the source queue pointing to a DLQ
    • C) Set visibility timeout to 3x the processing time
    • D) Use FIFO queue with content-based deduplication

25. An SQS queue has a visibility timeout of 30 seconds. A consumer takes 45 seconds to process a message. What happens?

    • A) The message is successfully deleted after processing
    • B) After 30 seconds, the message becomes visible again and may be processed by another consumer
    • C) The visibility timeout automatically extends
    • D) The message is moved to the DLQ

📚 Additional Resources#

• SQS Documentation
• SNS Documentation
• EventBridge Documentation
• Kinesis Documentation

Next → Serverless & Containers