Skip to content

Commit c384ab5

Browse files
julianwoodjulianwood
authored
Merge pull request #2950 from mscarimb/mscarimb-feature-eventbridge-pipes-sqs-to-dynamodb
New serverless pattern - eventbridge-pipes-sqs-to-dynamodb
2 parents 2b14d95 + e3f236b commit c384ab5

9 files changed

Lines changed: 503 additions & 0 deletions

File tree

‎eventbridge-pipes-sqs-to-dynamodb/README.md‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# Amazon SQS to Amazon DynamoDB using Amazon EventBridge Pipes with Amazon API Gateway and CDK/Python
2+
3+
This pattern will send messages from an SQS queue to a DynamoDB table via API Gateway using EventBridge Pipes.
4+
5+
![Architecture](./architecture.png)
6+
7+
Learn more about this pattern at Serverless Land Patterns: [https://serverlessland.com/patterns/eventbridge-pipes-sqs-to-dynamodb](https://serverlessland.com/patterns/eventbridge-pipes-sqs-to-dynamodb)
8+
9+
Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
10+
11+
## Requirements
12+
13+
- [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
14+
- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
15+
- [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
16+
- [AWS CDK](https://docs.aws.amazon.com/cdk/latest/guide/cli.html) installed and configured
17+
18+
## Deployment Instructions
19+
20+
1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
21+
```bash
22+
git clone https://github.com/aws-samples/serverless-patterns
23+
```
24+
2. Change directory to the pattern directory:
25+
```bash
26+
cd serverless-patterns/eventbridge-pipes-sqs-to-dynamodb
27+
```
28+
3. To manually create a virtualenv on MacOS and Linux:
29+
```bash
30+
$ python3 -m venv .venv
31+
```
32+
4. After the init process completes and the virtualenv is created, you can use the following
33+
step to activate your virtualenv.
34+
```bash
35+
$ source .venv/bin/activate
36+
```
37+
5. If you are a Windows platform, you would activate the virtualenv like this:
38+
```bash
39+
% .venv\Scripts\activate.bat
40+
```
41+
6. Once the virtualenv is activated, you can install the required dependencies.
42+
```bash
43+
$ pip install -r requirements.txt
44+
```
45+
7. To deploy the application:
46+
```bash
47+
$ cdk deploy
48+
```
49+
50+
## How it works
51+
52+
This template will create an SQS queue, EventBridge Pipe, API Gateway and a DynamoDB table.
53+
54+
Messages sent to the SQS queue are polled by EventBridge Pipe. EventBridge Pipe processes the messages and sends them to API Gateway endpoint. API Gateway transforms the message and writes the data to DynamoDB table using direct integration.
55+
56+
## Testing
57+
58+
Once this stack is deployed in your AWS account, copy the SQS queue name value from the output.
59+
60+
Alternatively, retrieve the queue URL using AWS CLI:
61+
```sh
62+
aws sqs list-queues --query 'QueueUrls[?contains(@, `EntryPointToEventbridgePipe`)]'
63+
```
64+
65+
Then, send a message to the SQS queue as follows:
66+
```sh
67+
aws sqs send-message \
68+
--queue-url "<REPLACE_WITH_OUTPUT_FROM_PREVIOUS_COMMAND>" \
69+
--message-body '{"Message": "{\"content\":\"Test message\",\"params\":{\"name\":\"Mario\",\"surname\":\"Rossi\"}}"}'
70+
```
71+
72+
Navigate to [AWS Console DynamoDB Tables](https://console.aws.amazon.com/dynamodbv2/home#tables) and check for a table with name `Audit-Table`.
73+
When you check the DynamoDB table, you can see the entry with all the attributes parsed by API Gateway.
74+
75+
Alternatively, you can scan the table using AWS CLI:
76+
```sh
77+
aws dynamodb scan --table-name Audit-Table
78+
```
79+
80+
## Cleanup
81+
82+
1. Delete the stack
83+
84+
```bash
85+
cdk destroy
86+
```
87+
88+
---
89+
90+
Copyright 2026 Amazon.com, Inc. or its affiliates. All Rights Reserved.
91+
92+
SPDX-License-Identifier: MIT-0

‎eventbridge-pipes-sqs-to-dynamodb/app.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
#!/usr/bin/env python3
2+
import os
3+
import aws_cdk as cdk
4+
from eventbridge_pipes_sqs_to_dynamodb.eventbridge_pipes_sqs_to_dynamodb import EventbridgePipesSqsToDynamodb
5+
6+
app = cdk.App()
7+
stack = EventbridgePipesSqsToDynamodb(app, "EventbridgePipesSqsToDynamodb")
8+
9+
app.synth()

‎eventbridge-pipes-sqs-to-dynamodb/architecture.png‎

28.7 KB
Loading

‎eventbridge-pipes-sqs-to-dynamodb/cdk.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
"app": "python3 app.py"
3+
}

‎eventbridge-pipes-sqs-to-dynamodb/eventbridge-pipes-sqs-to-dynamodb.json‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
{
2+
"title": "Amazon SQS to Amazon DynamoDB via Amazon EventBridge Pipes using CDK/Python",
3+
"description": "Send messages from an SQS queue to a DynamoDB table via API Gateway using EventBridge Pipes. Implemented with CDK/Python.",
4+
"language": "Python",
5+
"level": "300",
6+
"framework": "AWS CDK",
7+
"introBox": {
8+
"headline": "How it works",
9+
"text": [
10+
"Messages sent to the Amazon SQS queue are polled by Amazon EventBridge Pipe.",
11+
"Amazon EventBridge Pipe processes the messages and sends them to the Amazon API Gateway endpoint.",
12+
"Amazon API Gateway transforms the message and writes the data to the Amazon DynamoDB table using direct integration."
13+
]
14+
},
15+
"gitHub": {
16+
"template": {
17+
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/eventbridge-pipes-sqs-to-dynamodb",
18+
"templateURL": "serverless-patterns/eventbridge-pipes-sqs-to-dynamodb",
19+
"projectFolder": "eventbridge-pipes-sqs-to-dynamodb",
20+
"templateFile": "eventbridge_pipes_sqs_to_dynamodb/eventbridge_pipes_sqs_to_dynamodb.py"
21+
}
22+
},
23+
"services": {
24+
"from": {
25+
"serviceName": "Amazon SQS",
26+
"serviceURL": "/sqs/"
27+
},
28+
"to": {
29+
"serviceName": "Amazon DynamoDB",
30+
"serviceURL": "/dynamodb/"
31+
}
32+
},
33+
"patternArch": {
34+
"icon1": {
35+
"x": 10,
36+
"y": 50,
37+
"service": "sqs",
38+
"label": "Amazon SQS"
39+
},
40+
"icon2": {
41+
"x": 37,
42+
"y": 50,
43+
"service": "eventbridge",
44+
"label": "EventBridge Pipes"
45+
},
46+
"icon3": {
47+
"x": 63,
48+
"y": 50,
49+
"service": "apigw",
50+
"label": "API Gateway"
51+
},
52+
"icon4": {
53+
"x": 90,
54+
"y": 50,
55+
"service": "dynamodb",
56+
"label": "DynamoDB"
57+
},
58+
"line1": {
59+
"from": "icon1",
60+
"to": "icon2"
61+
},
62+
"line2": {
63+
"from": "icon2",
64+
"to": "icon3"
65+
},
66+
"line3": {
67+
"from": "icon3",
68+
"to": "icon4"
69+
}
70+
},
71+
"resources": {
72+
"bullets": [
73+
{
74+
"text": "Amazon SQS Developer Guide",
75+
"link": "https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html"
76+
},
77+
{
78+
"text": "Amazon EventBridge Pipes Documentation",
79+
"link": "https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-event-source.html"
80+
},
81+
{
82+
"text": "AWS CloudFormation API for Pipes",
83+
"link": "https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-pipes-pipe.html"
84+
},
85+
{
86+
"text": "Amazon EventBridge Pipes CDK v2 Python Reference",
87+
"link": "https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_pipes/CfnPipe.html"
88+
},
89+
{
90+
"text": "Amazon API Gateway DynamoDB Integration",
91+
"link": "https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-overview-developer-experience.html"
92+
}
93+
]
94+
},
95+
"deploy": {
96+
"text": ["cdk deploy"]
97+
},
98+
"testing": {
99+
"text": ["See the README in the GitHub repo for detailed testing instructions."]
100+
},
101+
"cleanup": {
102+
"text": ["Delete the stack: <code>cdk destroy</code>."]
103+
},
104+
"authors": [
105+
{
106+
"name": "Michele Scarimbolo",
107+
"image": "https://avatars.githubusercontent.com/u/229997073",
108+
"bio": "Technical Account Manager at AWS. Serverless specialist and enthusiast.",
109+
"linkedin": "michele-scarimbolo",
110+
"twitter": ""
111+
}
112+
],
113+
"patternType": "Serverless"
114+
}

‎eventbridge-pipes-sqs-to-dynamodb/eventbridge_pipes_sqs_to_dynamodb/__init__.py‎

Whitespace-only changes.

0 commit comments

Comments
 (0)