SQS Queues
Point an AWS SQS app at KubeMQ — send, receive, batch, visibility timeout, FIFO, and DLQ redrive, mapped onto the KubeMQ Queues pattern on sqs.{name} channels.
An SQS queue is point-to-point messaging: a producer sends messages, one or more consumers receive them, and each received message is hidden (its visibility window) until the consumer deletes it or the timeout expires and it is redelivered. The AWS connector maps this directly onto the KubeMQ Queues primitive — SQS queue orders becomes KubeMQ channel sqs.orders. Your AWS SDK code does not change; you only override the endpoint to point at the connector.
Overview
CreateQueue registers the queue in the connector's registry and binds it to a KubeMQ Queue channel. SendMessage writes to that channel; ReceiveMessage returns the message plus an opaque receipt handle and hides it for the visibility window; DeleteMessage(receiptHandle) acks it off the queue. If a consumer never deletes, a sweeper NAcks the message back to the tail at visibility expiry and ApproximateReceiveCount increments.
| SQS operation | KubeMQ mapping | Notes |
|---|---|---|
CreateQueue("orders") | Register channel sqs.orders | .fifo suffix makes it FIFO |
SendMessage | SendQueueMessage | Returns MessageId, MD5OfBody, MD5OfMessageAttributes |
SendMessageBatch | Batched send (≤ 10 entries) | Per-entry success/failure |
ReceiveMessage | Credit-driven Get long-poll | Receipt handle; hidden for the visibility window |
DeleteMessage | AckRange — message removed | By receipt handle |
DeleteMessageBatch | Batched ack (≤ 10 entries) | Per-entry success/failure |
ChangeMessageVisibility | Extend / shorten the hidden window | Per-message override |
| visibility expiry (no delete) | NAckRange — redelivered to the tail | ApproximateReceiveCount increments |
Long polling (WaitTimeSeconds up to 20) waits for messages; MaxNumberOfMessages pulls up to 10 at once. Message attributes (String / Number / Binary, ≤ 10) round-trip losslessly, and the only accepted system attribute is AWSTraceHeader.
How it works
A producer sends to a queue; the connector writes each message to the backing KubeMQ Queue channel. A consumer receives a message (which is then hidden for its visibility window) and deletes it by receipt handle to ack it off the queue.
A received message is hidden for its visibility window; deleting it by receipt handle acks it off the queue, while letting the window expire NAcks it back to the tail for redelivery.
Send and receive
The full lifecycle: CreateQueue → GetQueueUrl → SendMessage → ReceiveMessage → DeleteMessage. Each client overrides only the endpoint (KUBEMQ_AWS_URL, default http://localhost:4566) and supplies dummy static credentials — the connector's default accept-any mode does not cryptographically verify the SigV4 signature, but the SDK must still form a syntactically valid signed request. The region is not enforced.
package main
import (
"context"
"fmt"
"log"
"os"
"time"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/service/sqs"
)
func sqsClient(ctx context.Context) *sqs.Client {
url := "http://localhost:4566"
if v := os.Getenv("KUBEMQ_AWS_URL"); v != "" {
url = v
}
// Dummy static credentials are mandatory even in accept-any mode: the SDK
// must form a valid SigV4 request. The region is not enforced.
cfg, err := config.LoadDefaultConfig(ctx,
config.WithRegion("us-east-1"),
config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider("test", "test", "")),
)
if err != nil {
log.Fatalf("load config: %v", err)
}
return sqs.NewFromConfig(cfg, func(o *sqs.Options) { o.BaseEndpoint = aws.String(url) })
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
client := sqsClient(ctx)
// 1. CreateQueue → registers channel sqs.orders (idempotent on identical attrs).
created, err := client.CreateQueue(ctx, &sqs.CreateQueueInput{QueueName: aws.String("orders")})
if err != nil {
log.Fatalf("CreateQueue: %v", err)
}
queueURL := aws.ToString(created.QueueUrl)
fmt.Printf("CreateQueue: %s\n", queueURL)
// 2. SendMessage.
sent, err := client.SendMessage(ctx, &sqs.SendMessageInput{
QueueUrl: aws.String(queueURL),
MessageBody: aws.String("order #4242 — 3x widget"),
})
if err != nil {
log.Fatalf("SendMessage: %v", err)
}
fmt.Printf("SendMessage: MessageId=%s MD5OfBody=%s\n",
aws.ToString(sent.MessageId), aws.ToString(sent.MD5OfMessageBody))
// 3. ReceiveMessage (long-poll a few seconds so the message is ready).
recv, err := client.ReceiveMessage(ctx, &sqs.ReceiveMessageInput{
QueueUrl: aws.String(queueURL),
MaxNumberOfMessages: 1,
WaitTimeSeconds: 5,
})
if err != nil {
log.Fatalf("ReceiveMessage: %v", err)
}
if len(recv.Messages) != 1 {
log.Fatalf("expected 1 message, got %d", len(recv.Messages))
}
msg := recv.Messages[0]
fmt.Printf("ReceiveMessage: body=%q\n", aws.ToString(msg.Body))
// 4. DeleteMessage by receipt handle (acks the message off the queue).
if _, err := client.DeleteMessage(ctx, &sqs.DeleteMessageInput{
QueueUrl: aws.String(queueURL),
ReceiptHandle: msg.ReceiptHandle,
}); err != nil {
log.Fatalf("DeleteMessage: %v", err)
}
fmt.Println("DeleteMessage: ok (acked by receipt handle)")
}import os
import boto3
def sqs_client():
url = os.environ.get("KUBEMQ_AWS_URL", "http://localhost:4566")
# Dummy credentials are mandatory even in accept-any mode (the SDK must form
# a valid SigV4 request); the region is not enforced.
return boto3.client(
"sqs",
endpoint_url=url,
region_name="us-east-1",
aws_access_key_id="test",
aws_secret_access_key="test",
)
def main() -> None:
sqs = sqs_client()
# 1. CreateQueue → registers channel sqs.orders.
queue_url = sqs.create_queue(QueueName="orders")["QueueUrl"]
print(f"CreateQueue -> {queue_url}")
# 2. SendMessage.
body = "hello from boto3"
send = sqs.send_message(QueueUrl=queue_url, MessageBody=body)
print(f"SendMessage -> MessageId={send['MessageId']} MD5OfBody={send['MD5OfMessageBody']}")
# 3. ReceiveMessage (long-poll so the message is ready).
recv = sqs.receive_message(QueueUrl=queue_url, MaxNumberOfMessages=1, WaitTimeSeconds=5)
messages = recv.get("Messages", [])
if len(messages) != 1:
raise SystemExit(f"expected 1 message, got {len(messages)}")
msg = messages[0]
print(f"ReceiveMessage-> body={msg['Body']!r}")
# 4. DeleteMessage by receipt handle (acks it off the queue).
sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=msg["ReceiptHandle"])
print("DeleteMessage -> acknowledged by receipt handle")
if __name__ == "__main__":
main()import java.net.URI;
import java.util.List;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.sqs.SqsClient;
import software.amazon.awssdk.services.sqs.model.Message;
import software.amazon.awssdk.services.sqs.model.ReceiveMessageResponse;
import software.amazon.awssdk.services.sqs.model.SendMessageResponse;
public final class Main {
public static void main(String[] args) {
String url = System.getenv().getOrDefault("KUBEMQ_AWS_URL", "http://localhost:4566");
// Dummy credentials are mandatory even in accept-any mode (a valid SigV4
// request must be formed); the region is not enforced.
try (SqsClient sqs = SqsClient.builder()
.endpointOverride(URI.create(url))
.region(Region.US_EAST_1)
.credentialsProvider(StaticCredentialsProvider.create(
AwsBasicCredentials.create("test", "test")))
.build()) {
// 1. CreateQueue → registers channel sqs.orders.
String queueUrl = sqs.createQueue(b -> b.queueName("orders")).queueUrl();
System.out.println("CreateQueue -> " + queueUrl);
// 2. SendMessage.
SendMessageResponse sent = sqs.sendMessage(b -> b
.queueUrl(queueUrl)
.messageBody("order #1001"));
System.out.println("SendMessage -> MessageId=" + sent.messageId()
+ " MD5OfBody=" + sent.md5OfMessageBody());
// 3. ReceiveMessage (long-poll so the message is ready).
ReceiveMessageResponse recv = sqs.receiveMessage(b -> b
.queueUrl(queueUrl)
.maxNumberOfMessages(1)
.waitTimeSeconds(5));
List<Message> messages = recv.messages();
if (messages.size() != 1) {
throw new IllegalStateException("expected 1 message, got " + messages.size());
}
Message msg = messages.get(0);
System.out.println("ReceiveMessage-> body='" + msg.body() + "'");
// 4. DeleteMessage by receipt handle (acks it off the queue).
sqs.deleteMessage(b -> b.queueUrl(queueUrl).receiptHandle(msg.receiptHandle()));
System.out.println("DeleteMessage -> deleted by receipt handle");
}
}
}import {
SQSClient,
CreateQueueCommand,
SendMessageCommand,
ReceiveMessageCommand,
DeleteMessageCommand,
} from "@aws-sdk/client-sqs";
function sqsClient(): SQSClient {
// Dummy credentials are mandatory even in accept-any mode (the SDK must form a
// valid SigV4 request); the region is not enforced.
return new SQSClient({
endpoint: process.env["KUBEMQ_AWS_URL"] ?? "http://localhost:4566",
region: "us-east-1",
credentials: { accessKeyId: "test", secretAccessKey: "test" },
});
}
async function main(): Promise<void> {
const sqs = sqsClient();
// 1. CreateQueue → registers channel sqs.orders.
const created = await sqs.send(new CreateQueueCommand({ QueueName: "orders" }));
const queueUrl = created.QueueUrl!;
console.log(`CreateQueue -> ${queueUrl}`);
// 2. SendMessage.
const sent = await sqs.send(
new SendMessageCommand({ QueueUrl: queueUrl, MessageBody: "order #42: 3x widgets" }),
);
console.log(`SendMessage -> MessageId=${sent.MessageId} MD5OfBody=${sent.MD5OfMessageBody}`);
// 3. ReceiveMessage (long-poll so the message is ready).
const recv = await sqs.send(
new ReceiveMessageCommand({ QueueUrl: queueUrl, MaxNumberOfMessages: 1, WaitTimeSeconds: 5 }),
);
const msg = recv.Messages?.[0];
if (!msg) throw new Error("expected 1 message, got 0");
console.log(`ReceiveMessage -> body="${msg.Body}"`);
// 4. DeleteMessage by receipt handle (acks it off the queue).
await sqs.send(new DeleteMessageCommand({ QueueUrl: queueUrl, ReceiptHandle: msg.ReceiptHandle! }));
console.log("DeleteMessage -> acked by receipt handle");
}
main().catch((err) => {
console.error(err);
process.exit(1);
});using Amazon.Runtime;
using Amazon.SQS;
using Amazon.SQS.Model;
// Dummy credentials are mandatory even in accept-any mode (a valid SigV4 request
// must be formed); the region is not enforced. ServiceURL carries the full
// http://host:port, so leave UseHttp=false to avoid the port being dropped.
var config = new AmazonSQSConfig
{
ServiceURL = Environment.GetEnvironmentVariable("KUBEMQ_AWS_URL") ?? "http://localhost:4566",
AuthenticationRegion = "us-east-1",
};
using var sqs = new AmazonSQSClient(new BasicAWSCredentials("test", "test"), config);
// 1. CreateQueue → registers channel sqs.orders.
var created = await sqs.CreateQueueAsync(new CreateQueueRequest { QueueName = "orders" });
Console.WriteLine($"CreateQueue -> {created.QueueUrl}");
// 2. SendMessage.
var sent = await sqs.SendMessageAsync(new SendMessageRequest
{
QueueUrl = created.QueueUrl,
MessageBody = "order #1001",
});
Console.WriteLine($"SendMessage -> MessageId={sent.MessageId} MD5OfBody={sent.MD5OfMessageBody}");
// 3. ReceiveMessage (long-poll so the message is ready).
var recv = await sqs.ReceiveMessageAsync(new ReceiveMessageRequest
{
QueueUrl = created.QueueUrl,
MaxNumberOfMessages = 1,
WaitTimeSeconds = 5,
});
if (recv.Messages.Count != 1)
throw new InvalidOperationException($"expected 1 message, got {recv.Messages.Count}");
var msg = recv.Messages[0];
Console.WriteLine($"ReceiveMessage -> body='{msg.Body}'");
// 4. DeleteMessage by receipt handle (acks it off the queue).
await sqs.DeleteMessageAsync(new DeleteMessageRequest
{
QueueUrl = created.QueueUrl,
ReceiptHandle = msg.ReceiptHandle,
});
Console.WriteLine("DeleteMessage -> deleted by receipt handle");# frozen_string_literal: true
require "aws-sdk-sqs"
url = ENV.fetch("KUBEMQ_AWS_URL", "http://localhost:4566")
# Dummy credentials are mandatory even in accept-any mode (the SDK must form a
# valid SigV4 request); the region is not enforced.
sqs = Aws::SQS::Client.new(
endpoint: url,
region: "us-east-1",
access_key_id: "test",
secret_access_key: "test"
)
# 1. CreateQueue → registers channel sqs.orders.
queue_url = sqs.create_queue(queue_name: "orders").queue_url
puts "CreateQueue -> #{queue_url}"
# 2. SendMessage.
send_resp = sqs.send_message(queue_url: queue_url, message_body: "order #1138 — 2 widgets")
puts "SendMessage -> MessageId=#{send_resp.message_id} MD5OfBody=#{send_resp.md5_of_message_body}"
# 3. ReceiveMessage (long-poll so the message is ready).
recv = sqs.receive_message(queue_url: queue_url, max_number_of_messages: 1, wait_time_seconds: 5)
raise "no message received" if recv.messages.empty?
msg = recv.messages.first
puts "ReceiveMessage-> Body=#{msg.body.inspect}"
# 4. DeleteMessage by receipt handle (acks it off the queue).
sqs.delete_message(queue_url: queue_url, receipt_handle: msg.receipt_handle)
puts "DeleteMessage -> ok (acked)"The AWS SDK for Ruby ships an SQS-only plugin that rewrites the request path to the full queue URL. Because the connector dispatches only on POST / and GET / (it carries the action in the request, not the path), remove that plugin once at startup so send/receive/delete reach the connector: Aws::SQS::Client.remove_plugin(Aws::SQS::Plugins::QueueUrls). CreateQueue/GetQueueUrl carry no queue URL and work either way.
use aws_config::{BehaviorVersion, Region};
use aws_credential_types::Credentials;
use std::error::Error;
async fn sqs_client() -> aws_sdk_sqs::Client {
let url = std::env::var("KUBEMQ_AWS_URL").unwrap_or_else(|_| "http://localhost:4566".into());
// Dummy credentials are mandatory even in accept-any mode (a valid SigV4
// request must be formed); the region is not enforced.
let conf = aws_config::defaults(BehaviorVersion::latest())
.region(Region::new("us-east-1"))
.credentials_provider(Credentials::new("test", "test", None, None, "static"))
.endpoint_url(url)
.load()
.await;
aws_sdk_sqs::Client::new(&conf)
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
let sqs = sqs_client().await;
// 1. CreateQueue → registers channel sqs.orders.
let url = sqs
.create_queue()
.queue_name("orders")
.send()
.await?
.queue_url
.ok_or("CreateQueue returned no URL")?;
println!("CreateQueue -> {url}");
// 2. SendMessage.
let sent = sqs.send_message().queue_url(&url).message_body("order-42").send().await?;
println!(
"SendMessage -> MessageId={} MD5OfBody={}",
sent.message_id().unwrap_or("<none>"),
sent.md5_of_message_body().unwrap_or("<none>")
);
// 3. ReceiveMessage (long-poll so the message is ready).
let recv = sqs
.receive_message()
.queue_url(&url)
.max_number_of_messages(1)
.wait_time_seconds(5)
.send()
.await?;
let messages = recv.messages();
let msg = messages.first().ok_or("expected 1 message, got 0")?;
println!("ReceiveMessage-> body='{}'", msg.body().unwrap_or_default());
// 4. DeleteMessage by receipt handle (acks it off the queue).
let handle = msg.receipt_handle().ok_or("message has no receipt handle")?;
sqs.delete_message().queue_url(&url).receipt_handle(handle).send().await?;
println!("DeleteMessage -> ok (acked)");
Ok(())
}Receipt handles and in-flight messages are node-local. A receipt handle minted on one node is rejected on another (ReceiptHandleIsInvalid). In a clustered deployment, place a sticky load balancer (session affinity) in front of the connector so a consumer's receive, delete, and visibility-change calls all land on the same node. Single-node deployments are unaffected.
Batch and message attributes
SendMessageBatch and DeleteMessageBatch take up to 10 entries and return per-entry success/failure, so an oversize or malformed entry fails on its own without failing the whole batch. Combine batch sends with long polling (WaitTimeSeconds=20, MaxNumberOfMessages=10) to drain efficiently.
import os
import boto3
sqs = boto3.client(
"sqs",
endpoint_url=os.environ.get("KUBEMQ_AWS_URL", "http://localhost:4566"),
region_name="us-east-1",
aws_access_key_id="test",
aws_secret_access_key="test",
)
url = sqs.create_queue(QueueName="events")["QueueUrl"]
# Send a batch of 9 entries. Batch results are per-entry: a single bad entry
# comes back in Failed without failing the others.
entries = [{"Id": f"m{i}", "MessageBody": f"event-{i}"} for i in range(9)]
batch = sqs.send_message_batch(QueueUrl=url, Entries=entries)
print(f"SendMessageBatch -> {len(batch.get('Successful', []))} Successful, "
f"{len(batch.get('Failed', []))} Failed")
# Long-poll-drain up to 10 at a time.
drained: list[dict] = []
while len(drained) < 9:
recv = sqs.receive_message(QueueUrl=url, MaxNumberOfMessages=10, WaitTimeSeconds=20)
msgs = recv.get("Messages", [])
if not msgs:
break
drained.extend(msgs)
# Delete them in one batch.
del_entries = [{"Id": f"d{i}", "ReceiptHandle": m["ReceiptHandle"]} for i, m in enumerate(drained)]
result = sqs.delete_message_batch(QueueUrl=url, Entries=del_entries)
print(f"DeleteMessageBatch -> {len(result.get('Successful', []))} deleted")Typed message attributes (String / Number / Binary, up to 10) round-trip losslessly, and MD5OfMessageAttributes is returned alongside MD5OfBody. The only accepted system attribute is AWSTraceHeader; other system attributes are rejected. The full attribute-to-tag mapping is in the channel mapping reference.
Empty-queue short polls have a ~1 s latency floor, and ApproximateNumberOfMessagesDelayed is always "0" (the connector does not track delayed counts). Use WaitTimeSeconds for long polling rather than tight short-poll loops.
Visibility timeout and redelivery
A received message is hidden for the visibility window (per-request VisibilityTimeout, else the queue default, else 30 s). If you do not delete it before the window expires, a sweeper NAcks it back to the tail and ApproximateReceiveCount increments — the message is redelivered. ChangeMessageVisibility extends or shortens the window for an in-flight message.
import time
# Receive with a 1-second visibility window, do NOT delete, then receive again:
# the message is redelivered and ApproximateReceiveCount goes from 1 to 2.
first = sqs.receive_message(
QueueUrl=url,
MaxNumberOfMessages=1,
VisibilityTimeout=1,
AttributeNames=["ApproximateReceiveCount"],
)["Messages"][0]
print(f"receive #1: count={first['Attributes']['ApproximateReceiveCount']}")
time.sleep(2) # let the visibility window expire → redelivery
second = sqs.receive_message(
QueueUrl=url,
MaxNumberOfMessages=1,
WaitTimeSeconds=5,
AttributeNames=["ApproximateReceiveCount"],
)["Messages"][0]
print(f"receive #2: count={second['Attributes']['ApproximateReceiveCount']}") # → 2 (redelivered)FIFO queues, ordering, and dedup
A queue whose name ends in .fifo is a FIFO queue. MessageGroupId is required on send and preserves per-group order; MessageDeduplicationId (or content-based dedup) suppresses duplicates within a 5-minute window. Each FIFO group maps to its own per-group channel sqs.{name}.fifo.g.{enc(group)}, where enc percent-encodes bytes outside [a-zA-Z0-9_-]. Per-message DelaySeconds is rejected on FIFO queues.
import os
import boto3
from botocore.exceptions import ClientError
sqs = boto3.client(
"sqs",
endpoint_url=os.environ.get("KUBEMQ_AWS_URL", "http://localhost:4566"),
region_name="us-east-1",
aws_access_key_id="test",
aws_secret_access_key="test",
)
# A .fifo queue with content-based dedup. MessageGroupId is required on send.
url = sqs.create_queue(
QueueName="tasks.fifo",
Attributes={"FifoQueue": "true", "ContentBasedDeduplication": "true"},
)["QueueUrl"]
sent_ids = []
for i in range(1, 4):
resp = sqs.send_message(
QueueUrl=url,
MessageBody=f"task-{i}",
MessageGroupId="group-A",
MessageDeduplicationId=f"dedup-{i}",
)
sent_ids.append(resp["MessageId"])
# SequenceNumber is 20 digits. On send it is a UnixNano timestamp; on receive
# it is the true broker sequence — both strictly increasing per group.
print(f"sent task-{i} SequenceNumber={resp['SequenceNumber']}")
# Re-sending with the same MessageDeduplicationId returns the ORIGINAL MessageId
# and does not re-enqueue.
dup = sqs.send_message(
QueueUrl=url, MessageBody="task-1", MessageGroupId="group-A", MessageDeduplicationId="dedup-1"
)
assert dup["MessageId"] == sent_ids[0] # duplicate suppressed
# Per-message DelaySeconds is rejected on FIFO queues.
try:
sqs.send_message(
QueueUrl=url, MessageBody="late", MessageGroupId="group-A",
MessageDeduplicationId="dedup-late", DelaySeconds=5,
)
except ClientError as e:
print(f"DelaySeconds on FIFO -> {e.response['Error']['Code']}") # InvalidParameterValueThe FIFO SequenceNumber differs between send and receive: on send it is a UnixNano timestamp, on receive it is the true broker sequence. Both are strictly increasing within a group — do not compare a send-side number against a receive-side one.
DLQ and redrive
Set a RedrivePolicy on a queue (a deadLetterTargetArn plus a maxReceiveCount) and the broker moves a message to the dead-letter queue once it has been received maxReceiveCount times without being deleted. The DLQ message carries a DeadLetterQueueSourceArn identifying the source. See the reliability guide for the full redrive walkthrough.
# Main queue + DLQ; redrive after 2 failed receives.
dlq_url = sqs.create_queue(QueueName="work-dlq")["QueueUrl"]
dlq_arn = sqs.get_queue_attributes(QueueUrl=dlq_url, AttributeNames=["QueueArn"])["Attributes"]["QueueArn"]
work_url = sqs.create_queue(QueueName="work")["QueueUrl"]
sqs.set_queue_attributes(
QueueUrl=work_url,
Attributes={"RedrivePolicy": f'{{"deadLetterTargetArn":"{dlq_arn}","maxReceiveCount":2}}'},
)What SQS queues do not have
The connector implements 18 SQS actions but is deliberately scoped — these AWS features are not supported:
- No RPC. SQS is point-to-point queueing, not request/reply; there is no gRPC responder.
- No KMS / server-side encryption (SSE). Encryption attributes are not honored.
- No
MessageBody-scope filtering (that is an SNS subscription feature; onlyMessageAttributesscope is supported there). - Body size limit ≤ 256 KiB for the aggregate request, as on AWS.
- Region is not enforced, and there is a single AccountId (
000000000000);QueueOwnerAWSAccountIdis accepted and ignored (no cross-account access).
Related
Was this page helpful?
SNS Topics
Create, subscribe, and publish to AWS SNS topics over KubeMQ — virtual topics with MessageAttributes filtering, PublishBatch, and FIFO fan-out to SQS queues.
SQS queues and consumers
The AWS connector SQS surface end to end — queue lifecycle, send/receive/delete, visibility timeouts, long polling, batch operations, and DLQ/redrive.