KubeMQ Docs
KubeMQ.ioLogin / Register
  • Introduction
  • What's New
  • Getting Started
    • Quick Start
    • Build & Deploy
    • Create Cluster
      • Build & Deploy
      • Helm
      • Openshift
    • Create Connector
      • Build & Deploy
      • Helm
      • Openshift
    • Message Patterns
      • Queues
      • Pub/Sub
      • RPC
  • Learn
    • The Basics
      • Channels
      • Smart Routing
      • Grouping
    • Message Patterns
      • Queues
      • Pub/Sub
      • RPC
    • Access Control
      • Authentication
      • Authorization
      • Notifications
    • Clustering and HA
    • Connectors
      • KubeMQ Targets
      • KubeMQ Sources
      • KubeMQ Bridges
  • Configuration
    • Cluster
      • Set Cluster Name
      • Set Cluster Namespace
      • Set Persistent Volume
      • Set Cluster Replicas
      • Set Cluster Image
      • Set Cluster Security
      • Set Authentication
      • Set Authorization
      • Set Notification
      • Set License
      • Set gRPC Interface
      • Set Rest Interface
      • Set Api Interface
      • Set Store Settings
      • Set Queues Settings
      • Set Routing
      • Set Health Probe
      • Set Resources Limits
      • Set Logs
      • Set Node Selectors
    • Connectors
      • KubeMQ Targets
        • Standalone
          • Redis
          • Memcached
          • Postgres
          • Mysql
          • MSSql
          • Percona
          • Aerospike
          • ReThinkDB
          • MongoDB
          • Elastic Search
          • Cassandra
          • Couchbase
          • CockroachDB
          • Kafka
          • Nats
          • MQTT
          • ActiveMQ
          • IBM-MQ
          • Minio/S3
          • OpenFaas
          • HTTP
        • AWS
          • Athena
          • DynamoDB
          • Elastic Search
          • KeySpaces
          • MariaDB
          • MSSql
          • MySQL
          • Postgres
          • RedShift
          • RedShift Service
          • AmazonMQ
          • MSK
          • Kinesis
          • SQS
          • SNS
          • S3
          • Lambda
          • CloudWatch Logs
          • CloudWatch Events
          • CloudWatch Metrics
        • GCP
          • Redis
          • Memcached
          • Postgres
          • Mysql
          • BigQuery
          • BigTable
          • Firestore
          • Spanner
          • Firebase
          • Pub/Sub
          • Storage
          • Functions
        • Azure
          • Azure SQL
          • Mysql
          • Postgres
          • Blob
          • Files
          • Queue
          • Events Hub
          • Service Bus
        • Sources
          • Queue
          • Events
          • Events Store
          • Command
          • Query
      • KubeMQ Sources
        • HTTP
        • Messaging
          • Kafka
          • RabbitMQ
          • MQTT
          • ActiveMQ
          • IBM-MQ
          • Nats
        • AWS
          • AmazonMQ
          • MSK
          • SQS
        • GCP
          • Pub/Sub
        • Azure
          • EventHubs
          • ServiceBus
        • Targets
          • Queue
          • Events
          • Events Store
          • Command
          • Query
      • KubeMQ Bridges
        • Targets
          • Queue
          • Events
          • Events Store
          • Command
          • Query
        • Sources
          • Queue
          • Events
          • Events Store
          • Command
          • Query
    • Docker
  • HOW TO
    • Connect Your Cluster
    • Show Dashboard
    • Get Cluster Status
    • Get Cluster Logs
  • SDK
    • Java
    • Java (Springboot)
    • C# (.NET)
    • Go
    • Python
    • Node
    • Rest
  • Troubleshooting
    • Start Here
  • License
    • Open Source Software Notices
Powered by GitBook
On this page
  • Prerequisites
  • Configuration
  • Usage
  • List Streams
  • List Stream Consumers
  • Create Stream
  • List Shards
  • Get Shard Iterator
  • Put Record
  • Put Records
  • Put Records
  • Get Records

Was this helpful?

  1. Configuration
  2. Connectors
  3. KubeMQ Targets
  4. AWS

Kinesis

Kubemq kinesis target connector allows services using kubemq server to access aws kinesis service.

Prerequisites

The following required to run the aws-kinesis target connector:

  • kubemq cluster

  • aws account with kinesis active service

  • kubemq-source deployment

Configuration

sns target connector configuration properties:

Properties Key

Required

Description

Example

aws_key

yes

aws key

aws key supplied by aws

aws_secret_key

yes

aws secret key

aws secret key supplied by aws

region

yes

region

aws region

token

no

aws token ("default" empty string

aws token

Example:

bindings:
  - name: kubemq-query-aws-kinesis
    source:
      kind: kubemq.query
      name: kubemq-query
      properties:
        address: "kubemq-cluster:50000"
        client_id: "kubemq-query-aws-kinesis"
        auth_token: ""
        channel: "query.aws.kinesis"
        group:   ""
        auto_reconnect: "true"
        reconnect_interval_seconds: "1"
        max_reconnects: "0"
    target:
      kind: aws.kinesis
      name: aws-kinesis
      properties:
        aws_key: "id"
        aws_secret_key: 'json'
        region:  "region"
        token: ""

Usage

List Streams

list kinesis streams

List Streams:

Metadata Key

Required

Description

Possible values

method

yes

type of method

"list_streams"

Example:

{
  "metadata": {
    "method": "list_streams"
  },
  "data": null
}

List Stream Consumers

list kinesis Stream Consumers.

List Stream Consumers:

Metadata Key

Required

Description

Possible values

method

yes

type of method

"list_stream_consumers"

stream_arn

yes

aws stream arn of the desired stream

"arn::mystream"

Example:

{
  "metadata": {
    "method": "list_stream_consumers",
    "stream_arn": "arn::mystream"
  },
  "data": null
}

Create Stream

Create a kinesis Stream.

Create Stream:

Metadata Key

Required

Description

Possible values

method

yes

type of method

"create_stream"

stream_name

yes

aws stream name as string

"string"

shard_count

no

number of shards to create (default 1)

"1"

Example:

{
  "metadata": {
    "method": "create_stream",
    "stream_name": "my_stream",
    "shard_count": "1"
  },
  "data": null
}

List Shards

list stream Shards .

List Shards:

Metadata Key

Required

Description

Possible values

method

yes

type of method

"list_shards"

stream_name

yes

aws stream name as string

"string"

Example:

{
  "metadata": {
    "method": "list_shards",
    "stream_name": "my_stream"
  },
  "data": null
}

Get Shard Iterator

Get Shard Iterator used to preform get data .

Get Shard Iterator:

Metadata Key

Required

Description

Possible values

method

yes

type of method

"get_shard_iterator"

stream_name

yes

aws stream name as string

"string"

shard_iterator_type

yes

aws shard iterator type

shard_id

yes

aws shard full id (can be acquired using list shard)

Example:

{
  "metadata": {
    "method": "get_shard_iterator",
    "shard_iterator_type": "LATEST",
    "stream_name": "my_stream",
    "shard_id": "8619-AWE1"
  },
  "data": null
}

Put Record

Send data to stream .

Put Record:

Metadata Key

Required

Description

Possible values

method

yes

type of method

"put_record"

stream_name

yes

aws stream name as string

"string"

partition_key

yes

determines which shard in the stream the data record is assigned to.

data

yes

Message to send to stream

string

Example:

{
  "metadata": {
    "method": "put_record",
    "partition_key": "0356",
    "stream_name": "my_stream"
  },
  "data": "eyJteV9yZXN1bHQiOiJvayJ9"
}

Put Records

Send multi data to a stream .

Put Records:

Metadata Key

Required

Description

Possible values

method

yes

type of method

"put_records"

stream_name

yes

aws stream name as string

"string"

data

yes

Key value pair of partition_key(string) and message([]byte)

Example:

{
  "metadata": {
    "method": "put_records",
    "stream_name": "my_stream"
  },
  "data": "eyIxIjoiZXlKdGVWOXlaWE4xYkhRaU9pSnZheUo5IiwiMiI6ImV5SnRlVjl5WlhOMWJIUXlJam9pYjJzaEluMD0ifQ=="
}

Put Records

Send multi data to a stream .

Put Records:

Metadata Key

Required

Description

Possible values

method

yes

type of method

"put_records"

stream_name

yes

aws stream name as string

"string"

data

yes

Key value pair of partition_key(string) and message([]byte)

Example:

{
  "metadata": {
    "method": "put_records",
    "stream_name": "my_stream"
  },
  "data": "eyIxIjoiZXlKdGVWOXlaWE4xYkhRaU9pSnZheUo5IiwiMiI6ImV5SnRlVjl5WlhOMWJIUXlJam9pYjJzaEluMD0ifQ=="
}

Get Records

Get multi data from a stream .

Get Records:

Metadata Key

Required

Description

Possible values

method

yes

type of method

"get_records"

stream_name

yes

aws stream name as string

"string"

limit

no

Number of limit message to get (default "1")

"int value"

Example:

{
  "metadata": {
    "method": "put_records",
    "stream_name": "my_stream",
    "limit": "1"
  },
  "data": null
}
PreviousMSKNextSQS

Last updated 4 years ago

Was this helpful?

see

see

see

key value of partition_key and message as key value pair

key value of partition_key and message as key value pair

https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetShardIterator.html
https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetShardIterator.html
https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html
https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html
https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html