Kinesis
Kubemq kinesis target connector allows services using kubemq server to access aws kinesis service.

Prerequisites

The following required to run the aws-kinesis target connector:
  • kubemq cluster
  • aws account with kinesis active service
  • kubemq-source deployment

Configuration

sns target connector configuration properties:
Properties Key
Required
Description
Example
aws_key
yes
aws key
aws key supplied by aws
aws_secret_key
yes
aws secret key
aws secret key supplied by aws
region
yes
region
aws region
token
no
aws token ("default" empty string
aws token
Example:
1
bindings:
2
- name: kubemq-query-aws-kinesis
3
source:
4
kind: kubemq.query
5
name: kubemq-query
6
properties:
7
address: "kubemq-cluster:50000"
8
client_id: "kubemq-query-aws-kinesis"
9
auth_token: ""
10
channel: "query.aws.kinesis"
11
group: ""
12
auto_reconnect: "true"
13
reconnect_interval_seconds: "1"
14
max_reconnects: "0"
15
target:
16
kind: aws.kinesis
17
name: aws-kinesis
18
properties:
19
aws_key: "id"
20
aws_secret_key: 'json'
21
region: "region"
22
token: ""
Copied!

Usage

List Streams

list kinesis streams
List Streams:
Metadata Key
Required
Description
Possible values
method
yes
type of method
"list_streams"
Example:
1
{
2
"metadata": {
3
"method": "list_streams"
4
},
5
"data": null
6
}
Copied!

List Stream Consumers

list kinesis Stream Consumers.
List Stream Consumers:
Metadata Key
Required
Description
Possible values
method
yes
type of method
"list_stream_consumers"
stream_arn
yes
aws stream arn of the desired stream
"arn::mystream"
Example:
1
{
2
"metadata": {
3
"method": "list_stream_consumers",
4
"stream_arn": "arn::mystream"
5
},
6
"data": null
7
}
Copied!

Create Stream

Create a kinesis Stream.
Create Stream:
Metadata Key
Required
Description
Possible values
method
yes
type of method
"create_stream"
stream_name
yes
aws stream name as string
"string"
shard_count
no
number of shards to create (default 1)
"1"
Example:
1
{
2
"metadata": {
3
"method": "create_stream",
4
"stream_name": "my_stream",
5
"shard_count": "1"
6
},
7
"data": null
8
}
Copied!

List Shards

list stream Shards .
List Shards:
Metadata Key
Required
Description
Possible values
method
yes
type of method
"list_shards"
stream_name
yes
aws stream name as string
"string"
Example:
1
{
2
"metadata": {
3
"method": "list_shards",
4
"stream_name": "my_stream"
5
},
6
"data": null
7
}
Copied!

Get Shard Iterator

Get Shard Iterator used to preform get data .
Get Shard Iterator:
Metadata Key
Required
Description
Possible values
method
yes
type of method
"get_shard_iterator"
stream_name
yes
aws stream name as string
"string"
shard_id
yes
aws shard full id (can be acquired using list shard)
Example:
1
{
2
"metadata": {
3
"method": "get_shard_iterator",
4
"shard_iterator_type": "LATEST",
5
"stream_name": "my_stream",
6
"shard_id": "8619-AWE1"
7
},
8
"data": null
9
}
Copied!

Put Record

Send data to stream .
Put Record:
Metadata Key
Required
Description
Possible values
method
yes
type of method
"put_record"
stream_name
yes
aws stream name as string
"string"
partition_key
yes
determines which shard in the stream the data record is assigned to.
data
yes
Message to send to stream
string
Example:
1
{
2
"metadata": {
3
"method": "put_record",
4
"partition_key": "0356",
5
"stream_name": "my_stream"
6
},
7
"data": "eyJteV9yZXN1bHQiOiJvayJ9"
8
}
Copied!

Put Records

Send multi data to a stream .
Put Records:
Metadata Key
Required
Description
Possible values
method
yes
type of method
"put_records"
stream_name
yes
aws stream name as string
"string"
data
yes
Key value pair of partition_key(string) and message([]byte)
key value of partition_key and message as key value pair https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html
Example:
1
{
2
"metadata": {
3
"method": "put_records",
4
"stream_name": "my_stream"
5
},
6
"data": "eyIxIjoiZXlKdGVWOXlaWE4xYkhRaU9pSnZheUo5IiwiMiI6ImV5SnRlVjl5WlhOMWJIUXlJam9pYjJzaEluMD0ifQ=="
7
}
Copied!

Put Records

Send multi data to a stream .
Put Records:
Metadata Key
Required
Description
Possible values
method
yes
type of method
"put_records"
stream_name
yes
aws stream name as string
"string"
data
yes
Key value pair of partition_key(string) and message([]byte)
key value of partition_key and message as key value pair https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html
Example:
1
{
2
"metadata": {
3
"method": "put_records",
4
"stream_name": "my_stream"
5
},
6
"data": "eyIxIjoiZXlKdGVWOXlaWE4xYkhRaU9pSnZheUo5IiwiMiI6ImV5SnRlVjl5WlhOMWJIUXlJam9pYjJzaEluMD0ifQ=="
7
}
Copied!

Get Records

Get multi data from a stream .
Get Records:
Metadata Key
Required
Description
Possible values
method
yes
type of method
"get_records"
stream_name
yes
aws stream name as string
"string"
limit
no
Number of limit message to get (default "1")
"int value"
Example:
1
{
2
"metadata": {
3
"method": "put_records",
4
"stream_name": "my_stream",
5
"limit": "1"
6
},
7
"data": null
8
}
Copied!
Last modified 11mo ago