BigQuery
Kubemq gcp-bigquery target connector allows services using kubemq server to access google bigquery server.

Prerequisites

The following are required to run the gcp-bigquery target connector:
  • kubemq cluster
  • gcp-bigquery set up
  • kubemq-source deployment

Configuration

bigquery target connector configuration properties:
Properties Key
Required
Description
Example
project_id
yes
gcp bigquery project_id
"/myproject"
credentials
yes
gcp credentials files
"<google json credentials"
Example:
1
bindings:
2
- name: kubemq-query-gcp-bigquery
3
source:
4
kind: kubemq.query
5
name: kubemq-query
6
properties:
7
address: "kubemq-cluster:50000"
8
client_id: "kubemq-query-gcp-bigquery-connector"
9
auth_token: ""
10
channel: "query.gcp.bigquery"
11
group: ""
12
auto_reconnect: "true"
13
reconnect_interval_seconds: "1"
14
max_reconnects: "0"
15
target:
16
kind: gcp.bigquery
17
name: gcp-bigquery
18
properties:
19
project_id: "id"
20
credentials: 'json'
Copied!

Usage

Query Request

query request.
Query metadata setting:
Metadata Key
Required
Description
Possible values
method
yes
type of method
"query"
query
yes
the query body
"select * from table"
Example:
1
{
2
"metadata": {
3
"method": "query",
4
"query": "select * from table"
5
},
6
"data": null
7
}
Copied!

Create Table Request

create a new table under data set
This method required a body of rows of string [bigquery.TableMetadata]
Example how to create the struct:
1
mySchema := bigquery.Schema{
2
{Name: "name", Type: bigquery.StringFieldType},
3
{Name: "age", Type: bigquery.IntegerFieldType},
4
}
5
6
metaData := &bigquery.TableMetadata{
7
Schema: mySchema,
8
ExpirationTime: time.Now().AddDate(2, 1, 0), // Table will deleted in 2 years and 1 month.
9
}
10
bSchema, err := json.Marshal(metaData)
Copied!
Create table metadata setting:
Metadata Key
Required
Description
Possible values
method
yes
type of method
"create_table"
dataset_id
yes
dataset to assign the table to
"your data set ID"
table_name
yes
table name
"unique name"
Example:
1
{
2
"metadata": {
3
"method": "create_table",
4
"dataset_id": "<mySet>",
5
"table_name": "<myTable>"
6
},
7
"data": "eyJOYW1lIjoiIiwiTG9jYXRpb24iOiIiLCJEZXNjcmlwdGlvbiI6IiIsIlNjaGVtYSI6W3siTmFtZSI6Im5hbWUiLCJEZXNjcmlwdGlvbiI6IiIsIlJlcGVhdGVkIjpmYWxzZSwiUmVxdWlyZWQiOmZhbHNlLCJUeXBlIjoiU1RSSU5HIiwiUG9saWN5VGFncyI6bnVsbCwiU2NoZW1hIjpudWxsfSx7Ik5hbWUiOiJhZ2UiLCJEZXNjcmlwdGlvbiI6IiIsIlJlcGVhdGVkIjpmYWxzZSwiUmVxdWlyZWQiOmZhbHNlLCJUeXBlIjoiSU5URUdFUiIsIlBvbGljeVRhZ3MiOm51bGwsIlNjaGVtYSI6bnVsbH1dLCJNYXRlcmlhbGl6ZWRWaWV3IjpudWxsLCJWaWV3UXVlcnkiOiIiLCJVc2VMZWdhY3lTUUwiOmZhbHNlLCJVc2VTdGFuZGFyZFNRTCI6ZmFsc2UsIlRpbWVQYXJ0aXRpb25pbmciOm51bGwsIlJhbmdlUGFydGl0aW9uaW5nIjpudWxsLCJSZXF1aXJlUGFydGl0aW9uRmlsdGVyIjpmYWxzZSwiQ2x1c3RlcmluZyI6bnVsbCwiRXhwaXJhdGlvblRpbWUiOiIyMDIyLTExLTI1VDEwOjQxOjI1LjIyNjQ1NyswMjowMCIsIkxhYmVscyI6bnVsbCwiRXh0ZXJuYWxEYXRhQ29uZmlnIjpudWxsLCJFbmNyeXB0aW9uQ29uZmlnIjpudWxsLCJGdWxsSUQiOiIiLCJUeXBlIjoiIiwiQ3JlYXRpb25UaW1lIjoiMDAwMS0wMS0wMVQwMDowMDowMFoiLCJMYXN0TW9kaWZpZWRUaW1lIjoiMDAwMS0wMS0wMVQwMDowMDowMFoiLCJOdW1CeXRlcyI6MCwiTnVtTG9uZ1Rlcm1CeXRlcyI6MCwiTnVtUm93cyI6MCwiU3RyZWFtaW5nQnVmZmVyIjpudWxsLCJFVGFnIjoiIn0="
8
}
Copied!

Delete Table Request

delete a new table under data set
Delete table metadata setting:
Metadata Key
Required
Description
Possible values
method
yes
type of method
"create_table"
dataset_id
yes
dataset to assign the table to
"your data set ID"
table_name
yes
table name
"unique name"
Example:
1
{
2
"metadata": {
3
"method": "delete_table",
4
"dataset_id": "<mySet>",
5
"table_name": "<myTable>"
6
},
7
"data":null
8
}
Copied!

Create Data Set Request

Create a Data Set
Create Data Set metadata setting:
Metadata Key
Required
Description
Possible values
method
yes
type of method
"create_data_set"
dataset_id
yes
dataset to assign the table to
"your data set ID"
location
yes
dataset location to set
Example:
1
{
2
"metadata": {
3
"method": "create_data_set",
4
"dataset_id": "<mySet>",
5
"location": "US"
6
},
7
"data":null
8
}
Copied!

Delete Data Set Request

delete a Data Set
Delete Data Set metadata setting:
Metadata Key
Required
Description
Possible values
method
yes
type of method
"delete_data_set"
dataset_id
yes
dataset to assign the table to
"your data set ID"
Example:
1
{
2
"metadata": {
3
"method": "delete_data_set",
4
"dataset_id": "<mySet>"
5
},
6
"data":null
7
}
Copied!

Get DataSets Request

get data sets.
Get DataSets setting:
Metadata Key
Required
Description
Possible values
method
yes
type of method
"get_data_sets"
Example:
1
{
2
"metadata": {
3
"method": "get_data_sets"
4
},
5
"data": null
6
}
Copied!

Get Table Info

get basic information on a table by name
Get table Info
Metadata Key
Required
Description
Possible values
method
yes
type of method
"get_table_info"
dataset_id
yes
dataset to assign the table to
"your data set ID"
table_name
yes
table name
"unique table name"
Example:
1
{
2
"metadata": {
3
"method": "get_table_info",
4
"dataset_id": "<mySet>",
5
"table_name": "<myTable>"
6
},
7
"data": null
8
}
Copied!

Insert To Table

insert rows to table
Insert To Table this method required a body of rows as json array
1
[
2
{"id":"id-1","service":{"id":"service_id_1","value":0.1},"time":"1980-10-10T00:00:00Z"},
3
{"id":"id-2","service":{"id":"service_id_2","value":0.1},"time":"1980-10-10T00:00:00Z"}
4
]
Copied!
Metadata Key
Required
Description
Possible values
method
yes
type of method
"insert"
dataset_id
yes
dataset to assign the table to
"your data set ID"
table_name
yes
table name
"unique table name"
Example:
1
{
2
"metadata": {
3
"method": "insert",
4
"dataset_id": "<mySet>",
5
"table_name": "<myTable>"
6
},
7
"data": "W3sgIm5hbWUiOiJteU5hbWU0IiwgImFnZSI6MjV9XQ=="
8
}
Copied!
Last modified 11mo ago