capture nested fields in RECORD type col in bigquery extractor #462
Description
Is your feature request related to a problem? Please describe.
Meteor doesn't capture nested schema in a RECORD Type column in bigquery. Bigquery supports repeated and nested columns. Documentation - link
For example:
If the below is the DDL of the bigquery table -
CREATE TABLE `sample_project.sample_dataset.test_with_tags` (
non_record_field STRING,
record_field STRUCT<field_1 STRING, field_2 BOOL>
);
The schema of the table once extracted through meteor looks like this -
{
"schema": {
"columns": [
{
"name": "non_record_field",
"data_type": "STRING",
"properties": {
"attributes": {
"mode": "NULLABLE"
}
},
"is_nullable": true
},
{
"name": "record_field",
"data_type": "RECORD",
"properties": {
"attributes": {
"mode": "NULLABLE"
}
},
"is_nullable": true
}
]
}
}
Describe the solution you'd like
Meteor should also capture nested structure of the RECORD type column in bigquery. and resultant schema field in json can look something like this -
{
"schema": {
"columns": [
{
"name": "non_record_field",
"data_type": "STRING",
"properties": {
"attributes": {
"mode": "NULLABLE"
}
},
"is_nullable": true
},
{
"name": "record_field",
"data_type": "RECORD",
"properties": {
"attributes": {
"mode": "NULLABLE"
}
},
"is_nullable": true,
"schema" : {
"columns": [
{
"name": "field_1",
"data_type": "STRING",
"properties": {
"attributes": {
"mode": "NULLABLE"
}
},
"is_nullable": true
},
{
"name": "field_2",
"data_type": "BOOLEAN",
"properties": {
"attributes": {
"mode": "NULLABLE"
}
},
"is_nullable": true
}
]
}
}
]
}
}