added function that chunks text inputs in vectorize.table() function #142#166
added function that chunks text inputs in vectorize.table() function #142#166ChuckHend merged 11 commits intoChuckHend:mainfrom
Conversation
function that chunk text
|
@harshtech123 , are you collaborating with @asr2003 on #162? There seems to be very similar implementation on both of these PRs. |
|
@ChuckHend no i created a pr before he did and i have no idea about what he was doing ... |
|
@ChuckHend i just wants a neat and clean pr thats why i created a new one , before this i made changes on #161 |
extension/src/api.rs
Outdated
| chunk_input: default!(bool, false), // New parameter to enable chunking | ||
| max_chunk_size: default!(i32, 1000), // New parameter for chunk size | ||
| ) -> Result<String> { | ||
| if chunk_input { | ||
| // Call chunk_table if chunking is enabled | ||
| chunk_table(table, &columns[0], max_chunk_size, "'chunked_data'")?; | ||
| } | ||
|
|
||
| // Proceed with the original table initialization logic |
There was a problem hiding this comment.
let's make chunk_table() a stand-alone function. So, remove these lines of code from this PR.
|
Maybe I am using the function wrong? |
|
|
@ChuckHend the function is ready to test 3 insert data to test on - |
extension/src/api.rs
Outdated
| let max_chunk_size = max_chunk_size as usize; | ||
|
|
||
| // Retrieve rows from the input table, ensuring column existence | ||
| let query = format!("SELECT id, {} FROM {}", column_name, input_table); |
There was a problem hiding this comment.
This requires the input_table to have a column "id", right? Could this instead follow the same convention as vectorize.table where we have a parameter for the primary_key, then select from that instead of hardcoding to "id"?
There was a problem hiding this comment.
added primary key as parameter and selecting id from that instead of hardcoding id
added primary key as parameter so we dont have to hardcode id
|
/split @asr2003 |
/claim #142
/closes #142
@ChuckHend can you please review this function any new changes are welcome!