Skip to content

Conversation

@kangkaisen
Copy link
Contributor

@kangkaisen kangkaisen commented Sep 4, 2019

For #1746

1 Add invalid_argument and out_of_range check for to_bitmap function
2 change the RoaringBitmap single value type to uint32_t

For the following data:

10	100
11	xxxx
12	12000000000000000000000000000000000
13	130

The response of cat data | curl --location-trusted -u xxx:xxx -H "columns: id,id2, id2=to_bitmap(id2)" -T - http://xxx:8410/api/test/bitmap_test/_stream_load is :

{
    "TxnId": 23040,
    "Label": "e51499ef-8a55-43d3-bc20-e51f578e0bd4",
    "Status": "Fail",
    "Message": "too many filtered rows",
    "NumberTotalRows": 4,
    "NumberLoadedRows": 2,
    "NumberFilteredRows": 2,
    "NumberUnselectedRows": 0,
    "LoadBytes": 61,
    "LoadTimeMs": 101,
    "ErrorURL": "http://xxx:8410/api/_load_error_log?file=__shard_0/error_log_insert_stmt_f46b5ede95a9711-cd103dfa8d1456be_f46b5ede95a9711_cd103dfa8d1456be"

The error log is:

Reason: The to_bitmap function argument: xxxx type isn't integer family. src line: [[(11 xxxx)]];
Reason: The to_bitmap function argument: 12000000000000000000000000000000000 exceed unsigned integer max value 4294967295. src line: [[(12 12000000000000000000000000000000000)]];

int dest_index = ctx_idx++;
ExprContext* ctx = _dest_expr_ctx[dest_index];
void* value = ctx->get_value(_src_tuple_row);
std::string expr_error = ctx->get_error_msg();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we do this when value == nullptr? In this way, we can avoid doing this check in most cases

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update.

I also considered this pointer yesterday, but I found not all UDF return value is null when it set error msg to FunctionContext, but we could make all UDF used in data load follow
this convention.

Copy link
Contributor

@imay imay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@imay imay merged commit 3f22238 into apache:master Sep 5, 2019
swjtu-zhanglei pushed a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Jul 25, 2023
Fix issue http://39.106.86.136:8090/browse/CORE-2019

* Hard code to fix wrong column type string generated by FE
* [Improve](dynamic table) ensure all columns at schema_version should be persisted to rowset meta
  In some scenarios, some rowset meta with the same schema version may lose some columns:
  Eg.
  Load1 -> version(10) with schema [a, b, c, d, e]
  Load2 -> version(10) with schema [a, b, c] and has no extended columns

  Load2 will persist meta with [a, b, c] but Load1 will persist meta with [a, b, c, d, d]

* Check columns after sorted by unique id

---------

Co-authored-by: Gavin Chou <gavineaglechou@gmail.com>
Co-authored-by: eldenmoon <15605149486@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants