Skip to content

Curl Mini load timeout #1637

@kangkaisen

Description

@kangkaisen

Describe the bug
Yesterday, Our user run a large hive to doris job, which had 700+ doris Http mini load job, but a one http mini load get the following error:

curl: (28) Operation timed out after 600746 milliseconds with 0 out of -1 bytes received

the curl cmd:

2019-08-13 16:32:16,962 ERROR curl -m 600 --location-trusted -u xxxx:xxxxxxxx -T localfile   http://xxxx:8410/api/data_flow/app_log_sdk_mart_mv/_load?

1 I put the local file manually again, which was successfully. which means the file itself is normal.

2 I grep the fe log by load label, I found the http mini load request didn't reach the doris FE Server

3 After check the network monitor, I ensure the network is normal.

4 Finally, I check the TCP monitor, I found:

    7150 times the listen queue of a socket overflowed
    7150 SYNs to LISTEN sockets dropped

    TCPBacklogDrop: 433410

So I think the reason for curl cmd failed is TCP ListenOverflows.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions