Skip to content

Doris 0.14.7 version, there is a backlog of stream load tasks  #5635

@hf200012

Description

@hf200012

This problem did not occur in version 0.13.15
The specific phenomenon is as follows :
Stream load task error, report: failed to call frontend service
The number of txns quickly reached the upper limit:errCode = 2, detailMessage = current running txns on db 11001 is 150, larger than limit 150

View backends heartbeat loss
https://github.com/hf200012/images/blob/master/doris/20210412145426.png

Restart BE, the log shows that the heartbeat of FE has not been received

To check transactions, it shows that many tasks are not over :
https://github.com/hf200012/images/blob/master/doris/20210412145444.png

Finally, only restart FE, and then return to normal

Fe master keeps refreshing the following logs :
2021-04-12 11:45:43,053 WARN (thrift-server-pool-297|673) [ReportHandler.putToQueue():182] the report queue size exceeds the limit: 120. current: 121
2021-04-12 11:45:43,060 INFO (tablet scheduler|39) [BeLoadRebalancer.selectAlternativeTabletsForCluster():105] get number of low load paths: 10, with medium: HDD
2021-04-12 11:45:43,061 INFO (tablet scheduler|39) [BeLoadRebalancer.selectAlternativeTabletsForCluster():186] select alternative tablets for cluster: default_cluster, medium: HDD, num: 2, detail: [5092902, 5092902]
2021-04-12 11:45:43,061 INFO (tablet scheduler|39) [BeLoadRebalancer.selectAlternativeTabletsForCluster():84] cluster is balance: default_cluster with medium: SSD. skip
2021-04-12 11:45:43,119 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17605972] is expired, remove it from transaction manager
2021-04-12 11:45:43,165 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17605973] is expired, remove it from transaction manager
2021-04-12 11:45:43,211 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17605974] is expired, remove it from transaction manager
2021-04-12 11:45:43,257 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17605979] is expired, remove it from transaction manager
2021-04-12 11:45:43,302 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17605981] is expired, remove it from transaction manager
2021-04-12 11:45:43,348 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17605984] is expired, remove it from transaction manager
2021-04-12 11:45:43,400 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17605985] is expired, remove it from transaction manager
2021-04-12 11:45:43,451 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17605986] is expired, remove it from transaction manager
2021-04-12 11:45:43,502 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606000] is expired, remove it from transaction manager
2021-04-12 11:45:43,554 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606001] is expired, remove it from transaction manager
2021-04-12 11:45:43,605 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606007] is expired, remove it from transaction manager
2021-04-12 11:45:43,663 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606008] is expired, remove it from transaction manager
2021-04-12 11:45:43,720 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606009] is expired, remove it from transaction manager
2021-04-12 11:45:43,771 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606017] is expired, remove it from transaction manager
2021-04-12 11:45:43,823 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606018] is expired, remove it from transaction manager
2021-04-12 11:45:43,874 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606031] is expired, remove it from transaction manager
2021-04-12 11:45:43,926 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606033] is expired, remove it from transaction manager
2021-04-12 11:45:43,977 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606034] is expired, remove it from transaction manager
2021-04-12 11:45:44,029 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606040] is expired, remove it from transaction manager
2021-04-12 11:45:44,061 INFO (tablet scheduler|39) [BeLoadRebalancer.selectAlternativeTabletsForCluster():105] get number of low load paths: 10, with medium: HDD
2021-04-12 11:45:44,062 INFO (tablet scheduler|39) [BeLoadRebalancer.selectAlternativeTabletsForCluster():186] select alternative tablets for cluster: default_cluster, medium: HDD, num: 2, detail: [5092902, 5092902]
2021-04-12 11:45:44,062 INFO (tablet scheduler|39) [BeLoadRebalancer.selectAlternativeTabletsForCluster():84] cluster is balance: default_cluster with medium: SSD. skip
2021-04-12 11:45:44,080 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606041] is expired, remove it from transaction manager
2021-04-12 11:45:44,132 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606045] is expired, remove it from transaction manager
2021-04-12 11:45:44,183 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606046] is expired, remove it from transaction manager
2021-04-12 11:45:44,235 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606047] is expired, remove it from transaction manager
2021-04-12 11:45:44,286 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606074] is expired, remove it from transaction manager
2021-04-12 11:45:44,338 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606075] is expired, remove it from transaction manager
2021-04-12 11:45:44,389 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606081] is expired, remove it from transaction manager
2021-04-12 11:45:44,441 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606082] is expired, remove it from transaction manager
2021-04-12 11:45:44,492 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606083] is expired, remove it from transaction manager
2021-04-12 11:45:44,544 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606089] is expired, remove it from transaction manager
2021-04-12 11:45:44,595 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606092] is expired, remove it from transaction manager
2021-04-12 11:45:44,647 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606107] is expired, remove it from transaction manager
2021-04-12 11:45:44,698 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606109] is expired, remove it from transaction manager
2021-04-12 11:45:44,750 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606111] is expired, remove it from transaction manager
2021-04-12 11:45:44,801 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606119] is expired, remove it from transaction manager
2021-04-12 11:45:44,853 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [1760612] is expired, remove it from transaction manager
2021-04-12 11:45:44,904 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606124] is expired, remove it from transaction manager
2021-04-12 11:45:44,956 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606125] is expired, remove it from transaction manager
2021-04-12 11:45:45,007 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606126] is expired, remove it from transaction manager
2021-04-12 11:45:45,022 WARN (thrift-server-pool-852|14940) [ReportHandler.putToQueue():182] the report queue size exceeds the limit: 120. current: 121
2021-04-12 11:45:45,062 INFO (tablet scheduler|39) [BeLoadRebalancer.selectAlternativeTabletsForCluster():105] get number of low load paths: 10, with medium: HDD
2021-04-12 11:45:45,063 INFO (tablet scheduler|39) [BeLoadRebalancer.selectAlternativeTabletsForCluster():186] select alternative tablets for cluster: default_cluster, medium: HDD, num: 2, detail: [5092902, 5092902]
2021-04-12 11:45:45,063 INFO (tablet scheduler|39) [BeLoadRebalancer.selectAlternativeTabletsForCluster():84] cluster is balance: default_cluster with medium: SSD. skip
2021-04-12 11:45:45,066 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606135] is expired, remove it from transaction manager
2021-04-12 11:45:45,123 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606136] is expired, remove it from transaction manager

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions