Skip to content

[feat][starrocks] add starrocks sync/sql connector #1237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
FlechazoW merged 3 commits into from
Sep 9, 2022
Merged

[feat][starrocks] add starrocks sync/sql connector #1237

FlechazoW merged 3 commits into from
Sep 9, 2022

Conversation

@Paddy0523
Copy link
Contributor

@Paddy0523 Paddy0523 commented Sep 8, 2022

Purpose of this pull request

  1. sync : add starrocks source/sink connector
  2. sql : add source/sink/lookup connector

Which issue you fix

Fixes # (issue).

Checklist:

  • I have executed the 'mvn spotless:apply' command to format my code.
  • I have a meaningful commit message (including the issue id, the template of commit message is '[label-type-#issue-id][fixed-module] a meaningful commit message.')
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have made corresponding changes to the documentation.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have checked my code and corrected any misspellings.
  • My commit is only one. (If there are multiple commits, you can use 'git squash' to compress multiple commits into one.)

@Paddy0523 Paddy0523 added the good first issue Good for newcomers label Sep 8, 2022
@Paddy0523 Paddy0523 force-pushed the feat_master_starrocks branch from 0ffd5cf to 0de4a23 Compare September 9, 2022 01:44
@libailin
Copy link
Contributor

libailin commented Sep 9, 2022

LGTM

@Paddy0523 Paddy0523 force-pushed the feat_master_starrocks branch from 0de4a23 to 684e2dd Compare September 9, 2022 02:15
FlechazoW
FlechazoW reviewed Sep 9, 2022
View reviewed changes
...src/main/java/com/dtstack/chunjun/connector/starrocks/source/be/StarRocksSourceBeReader.java Outdated
import com.starrocks.shade.org.apache.thrift.protocol.TProtocol;
import com.starrocks.shade.org.apache.thrift.transport.TSocket;
import com.starrocks.shade.org.apache.thrift.transport.TTransportException;
import com.starrocks.thrift.*;
Copy link
Member

@FlechazoW FlechazoW Sep 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not import *.

Copy link
Contributor Author

@Paddy0523 Paddy0523 Sep 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

FlechazoW
FlechazoW reviewed Sep 9, 2022
View reviewed changes
...c/main/java/com/dtstack/chunjun/connector/starrocks/source/be/StarRocksQueryPlanVisitor.java Outdated

import java.io.IOException;
import java.io.Serializable;
import java.util.*;
Copy link
Member

@FlechazoW FlechazoW Sep 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not import *.

Copy link
Contributor Author

@Paddy0523 Paddy0523 Sep 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

FlechazoW
FlechazoW reviewed Sep 9, 2022
View reviewed changes
chunjun-connectors/chunjun-connector-starrocks/pom.xml Outdated
<version>5.1.49</version>
</dependency>

<!-- <dependency>-->
Copy link
Member

@FlechazoW FlechazoW Sep 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove useless code.

Copy link
Contributor Author

@Paddy0523 Paddy0523 Sep 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

FlechazoW
FlechazoW reviewed Sep 9, 2022
View reviewed changes
chunjun-connectors/chunjun-connector-starrocks/pom.xml

<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
Copy link
Member

@FlechazoW FlechazoW Sep 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate dependencies are depended on the project, and we have other json-processed dependencies, like Gson, in the root pom.

FlechazoW
FlechazoW reviewed Sep 9, 2022
View reviewed changes
chunjun-connectors/chunjun-connector-starrocks/pom.xml
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.49</version>
Copy link
Member

@FlechazoW FlechazoW Sep 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this version same as other connector?

@Paddy0523 Paddy0523 force-pushed the feat_master_starrocks branch 2 times, most recently from 6b3b8c7 to 061c85c Compare September 9, 2022 05:44
FlechazoW
FlechazoW reviewed Sep 9, 2022
View reviewed changes
...rocks/src/main/java/com/dtstack/chunjun/connector/starrocks/source/StarRocksInputFormat.java Outdated
}

@Override
protected void closeInternal() {}
Copy link
Member

@FlechazoW FlechazoW Sep 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing to do here?

.../src/main/java/com/dtstack/chunjun/connector/starrocks/source/StarRocksLruTableFunction.java Outdated
}
}
}
beReader.close();
Copy link
Member

@FlechazoW FlechazoW Sep 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'close()' method should be called in 'finally' block.

...ks/src/main/java/com/dtstack/chunjun/connector/starrocks/source/be/StarRocksArrowReader.java
if (offsetOfBatchForRead < flinkRowsCount) {
return true;
}
this.close();
Copy link
Member

@FlechazoW FlechazoW Sep 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why call the 'close()' method here ?

Copy link
Contributor Author

@Paddy0523 Paddy0523 Sep 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed：close in finally

...c/main/java/com/dtstack/chunjun/connector/starrocks/source/be/StarRocksQueryPlanVisitor.java Outdated
.getPartitions()
.forEach(
(tabletId, tablet) -> {
int tabletCount = Integer.MAX_VALUE;
Copy link
Member

@FlechazoW FlechazoW Sep 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this code block should be called in a method.

@FlechazoW
Copy link
Member

FlechazoW commented Sep 9, 2022

Review done.

@Paddy0523 Paddy0523 force-pushed the feat_master_starrocks branch from 061c85c to 6188550 Compare September 9, 2022 07:49
@Paddy0523 Paddy0523 force-pushed the feat_master_starrocks branch from 6188550 to e796c65 Compare September 9, 2022 09:18
@FlechazoW FlechazoW merged commit f5aa7d3 into DTStack:master Sep 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@FlechazoW FlechazoW FlechazoW left review comments

@libailin libailin libailin approved these changes

Assignees

No one assigned

Labels

good first issue Good for newcomers

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Paddy0523 @libailin @FlechazoW