Skip to content

[feat-1368][logminer]add enableFetchAll #1369

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
FlechazoW merged 3 commits into from
Nov 4, 2022
Merged

[feat-1368][logminer]add enableFetchAll #1369

FlechazoW merged 3 commits into from
Nov 4, 2022

Conversation

@mggger
Copy link
Contributor

@mggger mggger commented Nov 2, 2022 •
edited
Loading

Purpose of this pull request

Support oracle logminer historical + incremental data sync

Which issue you fix

Fixes #1368

Checklist:

  • I have executed the 'mvn spotless:apply' command to format my code.
  • I have a meaningful commit message (including the issue id, the template of commit message is '[label-type-#issue-id][fixed-module] a meaningful commit message.')
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have made corresponding changes to the documentation.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have checked my code and corrected any misspellings.
  • My commit is only one. (If there are multiple commits, you can use 'git squash' to compress multiple commits into one.)

@mggger
Copy link
Contributor Author

mggger commented Nov 2, 2022

@yanghuaiGit @FlechazoW please review it

新增enableFetchAll是否开启同步存量数据

  1. 获取要同步的表 (只能为单表， 不支持正测匹配多个表)
  2. 以ROW SHARE的方式锁定表
lock table xx IN ROW SHARE MODE
  1. 获取当前的SCN
select scn, seg_owner, table_name from v$logmnr_contents 
	where seg_owner='%s' and table_name='%s' order by scn desc
  1. 对这个表， 生成create table ddl事件
  2. 释放锁
rollback
  1. 根据SCN号， select数据，转换成column row data
select * from xxx as of scn xxx

注意: 不支持断点续传，同步存量数据失败时，需要手动删除目标表

@mggger mggger force-pushed the feat_1368 branch 2 times, most recently from f0ba988 to 123232d Compare November 3, 2022 11:08
FlechazoW
FlechazoW reviewed Nov 3, 2022
View reviewed changes
...-oraclelogminer/src/main/java/com/dtstack/chunjun/connector/oraclelogminer/util/SqlUtil.java
* 构建需要采集操作类型字符串的过滤条件
*
* @param listenerOptions 需要采集操作类型字符串 delete,insert,update
* @param ddlSkip 需要采集ddl数据
Copy link
Member

@FlechazoW FlechazoW Nov 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'ddlSkip' means skipping ddl or not.

FlechazoW
FlechazoW approved these changes Nov 3, 2022
View reviewed changes
Copy link
Member

@FlechazoW FlechazoW left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

FlechazoW
FlechazoW approved these changes Nov 4, 2022
View reviewed changes
Copy link
Member

@FlechazoW FlechazoW left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@FlechazoW FlechazoW merged commit 65c25e0 into DTStack:master Nov 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@FlechazoW FlechazoW FlechazoW approved these changes

@ll076110 ll076110 ll076110 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature][oracle][logminer]oracle logminer supports historical + incremental data sync

3 participants

@mggger @FlechazoW @ll076110