Skip to content

feat: add s3a transparent proxy (#276)#285

Merged
szbr9486 merged 3 commits intomainfrom
xuen
Sep 18, 2025
Merged

feat: add s3a transparent proxy (#276)#285
szbr9486 merged 3 commits intomainfrom
xuen

Conversation

@bigbigxu
Copy link
Copy Markdown
Contributor

No description provided.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add necessary annotations

public static String CACHE_TIMEOUT = "fs.cv.proxy.timeout";
public static String CLUSTER_NAME = "fs.cv.proxy.cluster.name";

public static String ZK_ENABLE_VALUE = "1";
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the ZK_ENABLE_VALUE means? Why used string value

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indicates whether the function is enabled. ZK stores bytes, so using strings is simpler and more direct.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could it be int type?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is essentially a judgment value, converted to boolean, it seems unnecessary to convert it again?

}

if (zkClient != null) {
zkClient.close();
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Catch possible exceptions

zkServer = conf.get(ZK_SERVER, "");
zkPath = conf.get(ZK_PATH, "/curvine/proxy_enable");
proxyEnable = conf.getBoolean(CACHE_ENABLE, true);
proxyTimeout = (int) conf.getTimeDuration(CACHE_TIMEOUT, "60s", TimeUnit.MILLISECONDS);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to throw an exception here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the configuration is wrong, it is reasonable to throw an exception during initialization

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's easy to encounter problems when throwing exceptions in the constructor

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What should I do if the configuration is wrong?

try {
zkClient = CuratorFrameworkFactory.newClient(
zkServer,
proxyTimeout,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both parameters are the same, maybe should make them difference

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They should be identical, hopefully failing fast on ZooKeeper exceptions.

}

public CurvineFileSystem getFs() {
if (!enable.get()) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Print warning log info

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method needs to be called every time a file is opened, which will result in a large amount of invalid logs.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean,enable is false normally?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has nothing to do with the default value, it depends on the user behavior

private CurvineFileSystem createCurvineFileSystem() throws Exception {
CompletableFuture<CurvineFileSystem> future = CompletableFuture.supplyAsync(() -> {
try {
String scheme = "cv";
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe the filesystem prefix scheme could be mutable

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyway, the scheme should be a constants variable

@bigbigxu bigbigxu force-pushed the xuen branch 3 times, most recently from 3ef1e0d to f71ba57 Compare September 17, 2025 11:58
}

public CurvineFileSystem getFs() {
if (!enable.get()) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean,enable is false normally?


private CuratorFramework zkClient;
private NodeCache nodeCache;
private final AtomicBoolean enable;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enable -> enabled


public static String ZK_SERVER = "fs.cv.proxy.zk.server";
public static String ZK_PATH = "fs.cv.proxy.zk.path";
public static String CACHE_ENABLE = "fs.cv.proxy.enable";
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CACHE_ ENABLE->CACHE_ENABLED
fs.cv.proxy.enable->fs.cv.proxy.enabled

public static String CACHE_TIMEOUT = "fs.cv.proxy.timeout";
public static String CLUSTER_NAME = "fs.cv.proxy.cluster.name";

public static String ZK_ENABLE_VALUE = "1";
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could it be int type?

@szbr9486 szbr9486 merged commit cb0b9d2 into main Sep 18, 2025
3 checks passed
@lzjqsdd lzjqsdd added the enhancement New feature or request label Sep 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants