Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] [OssResourceManager] The root path of the OSS configuration must not start with "/", which conflicts with the configuration of the History Server. #4088

Open
2 of 3 tasks
jiangwwwei opened this issue Dec 24, 2024 · 0 comments
Labels
Bug Something isn't working Waiting for reply Waiting for reply

Comments

@jiangwwwei
Copy link

jiangwwwei commented Dec 24, 2024

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

image

Dinky's OssResourceManager relies on the aws-sdk implementation, and when constructing requests using its API, the file path is the Object Name, which should not start with a "/" (the aws SDK will automatically add a "/" at the beginning of the object name when building the HTTP request). If the root path in Dinky's Oss configuration starts with "/", it will cause an error indicating an invalid object name.
image

However, the HistoryServer's request to oss is implemented by flink-s3-fs-presto, and it requests the direct path. If a "/" is not added at the beginning, it will report an error stating that it is not an absolute path.

What you expected to happen

Both the resource manager and the history server can access OSS normally.

How to reproduce

  1. Use OSS Resource
  2. In the root path configuration, add and do not add "/"
  3. Observe the file upload/download status of the Resource and the operation of the History Server.

Anything else

It is worth mentioning: In OssResourceManager, the "/" before the path in getFullDirectoryStructure() is specifically handled and removed, but other methods do not have this treatment.
image

Version

1.2.0

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@jiangwwwei jiangwwwei added Bug Something isn't working Waiting for reply Waiting for reply labels Dec 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working Waiting for reply Waiting for reply
Projects
None yet
Development

No branches or pull requests

1 participant