Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HADOOP-19207: [ABFS][FNSOverBlob] Response Handling of Blob Endpoint APIs and Metadata APIs #7210

Open
wants to merge 9 commits into
base: trunk
Choose a base branch
from

Conversation

anujmodi2021
Copy link
Contributor

@anujmodi2021 anujmodi2021 commented Dec 5, 2024

Description of PR

Target Jira: https://issues.apache.org/jira/browse/HADOOP-19207

As part of the new Fns Over Blob Support, there is a need to handle responses from 2 service endpoints of Azure Storage differently. Blob Endpoint API response format is different from DFS Endpoint APIs. DFS Endpoint primarily has response body in Json format where as BLOB Endpoint has XML.

In this PR following changes are included:

  1. Parsing List Status repsonse abstracted into DFS and Blob clients with additional schema files to support each of the endpoints due to difference in response body format
  2. Parsing of request headers abstracted into DFS and Blob clients due to difference in response header formats.
  3. Parsing on Storage Error repsonse abstracted into DFS and Blob clients due to difference in response body format.
  4. Handling getPathStatus(), listStatus(), getXAttr(), setXAttr() calls on implicit paths on Blob Endpoint.
  5. Handling Following behavioral differences from server in DFS and Blob Endpoint:
    • List Status on Non-existing path returns 404 on DFS and empty listing on Blob
    • List Status on File Path returns file in lisiting output on DFS and empty listing on Blob

How was this patch tested?

Existing test suite was ran and results added below to make sure existing DFS Endpoint support did not get disturbed.
New tests for FNS Blob need ingress on Blob endpoint to be supported which will be done in subsequest PRs.
Test Result: #7210 (comment)

Following JIRA tracks the test addition for new support: https://issues.apache.org/jira/browse/HADOOP-19234
Following JIRA tracks the ingress support for FNS Blob: https://issues.apache.org/jira/browse/HADOOP-19232

For code changes:

@anujmodi2021 anujmodi2021 marked this pull request as ready for review December 5, 2024 12:41
@anujmodi2021
Copy link
Contributor Author


:::: AGGREGATED TEST RESULT ::::

============================================================
HNS-OAuth

[WARNING] Tests run: 157, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 653, Failures: 0, Errors: 0, Skipped: 85
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 25
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 23

============================================================
HNS-SharedKey

[WARNING] Tests run: 157, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 653, Failures: 0, Errors: 0, Skipped: 37
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 25
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 10

============================================================
NonHNS-SharedKey

[WARNING] Tests run: 157, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 637, Failures: 0, Errors: 0, Skipped: 277
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 27
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 11

============================================================
AppendBlob-HNS-OAuth

[WARNING] Tests run: 157, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 653, Failures: 0, Errors: 0, Skipped: 87
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 49
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 23

Time taken: 107 mins 7 secs.

@hadoop-yetus

This comment was marked as outdated.

@hadoop-yetus

This comment was marked as outdated.

Copy link
Contributor

@snvijaya snvijaya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 58s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 7 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 40m 7s trunk passed
+1 💚 compile 0m 37s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 0m 34s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 0m 33s trunk passed
+1 💚 mvnsite 0m 39s trunk passed
+1 💚 javadoc 0m 43s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 34s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 1m 10s trunk passed
+1 💚 shadedclient 34m 18s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 28s the patch passed
+1 💚 compile 0m 31s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javac 0m 31s the patch passed
+1 💚 compile 0m 27s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 javac 0m 27s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 20s /results-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 1 new + 13 unchanged - 4 fixed = 14 total (was 17)
+1 💚 mvnsite 0m 30s the patch passed
+1 💚 javadoc 0m 27s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 24s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 1m 3s the patch passed
+1 💚 shadedclient 34m 2s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 20s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 38s The patch does not generate ASF License warnings.
122m 50s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7210/5/artifact/out/Dockerfile
GITHUB PR #7210
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 1c776a46fae2 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / cca8f1c
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7210/5/testReport/
Max. process+thread count 554 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7210/5/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants