Skip to content

Commit

Permalink
ORC-1569: Remove HadoopShimsPre2_3, HadoopShimsPre2_6, HadoopShimsPre…
Browse files Browse the repository at this point in the history
…2_7 classes

### What changes were proposed in this pull request?

This PR aims to remove `HadoopShimsPre2_3`, `HadoopShimsPre2_6`, `HadoopShimsPre2_7` classes and use `HadoopShimsCurrent` always.

### Why are the changes needed?

1. `HadoopShimsCurrent` supports not only Apache Hadoop 3+ but also Apache Hadoop 2.7+.

2. Apache ORC 2.0 uses Hadoop 3.x shaded client doesn't need old shims for Hadoop 2.6 and olders.
    - #1509

In addition, Apache Spark community also has been using the shaded Hadoop client since Spark 3.2 (SPARK-33212) and dropped `Hadoop 2` profile via [SPARK-42452](https://issues.apache.org/jira/browse/SPARK-42452) completely at Spark 3.5.0.

### How was this patch tested?

Pass the CIs.

Closes #1724 from dongjoon-hyun/ORC-1569.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: William Hyun <william@apache.org>
  • Loading branch information
dongjoon-hyun authored and williamhyun committed Jan 4, 2024
1 parent c2fad3d commit e8374e3
Show file tree
Hide file tree
Showing 4 changed files with 2 additions and 202 deletions.
18 changes: 2 additions & 16 deletions java/core/src/java/org/apache/orc/impl/HadoopShimsFactory.java
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,6 @@ public class HadoopShimsFactory {

private static final String CURRENT_SHIM_NAME =
"org.apache.orc.impl.HadoopShimsCurrent";
private static final String PRE_2_6_SHIM_NAME =
"org.apache.orc.impl.HadoopShimsPre2_6";
private static final String PRE_2_7_SHIM_NAME =
"org.apache.orc.impl.HadoopShimsPre2_7";

private static HadoopShims SHIMS = null;

private static HadoopShims createShimByName(String name) {
Expand All @@ -57,18 +52,9 @@ public static synchronized HadoopShims get() {
int major = Integer.parseInt(versionParts[0]);
int minor = Integer.parseInt(versionParts[1]);
if (major < 2 || (major == 2 && minor < 7)) {
LOG.warn("Hadoop " + VersionInfo.getVersion() + " support is deprecated. " +
"Please upgrade to Hadoop 2.7.3 or above.");
}
if (major < 2 || (major == 2 && minor < 3)) {
SHIMS = new HadoopShimsPre2_3();
} else if (major == 2 && minor < 6) {
SHIMS = createShimByName(PRE_2_6_SHIM_NAME);
} else if (major == 2 && minor < 7) {
SHIMS = createShimByName(PRE_2_7_SHIM_NAME);
} else {
SHIMS = createShimByName(CURRENT_SHIM_NAME);
LOG.warn("Hadoop " + VersionInfo.getVersion() + " support is dropped.");
}
SHIMS = createShimByName(CURRENT_SHIM_NAME);
}
return SHIMS;
}
Expand Down
60 changes: 0 additions & 60 deletions java/shims/src/java/org/apache/orc/impl/HadoopShimsPre2_3.java

This file was deleted.

60 changes: 0 additions & 60 deletions java/shims/src/java/org/apache/orc/impl/HadoopShimsPre2_6.java

This file was deleted.

66 changes: 0 additions & 66 deletions java/shims/src/java/org/apache/orc/impl/HadoopShimsPre2_7.java

This file was deleted.

0 comments on commit e8374e3

Please sign in to comment.