[SPARK-32432][SQL] Added support for reading ORC/Parquet files with SymlinkTextInputFormat #29330

moomindani · 2020-08-03T05:36:20Z

What changes were proposed in this pull request?

This pull-request is to add support for reading ORC/Parquet files with SymlinkTextInputFormat in Apache Spark.

Why are the changes needed?

Hive style symlink (SymlinkTextInputFormat) is commonly used in different analytic engines including prestodb and prestosql.
Currently SymlinkTextInputFormat works with JSON/CSV files but does not work with ORC/Parquet files in Apache Spark (and Apache Hive).
On the other hand, prestodb and prestosql support SymlinkTextInputFormat with ORC/Parquet files.
This pull-request is to add support for reading ORC/Parquet files with SymlinkTextInputFormat in Apache Spark.

See details in the JIRA. SPARK-32432

Does this PR introduce any user-facing change?

Yes.
Currently Spark returns exceptions if users try to use SymlinkTextInputFormat with ORC/Parquet files.
With this patch, Spark can handle symlink which indicates locations of ORC/Parquet files.

How was this patch tested?

I added a new test suite SymlinkSuite and confirmed it passed.

$ ./build/sbt "project hive" "test-only org.apache.spark.sql.hive.SymlinkSuite"

…kTextInputFormat

SparkQA · 2020-08-03T07:05:01Z

Test build #126954 has finished for PR 29330 at commit c97f003.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

moomindani · 2020-08-03T07:29:14Z

retest this please

SparkQA · 2020-08-03T11:12:15Z

Test build #126962 has finished for PR 29330 at commit c97f003.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

moomindani · 2020-08-05T10:28:17Z

retest this please

SparkQA · 2020-08-05T14:40:24Z

Test build #127093 has finished for PR 29330 at commit c97f003.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

moomindani · 2020-09-07T01:11:01Z

Could you please take a look and review this PR?

manuzhang · 2020-10-24T03:38:44Z

cc @cloud-fan @gengliangwang @viirya

SparkQA · 2020-11-11T15:25:14Z

Test build #130938 has finished for PR 29330 at commit c97f003.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

CHENXCHEN

Temporary files generated by hadoop will be read. @moomindani

CHENXCHEN · 2020-12-01T09:29:11Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/SymlinkTextInputFormatUtil.scala

+      fileSystem: FileSystem,
+      symlinkDir: Path): Seq[Path] = {
+
+    val symlinkIterator = fileSystem.listFiles(symlinkDir, true)


Hi, you should ignore the name start with "_" or ".", just like this hive implement, cause hadoop will generate temp file start with "_" or ".", we should ignore these files.

Thanks for the comments, I will reflect your comments on this.

github-actions · 2021-03-12T00:48:02Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

moomindani · 2021-03-12T01:49:07Z

Let me keep this PFR open.

maropu · 2021-03-13T05:21:15Z

ok to test

gatorsmile · 2021-04-26T20:57:28Z

cc @c21 Is it supported in FB?

c21 · 2021-04-28T03:21:57Z

@gatorsmile - we indeed support it, but the number of query everyday is very minimal, after checking the data.

gatorsmile · 2021-04-28T04:12:15Z

@c21 Can you help review the PR or open source your internal fix?

c21 · 2021-04-28T05:04:58Z

@gatorsmile - sure I will take a look, thanks.

jles01 · 2021-07-20T12:28:49Z

Hi,

Is there any updates on this ?

SparkQA · 2021-10-07T14:10:21Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48459/

SparkQA · 2021-10-07T14:18:45Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48459/

gengliangwang · 2021-11-10T13:26:29Z

cc @sadikovi as well

sadikovi

There are not that many details on the JIRA ticket and the provided links don't work - would appreciate it if you could update the description. I will do another pass once the inline comments are addressed

sadikovi · 2021-11-10T23:51:07Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala

          }
-          PartitionDirectory(values, files)
+          // Check leaf files since they might be symlink targets
+          if (files == Nil) {


nit: isEmpty?

sadikovi · 2021-11-10T23:54:27Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala

      val rootPaths: Seq[Path] = if (lazyPruningEnabled) {
-        Seq(tablePath)
+        if (SymlinkTextInputFormatUtil.isSymlinkTextFormat(inputFormat)) {
+          symlinkTargets


Should be replaced with SymlinkTextInputFormatUtil.getTargetPathsFromSymlink(fs, tablePath). I don't think we need the if statement above.

sadikovi · 2021-11-10T23:55:41Z

sql/hive/src/test/resources/data/files/sample1.csv

@@ -0,0 +1 @@
+1,2,3


There should be other examples of a CSV file in the resources, can you use those ones? The same applies to Parquet and ORC.

sadikovi · 2021-11-10T23:57:03Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/SymlinkSuite.scala

+
+      val testSymlinkPath1 = s"${temp.getCanonicalPath}/symlink_orc1.txt"
+      val testSymlinkPath2 = s"${temp.getCanonicalPath}/symlink_orc2.txt"
+      val prefix = System.getProperty("user.dir")


Can you explain? Is it possible to use a temporary path here?

sadikovi · 2021-11-10T23:57:45Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/SymlinkTextInputFormatUtil.scala

+import com.google.common.io.CharStreams
+import org.apache.hadoop.fs.{FileSystem, Path}
+
+object SymlinkTextInputFormatUtil {


Please add Javadoc to the class and the methods.

sadikovi · 2021-11-10T23:58:18Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/SymlinkTextInputFormatUtil.scala

+    inputFormat.equals("org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat")
+  }
+
+  // Mostly copied from SymlinkTextInputFormat#getTargetPathsFromSymlinksDirs of Hive 3.1


Is it compatible with other versions of Hive?

CHENXCHEN · 2022-02-15T10:29:29Z

Any update about this? thanks

github-actions · 2022-05-27T00:20:12Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

[SPARK-32432] Added support for reading ORC/Parquet files with Symlin…

c97f003

…kTextInputFormat

probot-autolabeler bot added the SQL label Aug 3, 2020

maropu changed the title ~~[SPARK-32432] Added support for reading ORC/Parquet files with SymlinkTextInputFormat~~ [SPARK-32432][SQL] Added support for reading ORC/Parquet files with SymlinkTextInputFormat Aug 3, 2020

CHENXCHEN suggested changes Dec 1, 2020

View reviewed changes

github-actions bot added the Stale label Mar 12, 2021

github-actions bot closed this Mar 13, 2021

maropu removed the Stale label Mar 13, 2021

maropu reopened this Mar 13, 2021

sadikovi suggested changes Nov 11, 2021

View reviewed changes

CHENXCHEN mentioned this pull request Mar 4, 2022

[SPARK-32432][SQL] Add support for reading ORC/Parquet files of SymlinkTextInputFormat table And Fix Analyze for SymlinkTextInputFormat table #35734

Closed

github-actions bot added the Stale label May 27, 2022

github-actions bot closed this May 28, 2022

		@@ -0,0 +1 @@
		1,2,3 No newline at end of file

[SPARK-32432][SQL] Added support for reading ORC/Parquet files with SymlinkTextInputFormat #29330

[SPARK-32432][SQL] Added support for reading ORC/Parquet files with SymlinkTextInputFormat #29330

Uh oh!

Conversation

moomindani commented Aug 3, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Aug 3, 2020

Uh oh!

moomindani commented Aug 3, 2020

Uh oh!

SparkQA commented Aug 3, 2020

Uh oh!

moomindani commented Aug 5, 2020

Uh oh!

SparkQA commented Aug 5, 2020

Uh oh!

moomindani commented Sep 7, 2020

Uh oh!

manuzhang commented Oct 24, 2020

Uh oh!

SparkQA commented Nov 11, 2020

Uh oh!

CHENXCHEN left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 12, 2021

Uh oh!

moomindani commented Mar 12, 2021

Uh oh!

maropu commented Mar 13, 2021

Uh oh!

gatorsmile commented Apr 26, 2021

Uh oh!

c21 commented Apr 28, 2021

Uh oh!

gatorsmile commented Apr 28, 2021

Uh oh!

c21 commented Apr 28, 2021

Uh oh!

jles01 commented Jul 20, 2021

Uh oh!

SparkQA commented Oct 7, 2021

Uh oh!

SparkQA commented Oct 7, 2021

Uh oh!

gengliangwang commented Nov 10, 2021

Uh oh!

sadikovi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CHENXCHEN commented Feb 15, 2022

Uh oh!

github-actions bot commented May 27, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

CHENXCHEN left a comment •

edited

Loading