fix: Make Sha2 fallback message more user-friendly #2213
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2213 +/- ##
============================================
+ Coverage 56.12% 58.55% +2.43%
- Complexity 976 1283 +307
============================================
Files 119 143 +24
Lines 11743 13226 +1483
Branches 2251 2363 +112
============================================
+ Hits 6591 7745 +1154
- Misses 4012 4250 +238
- Partials 1140 1231 +91 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| // expression, however DataFusion does not support that yet. | ||
| if (!expr.right.foldable) { | ||
| withInfo(expr, "For Sha2, non-foldable right argument is not supported") | ||
| withInfo(expr, "For Sha2, non literal numBits is not supported") |
There was a problem hiding this comment.
I think its called bitLength https://spark.apache.org/docs/latest/api/sql/#sha2
There was a problem hiding this comment.
Thanks for that link.
Looks like there is a bit of inconsistency between spark docs too.
If I look into
Spark4: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html#sha2(e:org.apache.spark.sql.Column,numBits:Int):org.apache.spark.sql.Column
and python-doc: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.sha2.html
they say numBits.
Since Spark4 doc says numBits, we can keep it the same.
Which issue does this PR close?
Closes #2193
Rationale for this change
With a recent change, the Sha2 fallback message is now not very user-friendly. This PR brings back the older message.
What changes are included in this PR?
A simple fallback message change.
How are these changes tested?
N/A