feat: Add Spark to_pretty_string function by ArnavBalyan · Pull Request #12780 · facebookincubator/velox

ArnavBalyan · 2025-03-24T17:58:55Z

Add support for to_pretty_string used by Spark.
Builds upon @jinchengchenghh's contribution here Add SparkSql function to_pretty_string #10359

ArnavBalyan · 2025-03-24T17:59:08Z

cc @rui-mo @jinchengchenghh can you PTAL thanks!

netlify · 2025-03-24T17:59:31Z

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`b093478`
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/6807748140394c0008681ce7

rui-mo

Thanks.

rui-mo · 2025-03-25T15:25:36Z

The document doesn't appear to render properly. Could you take a look?

rui-mo · 2025-03-25T15:46:38Z

It appears the date-to-string cast throws for runtime error, but here we return "NULL" for both user error and runtime error. Which is aligned with Spark?

velox/velox/expression/CastExpr.cpp

Lines 183 to 185 in ccd765d

if (!ue.isUserError()) {

throw;

}

rui-mo · 2025-03-25T15:53:14Z

CastExpr::applyToSelectedNoThrowLocal adopts the below logic for NULL.

velox/velox/expression/CastExpr-inl.h

Lines 206 to 214 in ccd765d

try {

func(row);

} catch (const VeloxException& e) {

if (!e.isUserError()) {

throw;

}

result->setNull(row, true);

} catch (const std::exception&) {

result->setNull(row, true);

Do you mean match the VeloxException, then catch std::exception? @rui-mo This is not addressed. @ArnavBalyan

The current implementation throws a user error when invalid timestamp is encountered, is this not what was intended? thanks!

Current implementation catches all the std::exception and converts them as user error, while in the cast, exception is thrown if not a user error, and set null result if it is. If the only differences with cast are the listed ones, we might align with cast here.

ArnavBalyan · 2025-03-27T13:09:06Z

cc @rui-mo if you could please take a look

rui-mo · 2025-03-28T15:33:58Z

Would you run the Spark expression fuzzer test for 2 minutes to make sure it works? You could specify --only to_pretty_string to test this function only.

The command for reference: https://github.com/facebookincubator/velox/blob/main/.github/workflows/scheduled.yml#L587-L596

ArnavBalyan · 2025-04-13T17:32:45Z

Would you run the Spark expression fuzzer test for 2 minutes to make sure it works? You could specify --only to_pretty_string to test this function only.

The command for reference: https://github.com/facebookincubator/velox/blob/main/.github/workflows/scheduled.yml#L587-L596

Done thanks:

E0413 17:29:58.007323 315291 ExpressionFuzzerVerifier.cpp:218] ==============================> Top 1 by number of rows processed
E0413 17:29:58.008945 315291 ExpressionFuzzerVerifier.cpp:220] Format: functionName numTimesSelected proportionOfTimesSelected numProcessedRows
E0413 17:29:58.009104 315291 ExpressionFuzzerVerifier.cpp:224] to_pretty_string 596890 100.00% 21313401
E0413 17:29:58.009243 315291 ExpressionFuzzerVerifier.cpp:231] ==============================> Bottom 1 by number of rows processed
E0413 17:29:58.009339 315291 ExpressionFuzzerVerifier.cpp:233] Format: functionName numTimesSelected proportionOfTimesSelected numProcessedRows
E0413 17:29:58.009405 315291 ExpressionFuzzerVerifier.cpp:238] to_pretty_string 596890 100.00% 21313401
E0413 17:29:58.009486 315291 ExpressionFuzzerVerifier.cpp:251] ==============================> All stats sorted by number of times the function was chosen
E0413 17:29:58.009547 315291 ExpressionFuzzerVerifier.cpp:253] Format: functionName numTimesSelected proportionOfTimesSelected numProcessedRows
E0413 17:29:58.009609 315291 ExpressionFuzzerVerifier.cpp:257] to_pretty_string 596890 100.00% 21313401
E0413 17:29:58.009671 315291 ExpressionFuzzerVerifier.cpp:486] Total test cases: 864029
E0413 17:29:58.009732 315291 ExpressionFuzzerVerifier.cpp:487] Total test cases verified in the reference DB: 864029
E0413 17:29:58.009791 315291 ExpressionFuzzerVerifier.cpp:489] Total test cases failed in both Velox and the reference DB: 0

rui-mo · 2025-04-17T08:21:43Z

@jinchengchenghh Do you have further comment? Thanks.

jinchengchenghh · 2025-04-17T09:01:31Z

LGTM @rui-mo
Please fix the code format @ArnavBalyan

assignUser

Please make sure CI passes, currently a test is failing. I'd recommend rebasing/merging main too.

ArnavBalyan · 2025-04-17T13:04:37Z

LGTM @rui-mo Please fix the code format @ArnavBalyan

Updated thanks!

ArnavBalyan · 2025-04-22T06:49:55Z

cc @assignUser can you pls rerun the tests

rui-mo · 2025-04-22T10:12:33Z

@ArnavBalyan You could apply the below diff to fix the format issue.

https://github.com/facebookincubator/velox/actions/runs/14532602668/attempts/2#summary-40925598950

ArnavBalyan · 2025-04-22T10:53:10Z

@ArnavBalyan You could apply the below diff to fix the format issue.

https://github.com/facebookincubator/velox/actions/runs/14532602668/attempts/2#summary-40925598950

Thanks, updated

ArnavBalyan · 2025-04-24T10:33:03Z

cc @assignUser could you pls take a look thanks!

assignUser

Thanks!

rui-mo · 2025-04-29T17:23:24Z

@ArnavBalyan It seems your updates are not pushed in, including #12780 (comment), could you confirm?

stale · 2025-07-28T20:07:34Z

This pull request has been automatically marked as stale because it has not had recent activity. If you'd still like this PR merged, please comment on the PR, make sure you've addressed reviewer comments, and rebase on the latest main. Thank you for your contributions!

Summary: - Add support for to_pretty_string used by Spark, this operator is used for `df.show()` to show the result to terminal. - Builds upon jinchengchenghh's contribution here #10359 and ArnavBalyan's contribution here #12780 Pull Request resolved: #16245 Reviewed By: tanjialiang Differential Revision: D92490039 Pulled By: xiaoxmeng fbshipit-source-id: fb70a1c8a8c9f7f7402f8626a29c810ad6a01048

ArnavBalyan requested review from assignUser and majetideepak as code owners March 24, 2025 17:58

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Mar 24, 2025

rui-mo reviewed Mar 25, 2025

View reviewed changes

rui-mo changed the title ~~feat: Add support for Spark ToPrettyString~~ feat: Add Spark to_pretty_string function Mar 25, 2025

ArnavBalyan force-pushed the arnavb/pretty-support-v branch from 0429c3e to bd0f369 Compare April 13, 2025 17:31

assignUser requested changes Apr 17, 2025

View reviewed changes

Comment thread velox/functions/sparksql/ToPrettyString.h Outdated

ArnavBalyan force-pushed the arnavb/pretty-support-v branch from bd0f369 to 07c3af1 Compare April 17, 2025 13:02

ArnavBalyan force-pushed the arnavb/pretty-support-v branch from 07c3af1 to 6f5963a Compare April 18, 2025 08:31

update

5568329

ArnavBalyan force-pushed the arnavb/pretty-support-v branch from 6f5963a to 5568329 Compare April 18, 2025 08:54

format

b093478

ArnavBalyan requested a review from assignUser April 23, 2025 05:00

assignUser approved these changes Apr 24, 2025

View reviewed changes

stale Bot added the stale label Jul 28, 2025

stale Bot closed this Aug 11, 2025

jinchengchenghh mentioned this pull request Feb 5, 2026

feat(sparksql): Add Spark to_pretty_string function #16245

Closed

FelixYBW mentioned this pull request Feb 7, 2026

[VL] useful Velox PRs not merged into upstream apache/gluten#11585

Open

	try {
	func(row);
	} catch (const VeloxException& e) {
	if (!e.isUserError()) {
	throw;
	}
	result->setNull(row, true);
	} catch (const std::exception&) {
	result->setNull(row, true);

Conversation

ArnavBalyan commented Mar 24, 2025

Uh oh!

ArnavBalyan commented Mar 24, 2025

Uh oh!

netlify Bot commented Mar 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for meta-velox canceled.

Uh oh!

rui-mo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArnavBalyan commented Mar 27, 2025

Uh oh!

rui-mo commented Mar 28, 2025

Uh oh!

ArnavBalyan commented Apr 13, 2025

Uh oh!

rui-mo commented Apr 17, 2025

Uh oh!

jinchengchenghh commented Apr 17, 2025

Uh oh!

assignUser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ArnavBalyan commented Apr 17, 2025

Uh oh!

ArnavBalyan commented Apr 22, 2025

Uh oh!

rui-mo commented Apr 22, 2025

Uh oh!

ArnavBalyan commented Apr 22, 2025

Uh oh!

ArnavBalyan commented Apr 24, 2025

Uh oh!

assignUser left a comment

Choose a reason for hiding this comment

Uh oh!

rui-mo commented Apr 29, 2025

Uh oh!

stale Bot commented Jul 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

netlify Bot commented Mar 24, 2025 •

edited

Loading