Skip to content

feat: add sort expressions#3962

Closed
joelostblom wants to merge 3 commits intomainfrom
sort-functions
Closed

feat: add sort expressions#3962
joelostblom wants to merge 3 commits intomainfrom
sort-functions

Conversation

@joelostblom
Copy link
Contributor

I think it would be useful to have sort function in the expression language. My immediate use case is for vega/altair#3394 (review), where we need to sort an array returned by domain().

I'm not at all sure what the best implementation is here, or what tests are needed, so this PR is partly to move the discussion I started with @hydrosquall on slack and get additional input. I think that ideally there would just be one sort function that handles both numbers, strings, and mixed types, but that seems considerably more complex and I'm not comfortable writing it so I'm suggesting these simple ones which @hydrosquall showed me on slack and that directly maps to the built in JS array methods.

@joelostblom joelostblom requested a review from a team as a code owner August 26, 2024 12:59
@cloudflare-workers-and-pages
Copy link

cloudflare-workers-and-pages bot commented Sep 5, 2024

Deploying vega with  Cloudflare Pages  Cloudflare Pages

Latest commit: aabd3ac
Status: ✅  Deploy successful!
Preview URL: https://79a76ad7.vega-628.pages.dev
Branch Preview URL: https://sort-functions.vega-628.pages.dev

View logs

@domoritz
Copy link
Member

domoritz commented Sep 5, 2024

Thanks. I added a more generic comparator so we don't need two methods and added docs as well.

@joelostblom
Copy link
Contributor Author

That's great, thank you! Having just one method is much preferred.

@domoritz domoritz changed the title feat: Add sort expressions feat: add sort expressions Sep 6, 2024
@hydrosquall
Copy link
Member

Hey @joelostblom, thanks for putting this together!

Using the branch preview ( https://sort-functions.vega-628.pages.dev/#/ ), could you link to a sample Vega or Vega-Lite definition that would make use of this new expression? We probably want at least 1 case each for each happy path (numbers, strings, maybe one with missing data). I'm not sure we need to have an especially "nice" behavior for the mixed datatype case other than to not crash.

what tests are needed,

I looked to see how tests in this package vega-interpreter are written ( https://github.com/vega/vega/tree/main/packages/vega-interpreter/test ), and found that it appears to involve adding an additional JSON file each to to https://github.com/vega/vega/tree/main/packages/vega/test/specs-valid and https://github.com/vega/vega/tree/main/packages/vega/test/scenegraphs . Once we confirm that this expression does what you need in the preview deploy, we can copy that JSON over to the test suite.

@joelostblom
Copy link
Contributor Author

Sorry for the delay in getting back to you @hydrosquall . I was trying out the branch preview editor right now, and it seems like it doesn't recognize the sort function. Try replacing reverse with sort on line 15 in the following spec and you will see that it errors with Unrecognized function: sort. Open the Chart in the Vega Editor

@domoritz
Copy link
Member

I tried this spec btw. It definitely worked for me before but I also get Unrecognized function: sort right now.

{
  "$schema": "https://vega.github.io/schema/vega/v5.json",


  "signals": [
    {
      "name": "foo",
      "update": "sort([1,2,3])"
    }
  ]
}

@hydrosquall
Copy link
Member

hydrosquall commented Sep 19, 2024 via email

@joelostblom
Copy link
Contributor Author

From grepping the code for “unrecognized”, I think we might have to add
this new sort function to Vega-expression too.

Hmm, doesn't that sound odd given that other array functions such as reverse are not listed there?

Give that @domoritz mentioned the spec worked for him before, maybe there is something going on with the deploy preview?

@domoritz
Copy link
Member

I couldn't get it to work locally either yesterday. Need to dig a bit more what I did before

@hydrosquall
Copy link
Member

hydrosquall commented Sep 19, 2024 via email

@domoritz
Copy link
Member

Yes, but reverse works so I'm confused. Need to dig into it more.

@hydrosquall
Copy link
Member

hydrosquall commented Sep 20, 2024 via email

@domoritz
Copy link
Member

Something with the linking in the editor doesn't seem to work since the sort method is missing here. It should be okay to only change the code changed here so I don't think we need extra docs.

Screenshot 2024-09-20 at 14 40 59

@hydrosquall
Copy link
Member

hydrosquall commented Sep 21, 2024

I can't speak to why the relevant code is missing from that localhost screenshot (that does seem odd), but adding the method to codegen.js seems to have successfully added sort to the editor preview. (see the test link on #3973 ).

image

From revisiting the package descriptions, it looks like vega-interpreter is only used for the CSP compliant mode.

Since the editor doesn't run in CSP mode, it didn't call vega-interpreter and wouldn't know about this function.

vega-expression appears to depend on the contents of vega-functions, so I suppose it makes sense to add an entry there. If vega-expression shouldn't depend on functionContext, that's a separate issue to sort out.

(Apologies @joelostblom , as I steered you incorrectly when I suggested vega-interpreter as the package to change!)

I'm not sure about how extensively we need to test here (I still think having a realistic Vega spec that shows the application of this sort function is desirable), but I'll add something more isolated in the downstream PR for consideration in vega-functions for the unit test.

@domoritz
Copy link
Member

Moved to #3973

@domoritz domoritz closed this Sep 25, 2024
domoritz added a commit that referenced this pull request Sep 25, 2024
…3973)

## Motivation

This PR tests a
[fix](#3962 (comment))
for #3962 , with the goal of adding a
`sort` function to the default set of Vega functions in the expression
language.


## Changes

- Includes @domoritz and @joelostblom's doc and `vega-interpreter`
commits from #3962, rebased against
latest `main`
- Added + tests sort in `vega-functions`:
5e6190d#diff-328f6311e6ab7be84b1167d9ca245bea77f0ae6e078ffcd4b71bb7b2340e810b

## Testing

- This worked locally when using `yarn link vega`
- This is also working in the [Cloudflare
preview](https://cameron-yick-sort-functions.vega-628.pages.dev/#/url/vega/N4IgJAzgxgFgpgWwIYgFwhgF0wBwqgegIDc4BzJAOjIEtMYBXAI0poHsDp5kTykSArJQBWENgDsQAGhBMkUANZkATmwbiAJmhAB3GHTjSQOJBo01xZNAJk6aG+mgBMABhkRMATwA2h9FDhvbyMNJEwUVABtUHEkBD8QMQZlAIB9FyNiJG8GOAg0aJAIkABBIyZnAA4AXwBdaqkYuITQ8PSjJJSEzrSMmUxlJHEIADM2ZQQC0C8cBLGJhm8UGTgADxxlbTFlTAAKSIByEYOpAAIDpBPzzCuDpgPagEojJHz0TDzMEDq69xoyWLeN6FWLxbSeVIeOA4TLZXLOFwNJpg9Dwf5YIwMHCtBJyTQQEwBXYaNjICy7A6eA6PSi+Sz0M4ARjOLkoAkepwAVKcIVCYb8QMhlApgciEkKRUYZgkPqsvu4vL4CiBZV9ajIRqpJqhQK1inr2g0QHBxFA2Bo-KAsTi0KARjQgraQFkcrilopvjILdBlDQcJh2JIdYl-oDtAAdFWfVCnSOnADUp12NAgADVsvZiWEGAhIpGPh5I09TgB+U6tHN5qOFkC1U4xyOR+MV3P5z5F55G1ZOiCh7LaOwOGBGBCLL6oVkCI2eHtQbIJGcamiBLTvT6eoreUNOl3w-wmj6bI1yCCBCyW51w8X2DRK6r3gXQeeikCghdSzyzbQ4NgWeUgEkySDXUwn1UD2iXFdtALf9tnHAZciNQZLAvPkez7YJ0F5D5+SNEwzAsKwJzZAUkFWPIpkSOclSwoxxmXcRxxAXwRn-FR7DQEZslPfo6Bo6t-wALwsC1uwnH4gA)

Uses test spec JSON:

```
{
  "$schema": "https://vega.github.io/schema/vega/v5.json",
  "background": "white",
  "padding": 5,
  "width": 20,
  "style": "cell",
  "data": [
    {"name": "source_0", "values": [{"a": "A", "b": 28}]},
    {
      "name": "data_0",
      "source": "source_0",
      "transform": [
        {"type": "formula", "expr": "sort(['f', 'a', 't', 'b'])", "as": "test"}
      ]
    }
  ],
  "signals": [
    {"name": "y_step", "value": 20},
    {
      "name": "height",
      "update": "bandspace(domain('y').length, 1, 0.5) * y_step"
    }
  ],
  "marks": [
   
  ],
  "scales": [
    {
      "name": "y",
      "type": "point",
      "domain": {"data": "data_0", "field": "test", "sort": true},
      "range": {"step": {"signal": "y_step"}},
      "padding": 0.5
    }
  ],
  "axes": [
    {
      "scale": "y",
      "orient": "left",
      "grid": false,
      "title": "test",
      "zindex": 0
    }
  ]
}
```

---------

Co-authored-by: Joel Ostblom <joelostblom@users.noreply.github.com>
Co-authored-by: Dominik Moritz <domoritz@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants