SPARQL Query Results JSON format by dpetran · Pull Request #973 · fluree/db

dpetran · 2025-02-19T16:43:27Z

This adds the ability to produce query results in SPARQL 1.1 Query Results JSON Format.

https://github.com/fluree/core/issues/170

This introduces the :output query option. If it is not supplied it defaults to :fql, which is what we currently return. Supplying :sparql provides the output like this:

{
  "head" : {
    "vars" : [ "favNum", "handle" ]
  },
  "results" : {
    "bindings" : [ {
      "handle" : {
        "value" : "bbob",
        "type" : "literal"
      },
      "favNum" : {
        "value" : "23",
        "type" : "literal",
        "datatype" : "http://www.w3.org/2001/XMLSchema#integer"
      }
    }, {
      "handle" : {
        "value" : "jdoe",
        "type" : "literal"
      },
      "favNum" : {
        "value" : "42",
        "type" : "literal",
        "datatype" : "http://www.w3.org/2001/XMLSchema#integer"
      }
    }, {
      "handle" : {
        "value" : "jdoe",
        "type" : "literal"
      },
      "favNum" : {
        "value" : "99",
        "type" : "literal",
        "datatype" : "http://www.w3.org/2001/XMLSchema#integer"
      }
    } ]
  }
}

Note that it retains the datatype and language metadata about values, as well as distinguishes IRIs from strings. It's also a tabular format, so aggregates are "unnested" and flattened.

dpetran · 2025-02-19T16:45:15Z

There's still room for further cleanup, especially around the select/display abstraction, but I want to leave it as-is while I tackle CONSTRUCT support, which will introduce a new case to display results as json-ld. I consider this a working but still immature abstraction.

zonotope · 2025-02-25T18:09:00Z

src/clj/fluree/db/query/exec/select.cljc

  "Format a where-pattern match for presentation based on the match's datatype.
  Return an async channel that will eventually contain the formatted match."
-  (fn [match _compact]
+  (fn [match output-format _compact]


This new output-format binding exists only to provide runtime polymorphism. This implementation also litters this output-format argument throughout the call stack forcing already working existing code to be modified to support this new feature.

We should use the built in language facilities to support run time polymorphism instead of rolling our own. Either new implementations of existing protocols/multi-methods, or a new protocol or multimethod would be a better fit.

My current plan is to change the where/match into a record and implement a display method for it. I've got it work-in-progress for construct support, but didn't want to block Nexus and CAM work while I was still figuring out the proper abstraction.

dpetran · 2025-02-28T19:00:41Z

Ok, I've overhauled the approach here. Instead of checking the output type each time we format a result, we now just check during query parsing and add the correct implementation of format-value for the output type.

dpetran · 2025-03-10T15:53:00Z

Rebased on main

Instead of always inferring the datatype and throwing away lang, just use the result from the TypedResult if it's available.

Instead of checking what the output should be on each match, decide the `display` implementation at parse time and use that.

dpetran · 2025-03-12T15:26:59Z

Rebased on main.

zonotope · 2025-03-12T15:41:29Z

src/fluree/db/query/api.cljc

@@ -35,6 +35,7 @@
                        ;; ensure :max-fuel key is present
                        (-> (assoc opts :max-fuel max-fuel)
                            (merge opts override-opts)


Though not explicitly part of this pr, there is an extra opts on line 36. I think line 36 and 37 should be replaced with:

(-> opts (assoc :max-fuel max-fuel) (merge override-opts) ...

zonotope · 2025-03-12T15:45:37Z

src/fluree/db/query/api.cljc


+(defn wrap-sparql-results
+  [results]
+  (let [format (fn [results] {"head" {"vars" (vec (sort (keys (first results))))}


why isn't this anonymous function defined at the top level? it doesn't look like it closes over anything.

It was to keep the function definition local. This entire function is obsolete, in the CONSTRUCT PR (#985) I move all the wrapping into the collect-results phase of query execution.

I'll merge that PR into this one so the whole chain is visible.

zonotope · 2025-03-12T15:50:37Z

src/fluree/db/query/api.cljc

+          _       (log/debug "query-connection SPARQL fql: " fql "override-opts:" override-opts)
+          results (<? (query-connection-fql conn fql override-opts))]
+      (if (= (:output override-opts) :sparql)
+        (wrap-sparql-results results)


Is there any reason to disallow someone from setting the output format to sparql for an fql query?

No, and this is made possible in the CONSTRUCT PR by moving the wrapping into the collect-results phase of query execution.

zonotope · 2025-03-12T15:53:52Z

src/fluree/db/query/exec/group.cljc

+
+(defmethod display/fql ::grouping
+  [match compact]
+  ((display-aggregate display/fql) match compact))


I think this code would be easier to understand if these functions derived from apply display-aggregate were pre-defined at the top level.

zonotope · 2025-03-12T16:45:31Z

src/fluree/db/query/exec/select.cljc

-  (some-> match where/get-value vec))
-
 (defprotocol ValueSelector
+  :extend-via-metadata true


I think this namespace is convoluted by relying on this, and I don't see any benefit.

I think instead, there should be two additional namespaces: fluree.db.query.exec.select.fql, and fluree.db.query.exec.select.sparql for the different output formats, each with their own implementations of the ValueSelector protocol that don't reference each other.

Then, during parse time, you can read the output format once and decide which ValueSelector records to instantiate based on the select clause.

I chose :extend-via-metadata because the other protocols the records implement (ValueAdapter, SolutionModifier) aren't affected by the display and if I create separate Selector records for FQL vs SPARQL I'd have to move the common implementations to a third, common namespace or duplicate them, and on the balance it seemed that :extend-via-metadata was the simpler choice.

Ok. I'm worried about the performance implications of extending via metadata as well as the semantic implications of having opaque functions hanging off of records.

This pr has been held open for long enough so I'll let this through and we can address these concerns later if they prove to be a problem.

zonotope · 2025-03-12T16:51:26Z

src/fluree/db/query/exec/select/display.cljc

+            [fluree.db.query.exec.where :as where]
+            [fluree.db.util.json :as json]))
+
+(defmulti fql (fn [match compact] (where/get-datatype-iri match)))


I think this should also be split. Instead of this new namespace with these two multimethods, we should instead put the display/fql multimethod in fluree.db.query.exec.select.fql and the sparql multimethod in the sparql ns.

Essentially, we should take everything except for the protocol definitions from fluree.db.exec.select.fql and move them to a new namespace fluree.db.query.exec.select.fql, and then define a parallel fluree.db.query.exec.select.sparql namespace with the sparql functionality.

The official grammar only has it in all caps, but even the examples in the spec use lower case, so we may as well support it.

Also added compact iris to construct results, which required holding on to the original unparsed context.

Since solutions are processed concurrently there's no guarantee on the order blank node labels will be assigned.

Support for SPARQL CONSTRUCT clauses

zonotope · 2025-03-13T15:39:52Z

src/fluree/db/query/fql/parse.cljc

  variable is a string, symbol, or keyword whose name starts with '?'."
  [x]
-  (when (v/variable? x)
+  (when (or (v/variable? x)


I think it would be useful to define a more general function for detecting variables because you repeat this or call a lot. It might even be useful to just change the definition of the variable? function to include that of bnode-variable? if the current variable? function is never called alone.

We actually need to distinguish between the two types - in a transaction you can have bnodes and they are not variables, they are "fresh" identifiers that will be mapped to new, internal identifiers. So it's only inside of a query where pattern that we need these or statements.

Ok, so I think it makes sense to have a general function for detecting variables in queries to remove the or repetition here.

I see this branch is already merged, so we can address this later.

Here's a follow-up PR to address it:
#989

zonotope · 2025-03-13T16:12:57Z

src/fluree/db/query/exec/select.cljc

-  (solution-value
-    [_ _ solution]
-    solution))
+  (solution-value [_ _ solution] solution))


I think the argument list and function definitions should both be on new lines.

zonotope · 2025-03-13T16:14:09Z

src/fluree/db/query/exec/select.cljc

  ValueSelector
  (format-value
-    [_ _ _ _ _ _ error-ch solution]
+    [_ _ _ _ compact _ error-ch solution]


It doesn't look like compact is used.

zonotope · 2025-03-13T16:28:32Z

src/fluree/db/query/exec/select.cljc

-  (some-> match where/get-value vec))
-
 (defprotocol ValueSelector
+  :extend-via-metadata true


Ok. I'm worried about the performance implications of extending via metadata as well as the semantic implications of having opaque functions hanging off of records.

This pr has been held open for long enough so I'll let this through and we can address these concerns later if they prove to be a problem.

dpetran requested a review from a team February 19, 2025 16:51

zonotope reviewed Feb 25, 2025

View reviewed changes

dpetran force-pushed the feature/sparql-output branch from b15d51b to 511e200 Compare February 28, 2025 18:43

dpetran force-pushed the feature/sparql-output branch from f74b898 to 4b1fd82 Compare March 10, 2025 15:52

dpetran mentioned this pull request Mar 10, 2025

Feature/sparql protocol query fluree/server#117

Merged

dpetran added 15 commits March 12, 2025 10:24

add :output opt and ensure passage of opts to execution

906d8e7

add output-format parameter to select formatters

88ea774

handle nil solution keys from implicit grouping

ff3d99b

make sure to properly display value of AsSelector formatter

9d8ddcc

use lang and datatype from as-fn result

bc8194e

Instead of always inferring the datatype and throwing away lang, just use the result from the TypedResult if it's available.

return sparql json formatted results when directed

778f84f

provide sparql json formatting for scalar values

58e65d6

handle aggregated values

9b75e9d

short circuit format-xf creation

15de0d1

handle query-connection sparql results

062ac31

add opts to schema

e077469

move output polymorphic dispatch to parse stage

1b519b4

Instead of checking what the output should be on each match, decide the `display` implementation at parse time and use that.

consolidate to display ns

8bdfbef

remove output-format from format-value args

855a139

handle meta wrapper for sparql result format

1193d39

dpetran force-pushed the feature/sparql-output branch from 4b1fd82 to 1193d39 Compare March 12, 2025 15:26

dpetran mentioned this pull request Mar 12, 2025

Support for SPARQL CONSTRUCT clauses #985

Merged

zonotope reviewed Mar 12, 2025

View reviewed changes

dpetran added 3 commits March 12, 2025 12:02

remove extraneous opts merge

bfaaccf

define top-level aggregate display fns

e3dc66c

move output-specific functionality in select.<output> ns

0fe82ab

dpetran added 17 commits March 12, 2025 12:26

basic sparql CONSTRUCT parsing support

381aa58

support lower-case 'asc' and 'desc'

76cd0e6

The official grammar only has it in all caps, but even the examples in the spec use lower case, so we may as well support it.

add support for SPARQL CONSTRUCT WHERE

85b560a

add parser support for :construct clause

5cae6bc

use functions to access match map

88f5d58

basic single-pattern construct support

23d69e7

handle result wrapping in collect-results

21578fb

Also added compact iris to construct results, which required holding on to the original unparsed context.

support multi-pattern construct clause

967c190

handle multicardinal values in construct output

b985a24

handle bnode variables in construct statement

92ac8ea

comment out non-deterministic test for now

7acff68

Since solutions are processed concurrently there's no guarantee on the order blank node labels will be assigned.

add SPARQL CONSTRUCT tests

3fd56b2

add comments about unbound vars

1e51fa4

remove extraneous channel while collecting results

0324a64

basic sparql update parsing

1bb883a

move json-ld display fns to json-ld ns

6b03026

Merge pull request #985 from fluree/feature/construct

a484022

Support for SPARQL CONSTRUCT clauses

zonotope approved these changes Mar 13, 2025

View reviewed changes

fix whitespace and unused arg binding

b98f536

dpetran merged commit e345dd9 into main Mar 13, 2025
6 checks passed

dpetran deleted the feature/sparql-output branch March 13, 2025 16:50

Conversation

dpetran commented Feb 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dpetran commented Feb 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dpetran commented Feb 28, 2025

Uh oh!

dpetran commented Mar 10, 2025

Uh oh!

dpetran commented Mar 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dpetran commented Feb 19, 2025 •

edited

Loading