(Discussed with @petermattis)

We wish to embed the SQL tokenizer (not parser) in a 3rd party go code base for the CC management console, with minimal dependencies on the remainder of the CockroachDB code base.

For this we want to move the scanner to a new package with minimal go imports: pkg/sql/scanner.

The code in pkg/sql/parser/scan.go is already pretty well self-contained.

What remains to be done is to remove the imports forpkg/sql/lex and pkg/sql/sem/tree and stop relying on pkg/sql/parser/sql.go.

before we can create a new pkg/sql/scanner.

How to achieve this?

Regarding the pkg/sql/lex dependency [done]

The only use of it is for lex.GetKeywordID(). This function is a "leaf" function and does not need to remain in the lex package. It can be moved to lexbase or a similar self-contained package.

Regarding the pkg/sql/sem/tree dependency [done]

There are just two uses: tree.NewNumVal and tree.NewPlaceholder.

We cannot really move these functions to a different package to make them standalone, because the tree.NumVal and tree.NewPlaceholder are (currently) intimately linked to other things in sem/tree.

What we can do however is dependency injection: define two global function callbacks in the new pkg/sql/scanner package:

var NewNumValFn = func(constant.Value, string, bool) interface{} { return &fakeNum{}  }
var NewPlaceholderFn = func(string) interface{} { return &fakePlaceholder{} }

(where the fakeNum and fakePlaceholder types are suitable defined locally in scanner)

Then in the sql/parser package, define an init function:

func init() {
   scanner.NewNumValFn = func(a constant.Value, s string, b bool) interface{} { return tree.NewNumVal(a,s,b) }
   scanner.NewPlaceholderFn = func(s string) interface{} { return tree.NewPlaceholder(s) }
}

Regarding pkg/sql/parser/sql.go

This is a generated file based on sql.y. scan.go uses it to get the sqlSymType and sqlSymUnion types. pkg/sql/parser/sql.go has a lot of dependencies. We need to extract sqlSymType and sqlSymUnion to a separate package.

Epic CRDB-7217