Hello @GetToSet,

Thank you for your contribution. I won't be able to review your pull request until mid September. Please be patient.

Hello @GetToSet, I do not quite understand what feature is made possible with this pull request. Would you please explain what you need?

The main purpose of this PR is to make init(rawPattern: String, ...) of FTS5Pattern public rather than internal so that patterns can be constructed directly from raw pattern directly while FTS3Pattern supports this kind of initialization, but FTS5Patterns don't.

Thanks @GetToSet, but "because FTS3 can" is not a reason. FTS5 is not FTS3, and FTS5 pattern validation is not the same as FTS3 pattern validation (FTS5 cares about tables and columns when FTS3 does not). You have to find a better argument. For example, tell about what you can't possibly do, without any known workaround?

My use cases contain prefix queries like one + two + three*, these kind of queries may not be easily constructed by current initializers, since they transform queries into tokens and then reconstruct them into patterns to suit specific needs.

While using FTS3, I use the init(rawPattern: String, ...) to construct queries with prefix tokens, but after migrating to FTS5, I found I have no way to construct FTS5Patterns that contains prefix tokens.

My use cases contain prefix queries like one + two + three*, these kind of queries may not be easily constructed by current initializers, since they transform queries into tokens and then reconstruct them into patterns to suit specific needs.

While using FTS3, I use the init(rawPattern: String, ...) to construct queries with prefix tokens, but after migrating to FTS5, I found I have no way to construct FTS5Patterns that contains prefix tokens.

Or maybe I can enhance current tokenization process to correctly support prefix tokens rather than exposing rawPattern initializer?

Now we're talking :-)

It happens I had a similar conversation with one of my coworker yesterday, and I pushed on the development branch four new APIs:

• FTS3Pattern(matchingAllPrefixesIn:)
• FTS5Pattern(matchingAllPrefixesIn:)
• FTS3.tokenize(_:)
• FTS5.tokenize(_:)

The two new initializers accept user input and turn "foo bar" into the foo* bar* pattern. They are suitable for live updates of a list of results as a user is typing a query in a search bar. https://github.com/groue/GRDB.swift/blob/development/Documentation/FullTextSearch.md#fts5pattern

The two tokenize methods accept user input and turn "foo bar" into the ["foo", "bar"] array of tokens, and make it easier for applications to build raw tokens that are not built by ready-made initializers. https://github.com/groue/GRDB.swift/blob/development/Documentation/FullTextSearch.md#fts5-tokenizers

You still need a database connection in order to call Database.makeFTS5Pattern(rawPattern:forTable:), though. If this is too much an inconvenience, please explain why.

• FTS3.tokenize(_:)
• FTS5.tokenize(_:)

Those methods accept a tokenizer:

// Default tokenization using the `ascii` tokenizer:
try FTS5.tokenize("SQLite database")  // ["sqlite", "database"]
try FTS5.tokenize("Gustave Doré")     // ["gustave", "doré"])

// Tokenization with an explicit tokenizer:
try FTS5.tokenize("SQLite database", withTokenizer: .porter()) // ["sqlite", "databas"]
try FTS5.tokenize("Gustave Doré", withTokenizer: .unicode61()) // ["gustave", "dore"])

They currently do not accept a custom FTS5 tokenizer - I'll have to make it possible.

Thanks I'll checkout the latest development branch to have a try.

They currently do not accept a custom FTS5 tokenizer - I'll have to make it possible.

Done. See https://github.com/groue/GRDB.swift/blob/development/Documentation/FullTextSearch.md#fts5-tokenization

Note that a database connection is needed for FTS5 tokenization.

@GetToSet I'm closing this PR because it will not be merged. But you can provide feedback about the new APIs, of course. They are not merged into master yet, so we can discuss them.