The latest updates on your projects. Learn more about Vercel for Git ↗︎

I'm not sure I understand the purpose of this. Can you please clarify?

Hi,
sorry for the delay.
I'll try to describe the proposal in a better way.
Sorry for the wall.

Context

Now, Orama uses tree structures to store strings and numbers. In particular, it uses radix trees and avl trees to store respectively strings and numbers.

Problem

The time insertion is high when the developer trying to insert more than 100k of documents.
Some users opened issues or report that issues on slack.
In particular, the insertion inside avl trees took times.

We uses trees because that structures have some guarantees like:

• find all the documents mapped by a defined value
• find all documents greater or lesser that a value
• for radix tree, find all documents using a tolerance

All those features are great for some kind of data:

• strings where you want to find a substring
• filter for number equity
• filter for number using a comparator

All those searches (and others) are yet supported by Orama.

Anyway, if I think generally a search, I'm thinking a search bar + filters. Meanwhile the search bar completely leverages on radix tree for typo tolerance for instance, the filters are typically implemented using two kind of interfaces:

• the final user can specify a range of a value. The example is the price or the distance from the city center.
• the final user can specify the presence or the absence of a specific value inside the document. For instance, if an hotel has a services like a parking, or if a book has the Kindle version, or t-shirt size.

For the first use case, we can leverage on avl because we need the comparators.
For the second use case, we could leverage on the same implementation but it creates an overhead because we are still keeping the order also for fields where the order or the typo tolerance don't make any sense.

Solution

I'm proposing a different approach to not keep the order for fields that doesn't require it. Like the example above, the user still want to filter on it with an exact match and without keeping an order.
This solution offers a way to implement it using a Map<any, InternalDocumentId[]>.
Benefits:

• 30% improvement for number insertion
• less memory usage (not yet tested)
• reduce the dump size (not yet tested)

Const:

• add another type that cannot so easy to understand.

The implementation is still in progress because I would like to understand if this makes sense before finalizing it.

I see!
The proposal makes sense to me and an enum searching it definitely worth a try.
I'll review the code once it's finalized but I have a question here: do you think you can implement this using a new component (which means: do not use the usual radix+avl based component when performing operations) rather than modifying sparse part of Orama? This way we have a cleaner and "black-box like" implementation.

Hi!
What do you mean by a new component? Which component are you expecting to have?
It is always an index, right?

Exactly, a new Index component that can replace the default one. This use-case is an example of the ratio behind Orama's architecture.

What do you mean? I want to add this feature to the default index.
If this is not the case, could you give me an example of your thoughts?

I think that in order to keep Orama clean things should be separated. Nothing prevents you to implement the new feature as new index and then call within the default index.
But I still think this should be kept as separated component.