Write column OID to the source#14636
Conversation
Add flag to denote that a column is dropped.
5cd0e82 to
ad3dbc1
Compare
8e8f391 to
4a8bf85
Compare
This is a pre-requisite for using column OID-s instead of names in the _source. Phase 1: Collect new columns and do a schema update. Don't create a Lucene document with source at this point. Phase 2: Index. Target references still might have unassigned OID since this is only a preparation step.
2f3d562 to
0622fcf
Compare
0622fcf to
15c0a06
Compare
|
|
||
| @Nullable | ||
| public final ValueIndexer<? super T> valueIndexer(RelationName table, | ||
| public final ValueIndexer<? super T> valueIndexer(DocTableInfo table, |
There was a problem hiding this comment.
The whole second commit was needed only to make ObjectIndexer aware of table.versionCreated. As you noted in the prev comment, we need to parse sourceKeyWriter instead.
I mean, I didn't want to inject version in the indexer like
if (valueIndexer instance of ObjectIndexer objectIndexer) { only ObjectIndexer cares about Version
objectIndexer.someNewMethodToInjectVersion(version)
}
as I thought it's not nice - but it looks shorter. I will drop second commit, and do direct injection instead. Will pass sourceKeyWriter as suggested above.
There was a problem hiding this comment.
ah ok, I recalled why I didn't want to do "direct injection" - in case of nested indexers, need to propagate it
for Object/Array/Dynamic indexers. I will think how I can propagate that shortly.
There was a problem hiding this comment.
I added soureKeyWriter to the indexValue and then it gets propagated to all nested (or newly created by DynamicIndexer) ObjectIndexers. Thanks for the suggestion!
I rebased the PR
cef00d6 to
cd5ee19
Compare
| addedColumnsByIdent.put(ref.column(), ref); | ||
| // Also add to columnsByIdent,so that getRef supplier doesn't get stale. | ||
| columnsByIdent.put(ref.column(), ref); | ||
| } |
There was a problem hiding this comment.
This is needed for the nested ObjectIndexer-s, after phase 1 table can get stale.
We update explicitly only targets via Indexer.updateTargets.
Before it was not important since we used column name anyway, now we are using getRef.apply on each ObjectIndexer.indexValue since we need to get OID now.
b37e849 to
6f94952
Compare
| Symbol[] returnValues) { | ||
| Iterator<Reference> refsIterator = table.iterator(); | ||
| while (refsIterator.hasNext()) { | ||
| Reference reference = refsIterator.next(); | ||
| columnsByIdent.put(reference.column(), reference); | ||
| } |
There was a problem hiding this comment.
Not using table.columns() since we need all columns, including nested ones.
6f94952 to
bab9e60
Compare
b94ee50 to
8c97f41
Compare
a992aca to
ce9e07c
Compare
Next step after #14635
TODO:
select _rawmust hide dropped columns