Per our conversation yesterday (and another ticket) we will maintain association of index to adapter. We discussed that this can be accomplished using stats, but this may not be sufficient.
We also discussed the idea that each index associated with an adapter has the same data set, indexed differently. This moves us towards the CBO concept and is consistent with data stores, in general. Of course, each index duplicates data. We may be able to add additional meta-data to the index, describing which data is present in the index. If the Query interface describes the return attributes, then a hash merges may be required to fulfill a request if an index only contains partial information.

How does the interface change with these concepts. Here are some thoughts:

Upon ingest, remove 'Index' from the method parameters. Instead, the ingest should consult meta-data. If no matching index is found associated with the data adapter, then error. This statement assumes index/adapter associations are available. if statistics are used, then statistics are essentially uninitialized (supported now).

If new association is added after data has been added to index for a given adapter (easily determined with statistics or other data store specific means), then label index/adapter association as invalid UNTIL the statistics match that of other index/adapter associations with the same adapter (counts specifically). This provides some assurance that the indices are consistent over the full data set associated one adapter.

The query interface need not change in look. However, a Query request without a specified index implies that the data store make some guess on the appropriate index to use. Here is where the COB will fit in. The CBO can be abstracted and pluggable. This may push some of the intelligence in the FeatureDataAdapter down into the data store. The intent is to use only one index or a mix if indices do not have all attributes duplicated, requiring some joins, hash or other.

Thinking about the interface changes for Query, I mentioned It would be nice to reduce the number of variants of both ingest and query. The issue I see is the proliferation of new methods each time some new feature is added. The intent of the current approach is to allow the addition of new methods when new parameters are added without forcing code changes across all uses. This desirable characteristic should be maintained.

One thought I had is to create something like Analytics (borrowing from the enum idea applied in the hadoop environment), a property management concept where parameters are all considered optional. The mandatory parameters can be excluded (e.g. DataAdapter and Items for ingest).
For example, ingest can be represented as only two methods:

ingest(WritableDataAdapter, T, Params) and
ingest(WritableDataAdapter, Iterator, Params)

The approach in taken analytics does perform runtime type checking. It does lose compile type checking of verifying that a parameter is appropriately associated with a type (e.g LIMIT is an integer).

One improvement is to create an interface
public interface DataStoreParam
{

}
Each parameter extends this interface
public class Limit implements
DataStoreParam
{
final int limit;

public Limit(
        int limit ) {
    super();
    this.limit = limit;
}

protected int getLimit() {
    return limit;
}

}

This achieves the type checking. The only thing lost is preventing a parameter that is not used by the interface. No harm done, as it is ignored; a be runtime check can occur flagging unrecognized parameters with a warning message. Consistently using this concept by a client could allow the client to compose parameter set across ingest and queries, ignoring the warnings, reusing objects and forcing consistent in client code.

In the end, I do feel that all combinations of optional parameters are permissible, allowing the user to choose from many different combinations. This does move away from -1 and null as indicators of 'no used'. At minimum, we should consider using Optional (Java 8).

Support a pluggable DataStore #238

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions