Although not a requirement this is related to #299.
There are two types of output that the framework should support:
-
The ingest framework should be able to support the ability to output to a plugin type's "intermediate format" as Avro binary to a Kafka topic.
-
To the final output as a Avro binary to a Kafka topic. This will likely require a data adapter to be able to translate the native output to an avro binary.
It seems that the framework should allow there to be at least one output medium defined, but allow for multiple to be defined. In other words, it could stage the intermediate format to HDFS and write it to a kafka topic, it could ingest into geowave, could output to multiple topics, etc.
How to intuitively express this through a command-line tool may be tricky, and in general multiplexing the output can be added complication so I propose this additional requirement go through a separate issue and that the initial issue of Kafka output be done as just handling a single input to a single output. But I do think the more complicated ideal end state would be worth considering as this issue is worked.
Although not a requirement this is related to #299.
There are two types of output that the framework should support:
The ingest framework should be able to support the ability to output to a plugin type's "intermediate format" as Avro binary to a Kafka topic.
To the final output as a Avro binary to a Kafka topic. This will likely require a data adapter to be able to translate the native output to an avro binary.
It seems that the framework should allow there to be at least one output medium defined, but allow for multiple to be defined. In other words, it could stage the intermediate format to HDFS and write it to a kafka topic, it could ingest into geowave, could output to multiple topics, etc.
How to intuitively express this through a command-line tool may be tricky, and in general multiplexing the output can be added complication so I propose this additional requirement go through a separate issue and that the initial issue of Kafka output be done as just handling a single input to a single output. But I do think the more complicated ideal end state would be worth considering as this issue is worked.