Background

Custom record header parsers have several limitations:

• They can only be used to parse records that have record headers at the beginning of the record
• They don't account for record footers
• Thay cannot be used to parse files having record separators instead of headers (e.g. text files where records are separated by LF)
• They cannot be used to parse files for which record length depends on the record number (See What is the current state of BDW+RDW? #336 for files having RDW+BDW)

A new way of handling variable-length records has been introduced to parse ASCII variable line length files and binary files with variable length OCCURS that don't have RDWs.

The interface of a raw record extractor is very simple. It is an iterator of record bytes in a file.

trait RawRecordExtractor extends Iterator[Array[Byte]]

The array of bytes for each record is going to be parsed according to the copybook. The record extractor has the freedom to parse records however it wants.

Feature

Raw record extractors can be adapted to make record extractor customizable, which fixed the issues 1-4.

For exampe a custom raw record parser can look like this,

class CustomRawRecordExtractor(startingRecordIndex: Int, data: SimpleStream, copybook: Copybook) extends RawRecordExtractor

The record index is needed since after a sparse index is generated the input file will be read in parallel staring from different offsets.