A trait for preprocessing input data elements into rows, representing an early stage in data transformation pipelines. The InputPreprocessor trait defines a contract for mapping input data of type Input into rows of type Row, encapsulated in Try to handle processing outcomes.
This abstraction is intended to facilitate robust and reusable data-parsing workflows, allowing the encapsulation of individual transformations or mappings with error-handling functionality. Implementations of this trait can provide specific strategies or rules for transforming input into rows, effectively bridging raw input data and downstream processing.
Type parameters
Input
the type of input elements to be processed, e.g., raw data points or records.
Row
the type of rows to be produced by the mapping, e.g., structured representations of the input (such as case classes or key-value mappings).
Processes an input iterator of type Input and maps its elements into rows of type Row while handling potential errors using a specified transformation function. The function can operate in two modes:
Processes an input iterator of type Input and maps its elements into rows of type Row while handling potential errors using a specified transformation function. The function can operate in two modes:
Multiline mode: input data can span multiple lines and will be joined using the provided Joinable type class.
Single-line mode: each input element is processed independently.
Value parameters
ev
An implicit instance of the Joinable[(Input, Int)] type class, which handles joining and validating input tuples during multiline mode.
g
A transformation function that takes a tuple (Input, Int), representing an input element and its index, and returns a Try[Row]. The function should encapsulate the logic for mapping input to a row, including error handling.
multiline
A Boolean flag indicating whether to process input in multiline mode. - true: Enables multiline processing using the provided Joinable type class. - false: Processes each input independently.
xs
The input iterator containing elements of type Input.
Attributes
Returns
An iterator of Try[Row], where each element represents either a successfully processed row or a failure. - In multiline mode, elements may span across multiple lines, joined using the logic provided by the Joinable type class. - In single-line mode, each input element is processed individually.