An abstract class InputTransformer that extends InputPreprocessor and defines methods for transforming input data into rows, including error handling and extracting valid data rows.
This class provides a scaffold for implementing data preprocessing pipelines by utilizing input-to-row transformations (mapTsToRows) and processing the results to extract valid rows while disregarding or handling errors (processTriedRows). It facilitates robust and reusable approaches in data processing workflows, making it easier to define reusable transformation logic for specific input data types and row structures.
Type parameters
Input
the type of input elements to be processed, for example, raw data records or strings.
Row
the type of rows to be produced after transformation, typically structured representations like case classes or tuples.
Processes an iterator of input elements, transforming them into an iterator of output rows. This method applies a two-step transformation: first, it maps the input elements into Try[Row] instances using the mapTsToRows method, then it processes these Try[Row] instances with processTriedRows to produce valid rows or handle errors.
Processes an iterator of input elements, transforming them into an iterator of output rows. This method applies a two-step transformation: first, it maps the input elements into Try[Row] instances using the mapTsToRows method, then it processes these Try[Row] instances with processTriedRows to produce valid rows or handle errors.
Value parameters
xs
An Iterator of input elements of type Input, representing the data to be transformed.
Attributes
Returns
An Iterator of output rows of type Row, resulting from the transformation process.
Processes an iterator of Try[Row] values, extracting successful rows while filtering out failed ones.
Processes an iterator of Try[Row] values, extracting successful rows while filtering out failed ones.
The method takes an iterator of Try[Row] instances, which represent rows that may have been successfully created or could contain errors. The successful rows are extracted and returned as an iterator, while any failures are effectively ignored.
Value parameters
rys
an iterator of Try[Row] elements where each element is a Try representing the outcome of a row transformation.
Attributes
Returns
an iterator of successfully processed rows (Row) extracted from the input iterator.
Processes an input iterator of type Input and maps its elements into rows of type Row while handling potential errors using a specified transformation function. The function can operate in two modes:
Processes an input iterator of type Input and maps its elements into rows of type Row while handling potential errors using a specified transformation function. The function can operate in two modes:
Multiline mode: input data can span multiple lines and will be joined using the provided Joinable type class.
Single-line mode: each input element is processed independently.
Value parameters
ev
An implicit instance of the Joinable[(Input, Int)] type class, which handles joining and validating input tuples during multiline mode.
g
A transformation function that takes a tuple (Input, Int), representing an input element and its index, and returns a Try[Row]. The function should encapsulate the logic for mapping input to a row, including error handling.
multiline
A Boolean flag indicating whether to process input in multiline mode. - true: Enables multiline processing using the provided Joinable type class. - false: Processes each input independently.
xs
The input iterator containing elements of type Input.
Attributes
Returns
An iterator of Try[Row], where each element represents either a successfully processed row or a failure. - In multiline mode, elements may span across multiple lines, joined using the logic provided by the Joinable type class. - In single-line mode, each input element is processed individually.