abstract class ParquetTableParser[R <: Product] extends TableBuilder[Table[R]]

A TableBuilder for Parquet sources (single files or dataset directories).

Reads a Parquet file or dataset directory into a Table[Row] using ParquetSchemaValidator, StandardParquetRowParser, and ParquetCellConverter.

Extends TableBuilder directly rather than TableParser, since the Parquet read path does not involve Iterator[String], header rows, or LineParser.

For flat schemas (no grouped sub-case-classes), only helper needs to be provided. For grouped schemas, override converterMap and groupedHelpers in addition to helper.

R

the target case class type.

Linear Supertypes
TableBuilder[Table[R]], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ParquetTableParser
  2. TableBuilder
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new ParquetTableParser()(implicit arg0: ClassTag[R])

Type Members

  1. type Row = R

    The row type.

    The row type.

    Definition Classes
    ParquetTableParser → TableBuilder

Abstract Value Members

  1. abstract val helper: ColumnHelper[Row]

    The ColumnHelper providing parameter-to-column name mapping for the top-level Row type.

    The ColumnHelper providing parameter-to-column name mapping for the top-level Row type. Always required.

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def builder(rows: Iterator[Row], header: Header): Table[Row]

    Build a Table[Row] from parsed rows and a Header derived from the Parquet schema.

    Build a Table[Row] from parsed rows and a Header derived from the Parquet schema. Can be overridden if a different Table implementation is needed.

    rows

    an iterator of Row objects representing the data rows.

    header

    a Header object representing the table's column headers.

    returns

    the constructed Table based on the input rows and header.

    Attributes
    protected
    Definition Classes
    ParquetTableParser → TableBuilder
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  7. val converterMap: Map[String, ParquetCellConverter[Any]]

    Map from top-level field name to ParquetCellConverter for that field's sub-case-class type.

    Map from top-level field name to ParquetCellConverter for that field's sub-case-class type. Only required for grouped schemas. Defaults to empty (flat schema behaviour).

  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  10. val forgiving: Boolean

    If true, individual row failures are logged but do not cause the overall parse to fail.

    If true, individual row failures are logged but do not cause the overall parse to fail.

    Attributes
    protected
    Definition Classes
    TableBuilder
  11. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. val groupedHelpers: Map[String, (ClassTag[_], ColumnHelper[_])]

    Map from top-level field name to (ClassTag, ColumnHelper) for each grouped sub-case-class field.

    Map from top-level field name to (ClassTag, ColumnHelper) for each grouped sub-case-class field. Used by ParquetSchemaValidator to recursively validate sub-class fields against the Parquet schema. Defaults to empty (flat schema behaviour).

  13. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  15. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. def parseParquet(path: Path): Try[Table[Row]]

    Parse a single Parquet file into a Table[Row].

    Parse a single Parquet file into a Table[Row]. Fails if path is a directory; use parseParquetDataset for dataset directories.

    Schema validation is performed before any rows are read. Fails fast with ParquetParserException on any schema mismatch or unsupported type.

    path

    a java.nio.file.Path to a .parquet file (not a directory).

    returns

    a Try[Table[Row]].

  19. def parseParquetDataset(path: Path): Try[Table[Row]]

    Parse a Parquet dataset directory into a Table[Row].

    Parse a Parquet dataset directory into a Table[Row]. Dataset directories contain multiple part-*.parquet files. Fails if path is not a directory.

    Schema validation is performed before any rows are read. Fails fast with ParquetParserException on any schema mismatch or unsupported type.

    path

    a java.nio.file.Path to a directory containing part files.

    returns

    a Try[Table[Row]].

  20. val predicate: (Try[Row]) => Boolean

    Predicate to filter rows.

    Predicate to filter rows. Defaults to including all rows.

    Attributes
    protected
    Definition Classes
    TableBuilder
  21. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  22. def toString(): String
    Definition Classes
    AnyRef → Any
  23. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  24. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  25. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from TableBuilder[Table[R]]

Inherited from AnyRef

Inherited from Any

Ungrouped