abstract class ParquetTableParser[R <: Product] extends TableBuilder[Table[R]]
A TableBuilder for Parquet sources (single files or dataset directories).
Reads a Parquet file or dataset directory into a Table[Row] using ParquetSchemaValidator, StandardParquetRowParser, and ParquetCellConverter.
Extends TableBuilder directly rather than TableParser, since the Parquet read path does not involve Iterator[String], header rows, or LineParser.
For flat schemas (no grouped sub-case-classes), only helper needs to be
provided. For grouped schemas, override converterMap and groupedHelpers
in addition to helper.
- R
the target case class type.
- Alphabetic
- By Inheritance
- ParquetTableParser
- TableBuilder
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new ParquetTableParser()(implicit arg0: ClassTag[R])
Type Members
- type Row = R
The row type.
The row type.
- Definition Classes
- ParquetTableParser → TableBuilder
Abstract Value Members
- abstract val helper: ColumnHelper[Row]
The ColumnHelper providing parameter-to-column name mapping for the top-level Row type.
The ColumnHelper providing parameter-to-column name mapping for the top-level Row type. Always required.
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def builder(rows: Iterator[Row], header: Header): Table[Row]
Build a Table[Row] from parsed rows and a Header derived from the Parquet schema.
Build a Table[Row] from parsed rows and a Header derived from the Parquet schema. Can be overridden if a different Table implementation is needed.
- rows
an iterator of Row objects representing the data rows.
- header
a Header object representing the table's column headers.
- returns
the constructed Table based on the input rows and header.
- Attributes
- protected
- Definition Classes
- ParquetTableParser → TableBuilder
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- val converterMap: Map[String, ParquetCellConverter[Any]]
Map from top-level field name to ParquetCellConverter for that field's sub-case-class type.
Map from top-level field name to ParquetCellConverter for that field's sub-case-class type. Only required for grouped schemas. Defaults to empty (flat schema behaviour).
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- val forgiving: Boolean
If true, individual row failures are logged but do not cause the overall parse to fail.
If true, individual row failures are logged but do not cause the overall parse to fail.
- Attributes
- protected
- Definition Classes
- TableBuilder
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- val groupedHelpers: Map[String, (ClassTag[_], ColumnHelper[_])]
Map from top-level field name to (ClassTag, ColumnHelper) for each grouped sub-case-class field.
Map from top-level field name to (ClassTag, ColumnHelper) for each grouped sub-case-class field. Used by ParquetSchemaValidator to recursively validate sub-class fields against the Parquet schema. Defaults to empty (flat schema behaviour).
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- def parseParquet(path: Path): Try[Table[Row]]
Parse a single Parquet file into a Table[Row].
Parse a single Parquet file into a Table[Row]. Fails if path is a directory; use parseParquetDataset for dataset directories.
Schema validation is performed before any rows are read. Fails fast with ParquetParserException on any schema mismatch or unsupported type.
- path
a java.nio.file.Path to a .parquet file (not a directory).
- returns
a Try[Table[Row]].
- def parseParquetDataset(path: Path): Try[Table[Row]]
Parse a Parquet dataset directory into a Table[Row].
Parse a Parquet dataset directory into a Table[Row]. Dataset directories contain multiple part-*.parquet files. Fails if path is not a directory.
Schema validation is performed before any rows are read. Fails fast with ParquetParserException on any schema mismatch or unsupported type.
- path
a java.nio.file.Path to a directory containing part files.
- returns
a Try[Table[Row]].
- val predicate: (Try[Row]) => Boolean
Predicate to filter rows.
Predicate to filter rows. Defaults to including all rows.
- Attributes
- protected
- Definition Classes
- TableBuilder
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)