package parquet
- Alphabetic
- By Inheritance
- parquet
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Type Members
- case class ParquetAnalyzer[Row <: Product](path: Path) extends Analyzer with Product with Serializable
Analyzer for Parquet sources (single files or datasets).
Analyzer for Parquet sources (single files or datasets). Reads schema and metadata from the Parquet file or dataset directory without materializing all rows. Column types are inferred from the Parquet schema; statistics are lazy (None).
- Row
the target case class type (not used for schema extraction, but kept for consistency).
- path
the path to a .parquet file or dataset directory.
- trait ParquetCellConverter[T] extends AnyRef
Type class for converting a field value from a Parquet SimpleGroup directly to a Scala type T, without going through String.
Type class for converting a field value from a Parquet SimpleGroup directly to a Scala type T, without going through String.
- T
the target Scala type.
- case class ParquetDatasetAnalyzer[Row <: Product](path: Path) extends Analyzer with Product with Serializable
Analyzer for Parquet dataset directories.
Analyzer for Parquet dataset directories. Same behavior as ParquetAnalyzer but validates that path is a directory.
- Row
the target case class type.
- path
the path to a dataset directory containing part-*.parquet files.
- case class ParquetParserException(msg: String, cause: Option[Throwable] = None) extends Exception with Product with Serializable
Exception thrown for all Parquet-specific failures, including: - Schema mismatch between case class and Parquet file - Schema mismatch between part files in a dataset - Unsupported Parquet type with no registered mapping - Corrupt or unreadable Parquet file
Exception thrown for all Parquet-specific failures, including: - Schema mismatch between case class and Parquet file - Schema mismatch between part files in a dataset - Unsupported Parquet type with no registered mapping - Corrupt or unreadable Parquet file
- msg
the error message.
- cause
an optional underlying cause.
- trait ParquetRowParser[Row] extends AnyRef
Converts a Parquet SimpleGroup record directly into a typed Row, bypassing the String-based CellParser machinery of core.
Converts a Parquet SimpleGroup record directly into a typed Row, bypassing the String-based CellParser machinery of core.
- Row
the target case class type.
- abstract class ParquetTableParser[R <: Product] extends TableBuilder[Table[R]]
A TableBuilder for Parquet sources (single files or dataset directories).
A TableBuilder for Parquet sources (single files or dataset directories).
Reads a Parquet file or dataset directory into a Table[Row] using ParquetSchemaValidator, StandardParquetRowParser, and ParquetCellConverter.
Extends TableBuilder directly rather than TableParser, since the Parquet read path does not involve Iterator[String], header rows, or LineParser.
For flat schemas (no grouped sub-case-classes), only
helperneeds to be provided. For grouped schemas, overrideconverterMapandgroupedHelpersin addition tohelper.- R
the target case class type.
- class StandardParquetRowParser[Row <: Product] extends ParquetRowParser[Row]
Standard implementation of ParquetRowParser.
Standard implementation of ParquetRowParser. Uses reflection to identify case class parameters and ParquetCellConverter to extract typed values from each field.
Supports both flat primitive fields and grouped sub-case-class fields. A grouped field is any field whose type has an implicit ParquetCellConverter[T] provided in its companion object.
- Row
the target case class type, must be a Product (case class).
Value Members
- implicit val datasetFactory: DatasetAnalyzerFactory
Implicit dataset analyzer factory implementation.
Implicit dataset analyzer factory implementation. In scope when you
import com.phasmidsoftware.tableparser.parquet._ - implicit val parquetColumnStatisticsProvider: ColumnStatisticsProvider
Implicit ColumnStatisticsProvider for Parquet sources.
Implicit ColumnStatisticsProvider for Parquet sources. Allows Column.statisticsFrom(path, columnName) to work on Parquet files and datasets. In scope when you
import com.phasmidsoftware.tableparser.parquet._ - implicit val singleFileFactory: SingleFileAnalyzerFactory
Implicit single-file analyzer factory implementation.
Implicit single-file analyzer factory implementation. In scope when you
import com.phasmidsoftware.tableparser.parquet._ - object ParquetCellConverter
- object ParquetColumnStatisticsProvider extends ColumnStatisticsProvider
Provider for computing column statistics from Parquet files.
Provider for computing column statistics from Parquet files. Attempts to use Parquet metadata for eager statistics. Falls back to lazy column scan if metadata unavailable and useMetadataOnly=false.
- object ParquetParser
Entry point for parsing Parquet files and datasets into typed Tables.
Entry point for parsing Parquet files and datasets into typed Tables.
Typical usage for single files:
import com.phasmidsoftware.tableparser.parquet.ParquetParser._ implicit val helper: ColumnHelper[YellowTaxiTrip] = columnHelper(camelToSnakeCaseColumnNameMapper) val result: Try[Table[YellowTaxiTrip]] = ParquetParser.parse[YellowTaxiTrip](Path.of("data/yellow_2024.parquet"))
Typical usage for dataset directories:
val result: Try[Table[YellowTaxiTrip]] = ParquetParser.parseDataset[YellowTaxiTrip](Path.of("data/yellow_2024"))
- object ParquetSchemaValidator
Validates a Parquet MessageType schema against a target case class, prior to reading any rows.
Validates a Parquet MessageType schema against a target case class, prior to reading any rows.
Supports both flat case classes (all fields map directly to Parquet columns) and grouped case classes (some fields are sub-case-classes, each of which maps its own fields to Parquet columns via its own ColumnHelper).
- object ParquetTypeMapper
Maps Parquet primitive types to Scala types, taking into account both physical type and logical type annotation.
- object StandardParquetRowParser
Companion object providing a factory method to build a StandardParquetRowParser from the case class structure and schema.