Packages

package parquet

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. parquet
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. case class ParquetAnalyzer[Row <: Product](path: Path) extends Analyzer with Product with Serializable

    Analyzer for Parquet sources (single files or datasets).

    Analyzer for Parquet sources (single files or datasets). Reads schema and metadata from the Parquet file or dataset directory without materializing all rows. Column types are inferred from the Parquet schema; statistics are lazy (None).

    Row

    the target case class type (not used for schema extraction, but kept for consistency).

    path

    the path to a .parquet file or dataset directory.

  2. trait ParquetCellConverter[T] extends AnyRef

    Type class for converting a field value from a Parquet SimpleGroup directly to a Scala type T, without going through String.

    Type class for converting a field value from a Parquet SimpleGroup directly to a Scala type T, without going through String.

    T

    the target Scala type.

  3. case class ParquetDatasetAnalyzer[Row <: Product](path: Path) extends Analyzer with Product with Serializable

    Analyzer for Parquet dataset directories.

    Analyzer for Parquet dataset directories. Same behavior as ParquetAnalyzer but validates that path is a directory.

    Row

    the target case class type.

    path

    the path to a dataset directory containing part-*.parquet files.

  4. case class ParquetParserException(msg: String, cause: Option[Throwable] = None) extends Exception with Product with Serializable

    Exception thrown for all Parquet-specific failures, including: - Schema mismatch between case class and Parquet file - Schema mismatch between part files in a dataset - Unsupported Parquet type with no registered mapping - Corrupt or unreadable Parquet file

    Exception thrown for all Parquet-specific failures, including: - Schema mismatch between case class and Parquet file - Schema mismatch between part files in a dataset - Unsupported Parquet type with no registered mapping - Corrupt or unreadable Parquet file

    msg

    the error message.

    cause

    an optional underlying cause.

  5. trait ParquetRowParser[Row] extends AnyRef

    Converts a Parquet SimpleGroup record directly into a typed Row, bypassing the String-based CellParser machinery of core.

    Converts a Parquet SimpleGroup record directly into a typed Row, bypassing the String-based CellParser machinery of core.

    Row

    the target case class type.

  6. abstract class ParquetTableParser[R <: Product] extends TableBuilder[Table[R]]

    A TableBuilder for Parquet sources (single files or dataset directories).

    A TableBuilder for Parquet sources (single files or dataset directories).

    Reads a Parquet file or dataset directory into a Table[Row] using ParquetSchemaValidator, StandardParquetRowParser, and ParquetCellConverter.

    Extends TableBuilder directly rather than TableParser, since the Parquet read path does not involve Iterator[String], header rows, or LineParser.

    For flat schemas (no grouped sub-case-classes), only helper needs to be provided. For grouped schemas, override converterMap and groupedHelpers in addition to helper.

    R

    the target case class type.

  7. class StandardParquetRowParser[Row <: Product] extends ParquetRowParser[Row]

    Standard implementation of ParquetRowParser.

    Standard implementation of ParquetRowParser. Uses reflection to identify case class parameters and ParquetCellConverter to extract typed values from each field.

    Supports both flat primitive fields and grouped sub-case-class fields. A grouped field is any field whose type has an implicit ParquetCellConverter[T] provided in its companion object.

    Row

    the target case class type, must be a Product (case class).

Value Members

  1. implicit val datasetFactory: DatasetAnalyzerFactory

    Implicit dataset analyzer factory implementation.

    Implicit dataset analyzer factory implementation. In scope when you import com.phasmidsoftware.tableparser.parquet._

  2. implicit val parquetColumnStatisticsProvider: ColumnStatisticsProvider

    Implicit ColumnStatisticsProvider for Parquet sources.

    Implicit ColumnStatisticsProvider for Parquet sources. Allows Column.statisticsFrom(path, columnName) to work on Parquet files and datasets. In scope when you import com.phasmidsoftware.tableparser.parquet._

  3. implicit val singleFileFactory: SingleFileAnalyzerFactory

    Implicit single-file analyzer factory implementation.

    Implicit single-file analyzer factory implementation. In scope when you import com.phasmidsoftware.tableparser.parquet._

  4. object ParquetCellConverter
  5. object ParquetColumnStatisticsProvider extends ColumnStatisticsProvider

    Provider for computing column statistics from Parquet files.

    Provider for computing column statistics from Parquet files. Attempts to use Parquet metadata for eager statistics. Falls back to lazy column scan if metadata unavailable and useMetadataOnly=false.

  6. object ParquetParser

    Entry point for parsing Parquet files and datasets into typed Tables.

    Entry point for parsing Parquet files and datasets into typed Tables.

    Typical usage for single files:

    import com.phasmidsoftware.tableparser.parquet.ParquetParser._
    
    implicit val helper: ColumnHelper[YellowTaxiTrip] =
      columnHelper(camelToSnakeCaseColumnNameMapper)
    
    val result: Try[Table[YellowTaxiTrip]] =
      ParquetParser.parse[YellowTaxiTrip](Path.of("data/yellow_2024.parquet"))

    Typical usage for dataset directories:

    val result: Try[Table[YellowTaxiTrip]] =
      ParquetParser.parseDataset[YellowTaxiTrip](Path.of("data/yellow_2024"))
  7. object ParquetSchemaValidator

    Validates a Parquet MessageType schema against a target case class, prior to reading any rows.

    Validates a Parquet MessageType schema against a target case class, prior to reading any rows.

    Supports both flat case classes (all fields map directly to Parquet columns) and grouped case classes (some fields are sub-case-classes, each of which maps its own fields to Parquet columns via its own ColumnHelper).

  8. object ParquetTypeMapper

    Maps Parquet primitive types to Scala types, taking into account both physical type and logical type annotation.

  9. object StandardParquetRowParser

    Companion object providing a factory method to build a StandardParquetRowParser from the case class structure and schema.

Inherited from AnyRef

Inherited from Any

Ungrouped