TableParser
API
Generated with
TableParser/com.phasmidsoftware.tableparser/com.phasmidsoftware.tableparser.parquet

com.phasmidsoftware.tableparser.parquet

package com.phasmidsoftware.tableparser.parquet

Members list

Type members

Classlikes

case class ParquetAnalyzer[Row <: Product](path: Path) extends Analyzer

Analyzer for Parquet sources (single files or datasets). Reads schema and metadata from the Parquet file or dataset directory without materializing all rows. Column types are inferred from the Parquet schema; statistics are lazy (None).

Analyzer for Parquet sources (single files or datasets). Reads schema and metadata from the Parquet file or dataset directory without materializing all rows. Column types are inferred from the Parquet schema; statistics are lazy (None).

Type parameters

Row

the target case class type (not used for schema extraction, but kept for consistency).

Value parameters

path

the path to a .parquet file or dataset directory.

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
trait Analyzer
class Object
trait Matchable
class Any
Show all
object ParquetCellConverter

Attributes

Companion
trait
Supertypes
class Object
trait Matchable
class Any
Self type
trait ParquetCellConverter[T]

Type class for converting a field value from a Parquet SimpleGroup directly to a Scala type T, without going through String.

Type class for converting a field value from a Parquet SimpleGroup directly to a Scala type T, without going through String.

Type parameters

T

the target Scala type.

Attributes

Companion
object
Supertypes
class Object
trait Matchable
class Any
Known subtypes
case class ParquetColumn(field: String, columnName: String, maybeClass: Option[Class[_]], optional: Boolean)

A Parquet column which is read by a ParquetCellConverter.

A Parquet column which is read by a ParquetCellConverter.

Value parameters

columnName

the name of the column in the Parquet schema.

field

the name of the parameter which the column is read into.

maybeClass

the Scala type which the column must map to (see ParquetTypeMapper), if any.

optional

true if the parameter is an Option (so the column may be OPTIONAL).

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
object ParquetColumnStatisticsProvider extends ColumnStatisticsProvider

Provider for computing column statistics from Parquet files. Attempts to use Parquet metadata for eager statistics. Falls back to lazy column scan if metadata unavailable and useMetadataOnly=false.

Provider for computing column statistics from Parquet files. Attempts to use Parquet metadata for eager statistics. Falls back to lazy column scan if metadata unavailable and useMetadataOnly=false.

Attributes

Supertypes
class Object
trait Matchable
class Any
Self type
case class ParquetDatasetAnalyzer[Row <: Product](path: Path) extends Analyzer

Analyzer for Parquet dataset directories. Same behavior as ParquetAnalyzer but validates that path is a directory.

Analyzer for Parquet dataset directories. Same behavior as ParquetAnalyzer but validates that path is a directory.

Type parameters

Row

the target case class type.

Value parameters

path

the path to a dataset directory containing part-*.parquet files.

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
trait Analyzer
class Object
trait Matchable
class Any
Show all
object ParquetParser

Entry point for parsing Parquet files and datasets into typed Tables.

Entry point for parsing Parquet files and datasets into typed Tables.

Typical usage for single files:

 given ColumnHelper[YellowTaxiTrip] =
   columnHelper(camelToSnakeCaseColumnNameMapper)

 val result: Try[Table[YellowTaxiTrip]] =
   ParquetParser.parse[YellowTaxiTrip](Path.of("data/yellow_2024.parquet"))

Typical usage for dataset directories:

 val result: Try[Table[YellowTaxiTrip]] =
   ParquetParser.parseDataset[YellowTaxiTrip](Path.of("data/yellow_2024"))

Row's ParquetCellConverter is its own given, if it has one; otherwise it's derived (see ParquetCellConverter.derived), using the ColumnHelper of each case class.

Attributes

Supertypes
class Object
trait Matchable
class Any
Self type
case class ParquetParserException(msg: String, cause: Option[Throwable] = ...) extends Exception

Exception thrown for all Parquet-specific failures, including:

Exception thrown for all Parquet-specific failures, including:

  • Schema mismatch between case class and Parquet file
  • Schema mismatch between part files in a dataset
  • Unsupported Parquet type with no registered mapping
  • Corrupt or unreadable Parquet file

Value parameters

cause

an optional underlying cause.

msg

the error message.

Attributes

Supertypes
trait Product
trait Equals
class Exception
class Throwable
trait Serializable
class Object
trait Matchable
class Any
Show all
object ParquetSchemaValidator

Validates a Parquet MessageType schema against the columns which a ParquetCellConverter reads, prior to reading any rows.

Validates a Parquet MessageType schema against the columns which a ParquetCellConverter reads, prior to reading any rows.

The columns come from the converter (see ParquetCellConverter.columns), so this supports both flat case classes (all fields map directly to Parquet columns) and grouped case classes (some fields are sub-case-classes, each of which maps its own fields to Parquet columns via its own ColumnHelper).

Attributes

Supertypes
class Object
trait Matchable
class Any
Self type
class ParquetTableParser[R](using converter: ParquetCellConverter[R]) extends TableBuilder[Table[R]]

A TableBuilder for Parquet sources (single files or dataset directories).

A TableBuilder for Parquet sources (single files or dataset directories).

Reads a Parquet file or dataset directory into a Table[Row] using ParquetSchemaValidator and the ParquetCellConverter of Row.

Extends TableBuilder directly rather than TableParser, since the Parquet read path does not involve Iterator[String], header rows, or LineParser.

The converter is normally derived (see ParquetCellConverter.derived), for flat and grouped case classes alike: each case class needs only its ColumnHelper.

Type parameters

R

the target case class type.

Value parameters

converter

the ParquetCellConverter of R.

Attributes

Supertypes
trait TableBuilder[Table[R]]
class Object
trait Matchable
class Any
object ParquetTypeMapper

Maps Parquet primitive types to Scala types, taking into account both physical type and logical type annotation.

Maps Parquet primitive types to Scala types, taking into account both physical type and logical type annotation.

Attributes

Supertypes
class Object
trait Matchable
class Any
Self type
class ProductParquetCellConverter[T](fieldNames: List[String], converters: List[ParquetCellConverter[_]], construct: Seq[Any] => T)(using helper: ColumnHelper[T]) extends ParquetCellConverter[T]

The ParquetCellConverter of a case class T, as derived by ParquetCellConverter.derived.

The ParquetCellConverter of a case class T, as derived by ParquetCellConverter.derived.

Type parameters

T

the type of the result.

Value parameters

construct

a function to construct a T from the values of its fields.

converters

the ParquetCellConverters of the fields of T, in the same order.

fieldNames

the names of the fields of T, in declaration order.

helper

the ColumnHelper[T] which maps field names to column names.

Attributes

Supertypes
class Object
trait Matchable
class Any

Givens

Givens

given datasetFactory: DatasetAnalyzerFactory

Implicit dataset analyzer factory implementation. In scope when you import com.phasmidsoftware.tableparser.parquet._

Implicit dataset analyzer factory implementation. In scope when you import com.phasmidsoftware.tableparser.parquet._

Attributes

given parquetColumnStatisticsProvider: ColumnStatisticsProvider

Implicit ColumnStatisticsProvider for Parquet sources. Allows Column.statisticsFrom(path, columnName) to work on Parquet files and datasets. In scope when you import com.phasmidsoftware.tableparser.parquet._

Implicit ColumnStatisticsProvider for Parquet sources. Allows Column.statisticsFrom(path, columnName) to work on Parquet files and datasets. In scope when you import com.phasmidsoftware.tableparser.parquet._

Attributes

given singleFileFactory: SingleFileAnalyzerFactory

Implicit single-file analyzer factory implementation. In scope when you import com.phasmidsoftware.tableparser.parquet._

Implicit single-file analyzer factory implementation. In scope when you import com.phasmidsoftware.tableparser.parquet._

Attributes

Generated with