ParquetColumnStatisticsProvider

com.phasmidsoftware.tableparser.parquet.ParquetColumnStatisticsProvider

Provider for computing column statistics from Parquet files. Attempts to use Parquet metadata for eager statistics. Falls back to lazy column scan if metadata unavailable and useMetadataOnly=false.

Attributes

Graph
Supertypes
class Object
trait Matchable
class Any
Self type

Members list

Value members

Concrete methods

def computeStatistics(path: Path, columnName: String, useMetadataOnly: Boolean = ...): Option[Column]

Compute statistics for a single numeric column in a Parquet file. Prefers Parquet metadata (eager, no row scan). Falls back to lazy column materialization if metadata unavailable.

Compute statistics for a single numeric column in a Parquet file. Prefers Parquet metadata (eager, no row scan). Falls back to lazy column materialization if metadata unavailable.

Value parameters

columnName

the name of the column to analyze.

path

the path to a .parquet file or dataset directory.

useMetadataOnly

if true, return None if metadata unavailable; if false, return lazy thunk.

Attributes

Returns

an optional Column with eager or lazy statistics.