package table
- Alphabetic
- By Inheritance
- table
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Type Members
- trait Aggregation[X] extends (Seq[X]) => X
- trait Analyzer extends AnyRef
Unsealed trait for polymorphic analysis of table sources.
Unsealed trait for polymorphic analysis of table sources. Implement this trait to add support for new sources (CSV, Parquet, etc.).
- abstract class BaseRow extends (String) => Try[String]
Abstract class to represent a (raw) row from a table.
- case class CellTransformation[X, Y](f: (X) => Y) extends Transformation[X, Y] with Product with Serializable
A case class representing a transformation for individual cell values in a table.
A case class representing a transformation for individual cell values in a table.
CellTransformationacts as a wrapper around a functionfthat maps values of typeXto values of typeY. It extends theTransformationtrait, which itself inherits from theFunction1trait, allowing this transformation to be used like a standard Scala function.- X
the input type of the function
f.- Y
the output type of the function
f.- f
the function that defines the transformation logic from type
Xto typeY.
- case class Column(clazz: String, optional: Boolean, maybeStatistics: Option[MaybeStatistics]) extends Product with Serializable
A representation of the analysis of a column.
A representation of the analysis of a column.
- clazz
a String denoting which class (maybe which variant of class) this column may be represented as.
- optional
if true then this column contains nulls (empty strings).
- maybeStatistics
an optional MaybeStatistics (eager or lazy); None if statistics are unavailable or deferred.
- case class ColumnStatistics(rows: Int, columns: Int, columnMap: Map[String, Column]) extends Product with Serializable
Case class to represent the analysis of a table.
Case class to represent the analysis of a table.
- rows
the number of rows.
- columns
the number of columns.
- columnMap
a map of column names to Column objects (the statistics of a column).
- trait ColumnStatisticsProvider extends AnyRef
Provider trait for computing column statistics from external sources (e.g., Parquet files).
Provider trait for computing column statistics from external sources (e.g., Parquet files). Implement this in modules that add support for new sources.
- case class Content[+Row](xs: ParIterable[Row]) extends Product with Serializable
Class to represent the rows of a
Table.Class to represent the rows of a
Table.At present, the rows are implemented as a
ParIterable. However, we might later change the internal representation; thusxsis private.In general, however, you should probably expect that storing rows inside Content will force an aggregation of all rows. There are alternatives (using InputProcessor) that allow you to process an
Iterator[Input]and yield anIterator[Row].- Row
the underlying Row type.
- xs
a ParIterable[Row].
- case class CsvAnalyzer(table: RawTable) extends Analyzer with Product with Serializable
Analyzer for CSV sources.
Analyzer for CSV sources. Operates on an in-memory RawTable and walks rows to compute statistics.
- table
the raw table to analyze.
- case class CsvAttributes(delimiter: String, quote: String) extends Product with Serializable
Case class for CSV attributes, especially for rendering as CSV.
Case class for CSV attributes, especially for rendering as CSV.
CONSIDER merging this with RowConfig.
- delimiter
the delimiter.
- quote
the quote character.
- trait DatasetAnalyzerFactory extends AnyRef
Marker trait for dataset analyzer factories.
Marker trait for dataset analyzer factories. Implemented by parquet module to enable Analysis.forParquetDataset.
- case class EagerStatistics(stats: Statistics) extends MaybeStatistics with Product with Serializable
Eager statistics, computed upfront (typically from Parquet metadata).
Eager statistics, computed upfront (typically from Parquet metadata).
- stats
the computed Statistics.
- trait HasKey[Row] extends AnyRef
This typeclass pertains to a table row type T that has one particular column which is considered the primary key (row-id, identifier, whatever).
This typeclass pertains to a table row type T that has one particular column which is considered the primary key (row-id, identifier, whatever).
- Row
the underlying type.
- case class HeadedTable[Row](content: Content[Row], header: Header) extends RenderableTable[Row] with Product with Serializable
Concrete case class implementing RenderableTable with a Header.
Concrete case class implementing RenderableTable with a Header. The unit and apply methods are such that rows is in fact an Array[Row] (??).
NOTE: the existence or not of a Header in a RenderableTable only affects how the table is rendered. The parsing of a table always has a header of some sort.
- Row
the underlying type of each Row
- content
the rows of the table, stored as an Array.
- header
the header.
- case class Header(xs: Seq[String], xss: Seq[Seq[String]]) extends Product with Serializable
Case class to represent a header.
Case class to represent a header.
- xs
the sequence of column names.
- xss
a sequence of sequence of String representing any additional header lines.
- case class Indexed[T](i: Int, t: T) extends Product with Serializable
A wrapper class to index a T.
A wrapper class to index a T.
- T
the underlying type.
- i
the index (ordinal value).
- t
the instance of T.
- case class LazyStatistics(compute: () => Option[Statistics]) extends MaybeStatistics with Product with Serializable
Lazy statistics, deferred as a thunk for later evaluation.
Lazy statistics, deferred as a thunk for later evaluation. Useful for expensive computations (e.g., column scan) that you may not need.
- compute
a function that computes the Statistics when called.
- sealed trait MaybeStatistics extends AnyRef
Sealed trait representing statistics that may be eager or lazy.
Sealed trait representing statistics that may be eager or lazy. Eager statistics are computed upfront (from Parquet metadata). Lazy statistics are deferred as a thunk (computed on demand).
- case class RawRow(ws: Seq[String], header: Header) extends BaseRow with Product with Serializable
- case class RawRowProjection(columns: Seq[Int]) extends Transformation[RawRow, RawRow] with Product with Serializable
A case class
RawRowProjectionthat transforms aRawRowinto anotherRawRowby selecting specific columns.A case class
RawRowProjectionthat transforms aRawRowinto anotherRawRowby selecting specific columns.This transformation operates based on the specified indices of the columns to be kept. It selects the raw string values (
ws) of the inputRawRowcorresponding to the provided indices and produces a newRawRowwith the filtered data, while retaining the header from the input.- columns
A sequence of integers representing column indices to retain in the transformation.
- case class RawRowTransformation(transformers: Map[Int, Transformation[String, String]]) extends Transformation[RawRow, RawRow] with Product with Serializable
Represents a transformation that can be applied to a
RawRow, transforming its individual string elements based on a collection of column-specific transformation functions.Represents a transformation that can be applied to a
RawRow, transforming its individual string elements based on a collection of column-specific transformation functions.This class is designed to process rows of data by applying transformations to specific columns, defined by their zero-based index. Any column index not explicitly configured with a transformation will default to an identity transformation, leaving the original value unchanged.
- transformers
A map where the key is the column index (0-based) and the value is a
Transformation[String, String]to be applied to the corresponding column's data. If a column index is not present in this map, an identity transformation is applied by default.
- type RawTable = Table[RawRow]
- case class RawTableAggregation(aggregators: Map[String, Transformation[String, String]]) extends Transformation[RawTable, RawTable] with Product with Serializable
This case class represents a transformation that performs column-wise aggregation on a
RawTable.This case class represents a transformation that performs column-wise aggregation on a
RawTable. Each aggregator specifies a mapping between a column name and its corresponding transformation function.- aggregators
A map where keys are column names (as
String) and values are transformations (subclasses ofTransformation) applied to the data in those columns. The transformation process is string-based (Transformation[String, String]), meaning that it takes string input from a column and produces a transformed string output.
- Annotations
- @unused()
- case class RawTableProjection(columns: Seq[String]) extends Transformation[RawTable, RawTable] with Product with Serializable
A transformation that projects a subset of columns from a
RawTablebased on the specified column names.A transformation that projects a subset of columns from a
RawTablebased on the specified column names.The
RawTableProjectionclass allows users to specify column names, and it produces a newRawTablecontaining only these columns. Internally, it determines the indices of the specified columns and applies the projection at the row level by retaining only the corresponding values.- columns
A sequence of column names to include in the resulting table.
- case class RawTableTransformation(transformers: Map[String, Transformation[String, String]]) extends Transformation[RawTable, RawTable] with Product with Serializable
A transformation class that applies a specific set of transformations to the rows of a
RawTablebased on matching column names.A transformation class that applies a specific set of transformations to the rows of a
RawTablebased on matching column names.- transformers
A map where the keys are column names (as
String) and the values areTransformation[String, String]instances, which define how to transform data in the corresponding columns. This class implements theTransformation[RawTable, RawTable]trait, meaning it takes aRawTableas input and produces a transformedRawTableas output. The transformation process is based on matching column names from thetransformersmap with the column names in the table's header. If a match is found, the corresponding transformation is applied to the relevant column in each row of the table.Example Usage
To use this class, first construct a
RawTableTransformationinstance by providing a map of column names and their corresponding transformations:val transformers: Map[String, Transformation[String, String]] = Map( "column1" -> CustomTransformation1, "column2" -> CustomTransformation2 ) val transformation = RawTableTransformation(transformers) // Apply the transformation to a RawTable object val transformedTable = transformation(rawTable)
Implementation Details
- The
applymethod retrieves the table'smaybeHeaderobject, which contains the column names. - For each column name in the
transformersmap, it fetches the column index usingHeader.getIndex. - The transformations are then compiled into a map associating column indices with their respective transformations.
- Finally, the table's rows are transformed using the
mapmethod, which applies aRawRowTransformationto each row.
- The
- abstract class RenderableTable[Row] extends Table[Row] with TableRenderable[Row]
CONSIDER eliminating this base class or, perhaps, rename it.
CONSIDER eliminating this base class or, perhaps, rename it.
- Row
the underlying type of each Row
- case class Row(ws: Seq[String], hdr: Header, index: Int) extends BaseRow with Product with Serializable
Case class to represent a (raw) row from a table.
Case class to represent a (raw) row from a table.
- ws
the (raw) Strings that make up the row.
- hdr
is the Header containing the column names.
- trait SingleFileAnalyzerFactory extends AnyRef
Marker trait for single-file analyzer factories.
Marker trait for single-file analyzer factories. Implemented by parquet module to enable Analysis.forParquet.
- case class Statistics(mu: Double, sigma: Double, min: Double, max: Double) extends Product with Serializable
Class to represent the statistics of a column.
Class to represent the statistics of a column.
- mu
the mean value.
- sigma
the standard deviation.
- min
the smallest value.
- max
the largest value.
- trait Table[Row] extends Iterable[Row]
A Table of Content.
A Table of Content.
- Row
the type of each row.
- case class TableException(w: String) extends Exception with Product with Serializable
Table Exception.
Table Exception.
- w
the message.
- abstract class TableJsonFormat[T] extends RootJsonFormat[Table[T]]
Abstract class TableJsonFormat[T] which extends RootJsonFormat of Table[T]
Abstract class TableJsonFormat[T] which extends RootJsonFormat of Table[T]
- T
for which there must be evidence of JsonFormat[T].
- trait Transformation[X, Y] extends (X) => Y
Trait which transforms an X into a Y.
Trait which transforms an X into a Y. Basically, this is just a sub-class of Function1.
- X
the input type.
- Y
the output type.
- case class UnheadedTable[Row](content: Content[Row]) extends RenderableTable[Row] with Product with Serializable
Concrete case class implementing RenderableTable without a Header.
Concrete case class implementing RenderableTable without a Header.
NOTE: the existence or not of a Header in a RenderableTable only affects how the table is rendered. The parsing of a table always has a header of some sort.
TESTME this: it is currently not used.
- Row
the underlying type of each Row
- content
the rows of the table.
- trait Validity[T] extends AnyRef
Deprecated Type Members
- trait OrderingWithZero[X] extends Ordering[X]
- Annotations
- @deprecated
- Deprecated
(Since version 1.7.0) Not used by TableParser: it will be removed in 2.0.0
- final class Sequence extends AnyVal
- Annotations
- @deprecated
- Deprecated
(Since version 1.7.0) Not used by TableParser: it will be removed in 2.0.0
- trait Sequential extends AnyRef
A trait that defines a contract for types that are associated with a
Sequence.A trait that defines a contract for types that are associated with a
Sequence.Types extending this trait must provide an implementation for the
sequencevalue, which represents an instance of theSequenceclass.NOTE none of this modules is currently used.
- Annotations
- @deprecated
- Deprecated
(Since version 1.7.0) Not used by TableParser: it will be removed in 2.0.0
Value Members
- object Analysis
The
Analysisobject provides factory methods to analyze tables from various sources.The
Analysisobject provides factory methods to analyze tables from various sources.Example usage:
// Analyze a CSV-sourced RawTable val stats: ColumnStatistics = Analysis(rawTable) // Analyze a CSV file by Path implicit val parser: RawTableParser = RawTableParser() val stats: ColumnStatistics = Analysis.forCsv(Path.of("data.csv")) // Analyze a Parquet file (requires parquet module in scope) val stats: ColumnStatistics = Analysis.forParquet[YellowTaxiTrip](Path.of("data.parquet"))
- object Column extends Serializable
- object ColumnStatistics extends Serializable
Companion object for ColumnStatistics.
Companion object for ColumnStatistics. Provides a deprecated alias for backward compatibility.
- object Content extends Serializable
The
Contentobject provides a factory method for creating instances of theContentclass.The
Contentobject provides a factory method for creating instances of theContentclass. This utility is designed to facilitate the creation ofContentobjects from standard iterable collections by converting them into parallel collections for potential parallel processing. - object CsvAttributes extends Serializable
Companion object for the
CsvAttributescase class.Companion object for the
CsvAttributescase class. Provides utility methods and implicit instances for working with CSV attributes. - object HeadedTable extends Serializable
Companion object for HeadedTable.
Companion object for HeadedTable. The apply methods provided arbitrarily use Vector as the collection for the rows of the table. CONSIDER using something else such as Array.
TESTME ?
- object Header extends Serializable
This companion object contains utility methods and constants for creating and manipulating instances of the Header class.
- object Indexed extends Serializable
Companion object for the
Indexedcase class, providing utility methods to create and work with indexed collections. - object Row extends Serializable
Companion object for the
Rowclass, providing utilities for working with rows, including CSV rendering and generation. - object Statistics extends Serializable
Companion object for the Statistics case class.
Companion object for the Statistics case class. Provides methods to create a
Statisticsinstance from a sequence of doubles. - object Table
- object UnheadedTable extends Serializable
Deprecated Value Members
- object Main
The Main object serves as the entry point of the application and provides functionality to parse and analyze a specified dataset of crime information.
The Main object serves as the entry point of the application and provides functionality to parse and analyze a specified dataset of crime information.
This object: - Loads a given CSV file containing crime data. - Sets up a source for reading the data. - Configures a
RawTableParserinstance to parse the table data with specified predicates. - Analyzes the parsed data and prints relevant results to the console. - Demonstrates how to handle the parsing process using Scala's Try-Success-Failure mechanism.The application expects the file to be properly located in the resources directory.
Members: -
crimeFile: Specifies the file name of the dataset (located in the resources directory). -sy: Sets up a data source using the provided file. -fraction: Defines a fraction value for sampling the data when parsing. -parser: Configures and prepares the table parser with a predicate for data sampling.Main functionalities: - Parses the source using the configured
RawTableParser. - On success: Performs analysis on the parsed data and logs the first 10 rows and summary information. - On failure: Throws an exception indicating the failure reason.Example Usage: To analyze a specific dataset, update the
crimeFilevariable with the correct resource file path before running the application.- Annotations
- @deprecated
- Deprecated
(Since version 1.7.0) Example program: it moves out of the library, into the tests, in 2.0.0
- object NonSequential
- Annotations
- @deprecated
- Deprecated
(Since version 1.7.0) Not used by TableParser: it will be removed in 2.0.0
- object OrderingWithZero extends Serializable
- Annotations
- @deprecated
- Deprecated
(Since version 1.7.0) Not used by TableParser: it will be removed in 2.0.0
- object Sequence
- Annotations
- @deprecated
- Deprecated
(Since version 1.7.0) Not used by TableParser: it will be removed in 2.0.0
- object Sequential
- Annotations
- @deprecated
- Deprecated
(Since version 1.7.0) Not used by TableParser: it will be removed in 2.0.0