package table

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. table
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. trait Aggregation[X] extends (Seq[X]) => X
  2. trait Analyzer extends AnyRef

    Unsealed trait for polymorphic analysis of table sources.

    Unsealed trait for polymorphic analysis of table sources. Implement this trait to add support for new sources (CSV, Parquet, etc.).

  3. abstract class BaseRow extends (String) => Try[String]

    Abstract class to represent a (raw) row from a table.

  4. case class CellTransformation[X, Y](f: (X) => Y) extends Transformation[X, Y] with Product with Serializable

    A case class representing a transformation for individual cell values in a table.

    A case class representing a transformation for individual cell values in a table.

    CellTransformation acts as a wrapper around a function f that maps values of type X to values of type Y. It extends the Transformation trait, which itself inherits from the Function1 trait, allowing this transformation to be used like a standard Scala function.

    X

    the input type of the function f.

    Y

    the output type of the function f.

    f

    the function that defines the transformation logic from type X to type Y.

  5. case class Column(clazz: String, optional: Boolean, maybeStatistics: Option[MaybeStatistics]) extends Product with Serializable

    A representation of the analysis of a column.

    A representation of the analysis of a column.

    clazz

    a String denoting which class (maybe which variant of class) this column may be represented as.

    optional

    if true then this column contains nulls (empty strings).

    maybeStatistics

    an optional MaybeStatistics (eager or lazy); None if statistics are unavailable or deferred.

  6. case class ColumnStatistics(rows: Int, columns: Int, columnMap: Map[String, Column]) extends Product with Serializable

    Case class to represent the analysis of a table.

    Case class to represent the analysis of a table.

    rows

    the number of rows.

    columns

    the number of columns.

    columnMap

    a map of column names to Column objects (the statistics of a column).

  7. trait ColumnStatisticsProvider extends AnyRef

    Provider trait for computing column statistics from external sources (e.g., Parquet files).

    Provider trait for computing column statistics from external sources (e.g., Parquet files). Implement this in modules that add support for new sources.

  8. case class Content[+Row](xs: ParIterable[Row]) extends Product with Serializable

    Class to represent the rows of a Table.

    Class to represent the rows of a Table.

    At present, the rows are implemented as a ParIterable. However, we might later change the internal representation; thus xs is private.

    In general, however, you should probably expect that storing rows inside Content will force an aggregation of all rows. There are alternatives (using InputProcessor) that allow you to process an Iterator[Input] and yield an Iterator[Row].

    Row

    the underlying Row type.

    xs

    a ParIterable[Row].

  9. case class CsvAnalyzer(table: RawTable) extends Analyzer with Product with Serializable

    Analyzer for CSV sources.

    Analyzer for CSV sources. Operates on an in-memory RawTable and walks rows to compute statistics.

    table

    the raw table to analyze.

  10. case class CsvAttributes(delimiter: String, quote: String) extends Product with Serializable

    Case class for CSV attributes, especially for rendering as CSV.

    Case class for CSV attributes, especially for rendering as CSV.

    CONSIDER merging this with RowConfig.

    delimiter

    the delimiter.

    quote

    the quote character.

  11. trait DatasetAnalyzerFactory extends AnyRef

    Marker trait for dataset analyzer factories.

    Marker trait for dataset analyzer factories. Implemented by parquet module to enable Analysis.forParquetDataset.

  12. case class EagerStatistics(stats: Statistics) extends MaybeStatistics with Product with Serializable

    Eager statistics, computed upfront (typically from Parquet metadata).

    Eager statistics, computed upfront (typically from Parquet metadata).

    stats

    the computed Statistics.

  13. trait HasKey[Row] extends AnyRef

    This typeclass pertains to a table row type T that has one particular column which is considered the primary key (row-id, identifier, whatever).

    This typeclass pertains to a table row type T that has one particular column which is considered the primary key (row-id, identifier, whatever).

    Row

    the underlying type.

  14. case class HeadedTable[Row](content: Content[Row], header: Header) extends RenderableTable[Row] with Product with Serializable

    Concrete case class implementing RenderableTable with a Header.

    Concrete case class implementing RenderableTable with a Header. The unit and apply methods are such that rows is in fact an Array[Row] (??).

    NOTE: the existence or not of a Header in a RenderableTable only affects how the table is rendered. The parsing of a table always has a header of some sort.

    Row

    the underlying type of each Row

    content

    the rows of the table, stored as an Array.

    header

    the header.

  15. case class Header(xs: Seq[String], xss: Seq[Seq[String]]) extends Product with Serializable

    Case class to represent a header.

    Case class to represent a header.

    xs

    the sequence of column names.

    xss

    a sequence of sequence of String representing any additional header lines.

  16. case class Indexed[T](i: Int, t: T) extends Product with Serializable

    A wrapper class to index a T.

    A wrapper class to index a T.

    T

    the underlying type.

    i

    the index (ordinal value).

    t

    the instance of T.

  17. case class LazyStatistics(compute: () => Option[Statistics]) extends MaybeStatistics with Product with Serializable

    Lazy statistics, deferred as a thunk for later evaluation.

    Lazy statistics, deferred as a thunk for later evaluation. Useful for expensive computations (e.g., column scan) that you may not need.

    compute

    a function that computes the Statistics when called.

  18. sealed trait MaybeStatistics extends AnyRef

    Sealed trait representing statistics that may be eager or lazy.

    Sealed trait representing statistics that may be eager or lazy. Eager statistics are computed upfront (from Parquet metadata). Lazy statistics are deferred as a thunk (computed on demand).

  19. case class RawRow(ws: Seq[String], header: Header) extends BaseRow with Product with Serializable
  20. case class RawRowProjection(columns: Seq[Int]) extends Transformation[RawRow, RawRow] with Product with Serializable

    A case class RawRowProjection that transforms a RawRow into another RawRow by selecting specific columns.

    A case class RawRowProjection that transforms a RawRow into another RawRow by selecting specific columns.

    This transformation operates based on the specified indices of the columns to be kept. It selects the raw string values (ws) of the input RawRow corresponding to the provided indices and produces a new RawRow with the filtered data, while retaining the header from the input.

    columns

    A sequence of integers representing column indices to retain in the transformation.

  21. case class RawRowTransformation(transformers: Map[Int, Transformation[String, String]]) extends Transformation[RawRow, RawRow] with Product with Serializable

    Represents a transformation that can be applied to a RawRow, transforming its individual string elements based on a collection of column-specific transformation functions.

    Represents a transformation that can be applied to a RawRow, transforming its individual string elements based on a collection of column-specific transformation functions.

    This class is designed to process rows of data by applying transformations to specific columns, defined by their zero-based index. Any column index not explicitly configured with a transformation will default to an identity transformation, leaving the original value unchanged.

    transformers

    A map where the key is the column index (0-based) and the value is a Transformation[String, String] to be applied to the corresponding column's data. If a column index is not present in this map, an identity transformation is applied by default.

  22. type RawTable = Table[RawRow]
  23. case class RawTableAggregation(aggregators: Map[String, Transformation[String, String]]) extends Transformation[RawTable, RawTable] with Product with Serializable

    This case class represents a transformation that performs column-wise aggregation on a RawTable.

    This case class represents a transformation that performs column-wise aggregation on a RawTable. Each aggregator specifies a mapping between a column name and its corresponding transformation function.

    aggregators

    A map where keys are column names (as String) and values are transformations (subclasses of Transformation) applied to the data in those columns. The transformation process is string-based (Transformation[String, String]), meaning that it takes string input from a column and produces a transformed string output.

    Annotations
    @unused()
  24. case class RawTableProjection(columns: Seq[String]) extends Transformation[RawTable, RawTable] with Product with Serializable

    A transformation that projects a subset of columns from a RawTable based on the specified column names.

    A transformation that projects a subset of columns from a RawTable based on the specified column names.

    The RawTableProjection class allows users to specify column names, and it produces a new RawTable containing only these columns. Internally, it determines the indices of the specified columns and applies the projection at the row level by retaining only the corresponding values.

    columns

    A sequence of column names to include in the resulting table.

  25. case class RawTableTransformation(transformers: Map[String, Transformation[String, String]]) extends Transformation[RawTable, RawTable] with Product with Serializable

    A transformation class that applies a specific set of transformations to the rows of a RawTable based on matching column names.

    A transformation class that applies a specific set of transformations to the rows of a RawTable based on matching column names.

    transformers

    A map where the keys are column names (as String) and the values are Transformation[String, String] instances, which define how to transform data in the corresponding columns. This class implements the Transformation[RawTable, RawTable] trait, meaning it takes a RawTable as input and produces a transformed RawTable as output. The transformation process is based on matching column names from the transformers map with the column names in the table's header. If a match is found, the corresponding transformation is applied to the relevant column in each row of the table.

    Example Usage

    To use this class, first construct a RawTableTransformation instance by providing a map of column names and their corresponding transformations:

    val transformers: Map[String, Transformation[String, String]] = Map(
      "column1" -> CustomTransformation1,
      "column2" -> CustomTransformation2
    )
    val transformation = RawTableTransformation(transformers)
    
    // Apply the transformation to a RawTable object
    val transformedTable = transformation(rawTable)

    Implementation Details

    • The apply method retrieves the table's maybeHeader object, which contains the column names.
    • For each column name in the transformers map, it fetches the column index using Header.getIndex.
    • The transformations are then compiled into a map associating column indices with their respective transformations.
    • Finally, the table's rows are transformed using the map method, which applies a RawRowTransformation to each row.
    See also

    Transformation

    RawTable

    Header.getIndex

  26. abstract class RenderableTable[Row] extends Table[Row] with TableRenderable[Row]

    CONSIDER eliminating this base class or, perhaps, rename it.

    CONSIDER eliminating this base class or, perhaps, rename it.

    Row

    the underlying type of each Row

  27. case class Row(ws: Seq[String], hdr: Header, index: Int) extends BaseRow with Product with Serializable

    Case class to represent a (raw) row from a table.

    Case class to represent a (raw) row from a table.

    ws

    the (raw) Strings that make up the row.

    hdr

    is the Header containing the column names.

  28. trait SingleFileAnalyzerFactory extends AnyRef

    Marker trait for single-file analyzer factories.

    Marker trait for single-file analyzer factories. Implemented by parquet module to enable Analysis.forParquet.

  29. case class Statistics(mu: Double, sigma: Double, min: Double, max: Double) extends Product with Serializable

    Class to represent the statistics of a column.

    Class to represent the statistics of a column.

    mu

    the mean value.

    sigma

    the standard deviation.

    min

    the smallest value.

    max

    the largest value.

  30. trait Table[Row] extends Iterable[Row]

    A Table of Content.

    A Table of Content.

    Row

    the type of each row.

  31. case class TableException(w: String) extends Exception with Product with Serializable

    Table Exception.

    Table Exception.

    w

    the message.

  32. abstract class TableJsonFormat[T] extends RootJsonFormat[Table[T]]

    Abstract class TableJsonFormat[T] which extends RootJsonFormat of Table[T]

    Abstract class TableJsonFormat[T] which extends RootJsonFormat of Table[T]

    T

    for which there must be evidence of JsonFormat[T].

  33. trait Transformation[X, Y] extends (X) => Y

    Trait which transforms an X into a Y.

    Trait which transforms an X into a Y. Basically, this is just a sub-class of Function1.

    X

    the input type.

    Y

    the output type.

  34. case class UnheadedTable[Row](content: Content[Row]) extends RenderableTable[Row] with Product with Serializable

    Concrete case class implementing RenderableTable without a Header.

    Concrete case class implementing RenderableTable without a Header.

    NOTE: the existence or not of a Header in a RenderableTable only affects how the table is rendered. The parsing of a table always has a header of some sort.

    TESTME this: it is currently not used.

    Row

    the underlying type of each Row

    content

    the rows of the table.

  35. trait Validity[T] extends AnyRef

Deprecated Type Members

  1. trait OrderingWithZero[X] extends Ordering[X]
    Annotations
    @deprecated
    Deprecated

    (Since version 1.7.0) Not used by TableParser: it will be removed in 2.0.0

  2. final class Sequence extends AnyVal
    Annotations
    @deprecated
    Deprecated

    (Since version 1.7.0) Not used by TableParser: it will be removed in 2.0.0

  3. trait Sequential extends AnyRef

    A trait that defines a contract for types that are associated with a Sequence.

    A trait that defines a contract for types that are associated with a Sequence.

    Types extending this trait must provide an implementation for the sequence value, which represents an instance of the Sequence class.

    NOTE none of this modules is currently used.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.7.0) Not used by TableParser: it will be removed in 2.0.0

Value Members

  1. object Analysis

    The Analysis object provides factory methods to analyze tables from various sources.

    The Analysis object provides factory methods to analyze tables from various sources.

    Example usage:

    // Analyze a CSV-sourced RawTable
    val stats: ColumnStatistics = Analysis(rawTable)
    
    // Analyze a CSV file by Path
    implicit val parser: RawTableParser = RawTableParser()
    val stats: ColumnStatistics = Analysis.forCsv(Path.of("data.csv"))
    
    // Analyze a Parquet file (requires parquet module in scope)
    val stats: ColumnStatistics = Analysis.forParquet[YellowTaxiTrip](Path.of("data.parquet"))
  2. object Column extends Serializable
  3. object ColumnStatistics extends Serializable

    Companion object for ColumnStatistics.

    Companion object for ColumnStatistics. Provides a deprecated alias for backward compatibility.

  4. object Content extends Serializable

    The Content object provides a factory method for creating instances of the Content class.

    The Content object provides a factory method for creating instances of the Content class. This utility is designed to facilitate the creation of Content objects from standard iterable collections by converting them into parallel collections for potential parallel processing.

  5. object CsvAttributes extends Serializable

    Companion object for the CsvAttributes case class.

    Companion object for the CsvAttributes case class. Provides utility methods and implicit instances for working with CSV attributes.

  6. object HeadedTable extends Serializable

    Companion object for HeadedTable.

    Companion object for HeadedTable. The apply methods provided arbitrarily use Vector as the collection for the rows of the table. CONSIDER using something else such as Array.

    TESTME ?

  7. object Header extends Serializable

    This companion object contains utility methods and constants for creating and manipulating instances of the Header class.

  8. object Indexed extends Serializable

    Companion object for the Indexed case class, providing utility methods to create and work with indexed collections.

  9. object Row extends Serializable

    Companion object for the Row class, providing utilities for working with rows, including CSV rendering and generation.

  10. object Statistics extends Serializable

    Companion object for the Statistics case class.

    Companion object for the Statistics case class. Provides methods to create a Statistics instance from a sequence of doubles.

  11. object Table
  12. object UnheadedTable extends Serializable

Deprecated Value Members

  1. object Main

    The Main object serves as the entry point of the application and provides functionality to parse and analyze a specified dataset of crime information.

    The Main object serves as the entry point of the application and provides functionality to parse and analyze a specified dataset of crime information.

    This object: - Loads a given CSV file containing crime data. - Sets up a source for reading the data. - Configures a RawTableParser instance to parse the table data with specified predicates. - Analyzes the parsed data and prints relevant results to the console. - Demonstrates how to handle the parsing process using Scala's Try-Success-Failure mechanism.

    The application expects the file to be properly located in the resources directory.

    Members: - crimeFile: Specifies the file name of the dataset (located in the resources directory). - sy: Sets up a data source using the provided file. - fraction: Defines a fraction value for sampling the data when parsing. - parser: Configures and prepares the table parser with a predicate for data sampling.

    Main functionalities: - Parses the source using the configured RawTableParser. - On success: Performs analysis on the parsed data and logs the first 10 rows and summary information. - On failure: Throws an exception indicating the failure reason.

    Example Usage: To analyze a specific dataset, update the crimeFile variable with the correct resource file path before running the application.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.7.0) Example program: it moves out of the library, into the tests, in 2.0.0

  2. object NonSequential
    Annotations
    @deprecated
    Deprecated

    (Since version 1.7.0) Not used by TableParser: it will be removed in 2.0.0

  3. object OrderingWithZero extends Serializable
    Annotations
    @deprecated
    Deprecated

    (Since version 1.7.0) Not used by TableParser: it will be removed in 2.0.0

  4. object Sequence
    Annotations
    @deprecated
    Deprecated

    (Since version 1.7.0) Not used by TableParser: it will be removed in 2.0.0

  5. object Sequential
    Annotations
    @deprecated
    Deprecated

    (Since version 1.7.0) Not used by TableParser: it will be removed in 2.0.0

Inherited from AnyRef

Inherited from Any

Ungrouped