Lazy frames

A lazy frame runs nothing until collect(). TypedLazyFrame is the lazy form of TypedDataFrame, and every operation has the same checks.

Scan, transform, collect

A scan_* reader with a schema gives a typed lazy frame. collect() gives the typed frame:

lazy.pymypy: no errors

1import polars as pl

2

3trades = pl.scan_csv("trades.csv", schema={"symbol": pl.String, "price": pl.Float64})

4large = trades.filter(pl.col("price") > 100).collect()

Resultlargedeclared

symbolStringpriceFloat64

Revealed type

TypedDataFrame[TypedDict({'symbol': String, 'price': Float64}, closed=True), None]

lazy() turns a TypedDataFrame into a TypedLazyFrame, and keeps the schema.

Two data types for one column

Polars plans each later step at the data type that collect_schema() shows. For a few expressions, that data type differs from the data type of the data. Such a column holds both, as Planned[run, planner], and collect() gives the run type:

lazy.pymypy: no errors

1from typing import Literal

2

3import polars as pl

4

5from astrolabe import Columns, Decimal, TypedLazyFrame

6

7

8class Lots(Columns):

9 price: Decimal[Literal[10], Literal[2]]

10

11

12lots: TypedLazyFrame[Lots]

13running = lots.with_columns(total=pl.col("price").cum_sum())

Resultrunningdeclared

priceDecimal[Literal[10], Literal[2]]totalPlanned[Decimal[Literal[38], Literal[2]], Decimal[Literal[10], Literal[2]]]

Revealed type

TypedLazyFrame[TypedDict({'price': Decimal[Literal[10], Literal[2]], 'total': Planned[Decimal[Literal[38], Literal[2]], Decimal[Literal[10], Literal[2]]]}, closed=True), None]

Resultrunning.collect()declared

priceDecimal[Literal[10], Literal[2]]totalDecimal[Literal[38], Literal[2]]

Revealed type

TypedDataFrame[TypedDict({'price': Decimal[Literal[10], Literal[2]], 'total': Decimal[Literal[38], Literal[2]]}, closed=True), None]

The cumulative sum gives a Decimal(38, 2), and Polars plans it as a Decimal(10, 2). A later step of the lazy frame works at the planned type.

To give the column one data type, cast it: pl.col("price").cum_sum().cast(pl.Decimal(38, 2)).