Skip to content

Guides

Runtime checks

TypedDataFrame and TypedLazyFrame are pl.DataFrame and pl.LazyFrame at run time. A typed frame costs nothing, and the checker does not look at the data. For data that comes from outside the program, bind the frame to its schema once, at the boundary.

Each function takes a frame and a frame type, and gives the frame with that type. For mypy, each call is a typing.cast to the frame type:

boundary.pymypy: no errors
1import polars as pl
2
3from astrolabe import Columns, TypedDataFrame, TypedLazyFrame, check_schema, force_schema
4
5
6class Trades(Columns):
7 symbol: pl.String
8 price: pl.Float64
9
10
11checked = check_schema(pl.read_parquet("trades.parquet"), TypedDataFrame[Trades])
12cast = force_schema(pl.read_csv("trades.csv"), TypedDataFrame[Trades])
13lazy = check_schema(pl.scan_parquet("trades.parquet"), TypedLazyFrame[Trades])
Resultlazydeclared
symbolStringpriceFloat64
Revealed type
TypedLazyFrame[TypedDict(Trades, {'symbol': String, 'price': Float64}, closed=True), None]
FunctionAt run time
check_schema(df, Frame)Checks that the frame has exactly the declared columns, with their data types. It casts nothing, and it keeps the order of the columns.
force_schema(df, Frame)Keeps only the declared columns, in their declared order, and casts each one to its declared data type. With strict=False, a value that Polars cannot cast becomes null.
assume_schema(df, Frame)Checks nothing.

check_schema raises if a column is missing, if a column has another data type, or if the frame has a column that the type does not declare. A column declared Unchecked must be present, and its data type is not checked.

Each function needs a closed frame type: a schema class, and no remainder. The undeclared columns of an open frame have no data type to check or to cast, so an open frame type is an error:

boundary.pymypy: 1 error
1import polars as pl
2
3from astrolabe import Columns, TypedDataFrame, check_schema, other_columns
4
5
6class Trades(Columns):
7 symbol: pl.String
8 price: pl.Float64
9
10
11Rest = other_columns()
12
13raw = pl.read_parquet("trades.parquet")
14trades = check_schema(raw, TypedDataFrame[Trades, Rest])
error: check_schema() needs a closed frame type, and this frame type has undeclared columns. Pass a frame type with a schema class and no remainder, for example TypedDataFrame[MyColumns] [frame-type-not-closed]

A frame type with no schema class, as TypedDataFrame[Rest], is the same error.

.schema on a typed frame is the Schema of Polars. Each declaration is an ordinary value at run time, so column("price", pl.Float64) can be inspected.