Skip to content

Guides

Frame operations

Each operation gives a frame with a new schema. The checker computes that schema from the schema of the input and the arguments of the call, with the rules of Polars. If Polars refuses the call, the checker reports it.

An operation that keeps the columns, for example sort, head, unique or filter, keeps the schema.

select, with_columns, drop and rename choose the columns and their names:

operations.pymypy: no errors
1class Trades(Columns):
2 symbol: pl.String
3 price: pl.Float64
4 quantity: pl.Int64
5
6
7trades: TypedDataFrame[Trades]
8trimmed = trades.drop("quantity").rename({"price": "px"})
Resulttrimmeddeclaredaddeddropped
symbolStringpxFloat64pricequantity
Revealed type
TypedDataFrame[TypedDict({'symbol': String, 'px': Float64}, closed=True), None]

A name that the frame does not have is an error. An operation that gives two columns with one name is an error too, because Polars refuses such a frame.

A join keeps the columns of both sides. Polars adds the suffix _right to a right column whose name is also a left column, and the checker does the same:

operations.pymypy: no errors
1class Trades(Columns):
2 symbol: pl.String
3 price: pl.Float64
4 quantity: pl.Int64
5
6
7trades: TypedDataFrame[Trades]
8
9
10class Quotes(Columns):
11 symbol: pl.String
12 price: pl.Float64
13 venue: pl.String
14
15
16quotes: TypedDataFrame[Quotes]
17joined = trades.join(quotes, on="symbol")
Resultjoineddeclaredadded
symbolStringpriceFloat64quantityInt64price_rightFloat64venueString
Revealed type
TypedDataFrame[TypedDict({'symbol': String, 'price': Float64, 'quantity': Int64, 'price_right': Float64, 'venue': String}, closed=True), None]

The keys of the two sides must have data types that Polars can join. A String key and an Int64 key are an error. join_asof and join_where are checked too.

group_by(...).agg(...) keeps the keys and one column for each aggregation. The shorthands, for example len(), sum() and max(), give their own columns:

operations.pymypy: no errors
1class Trades(Columns):
2 symbol: pl.String
3 price: pl.Float64
4 quantity: pl.Int64
5
6
7trades: TypedDataFrame[Trades]
8by_symbol = trades.group_by("symbol").agg(pl.col("price").mean(), pl.col("quantity").sum())
9counts = trades.group_by("symbol").len()
Resultby_symboldeclared
symbolStringpriceFloat64quantityInt64
Revealed type
TypedDataFrame[TypedDict({'symbol': String, 'price': Float64, 'quantity': Int64}, closed=True), None]
Resultcountsdeclaredadded
symbolStringlenUInt32
Revealed type
TypedDataFrame[TypedDict({'symbol': String, 'len': UInt32}, closed=True), None]

rolling(...).agg(...) and group_by_dynamic(...).agg(...) group the rows into windows along an index column. The checker checks that the index has a data type that the call accepts.

unpivot turns columns into rows. with_row_index adds a row number:

operations.pymypy: no errors
1class Trades(Columns):
2 symbol: pl.String
3 price: pl.Float64
4 quantity: pl.Int64
5
6
7trades: TypedDataFrame[Trades]
8long = trades.unpivot(index="symbol")
9numbered = trades.with_row_index()
Resultlongdeclaredaddeddropped
symbolStringvariableStringvalueFloat64pricequantity
Revealed type
TypedDataFrame[TypedDict({'symbol': String, 'variable': String, 'value': Float64}, closed=True), None]
Resultnumbereddeclaredadded
indexUInt32symbolStringpriceFloat64quantityInt64
Revealed type
TypedDataFrame[TypedDict({'index': UInt32, 'symbol': String, 'price': Float64, 'quantity': Int64}, closed=True), None]

pivot turns rows into columns. explode needs a List or an Array column, and unnest needs a Struct column:

operations.pymypy: 1 error
1import polars as pl
2
3from astrolabe import Columns, List, TypedDataFrame
4
5
6class Baskets(Columns):
7 basket: pl.String
8 symbols: List[pl.String]
9
10
11baskets: TypedDataFrame[Baskets]
12exploded = baskets.explode("symbols")
13baskets.explode("basket")
error: Column "basket" is not List/Array, cannot explode [wrong-column-shape]
Resultexplodeddeclared
basketStringsymbolsString
Revealed type
TypedDataFrame[TypedDict({'basket': String, 'symbols': String}, closed=True), None]

pl.concat with how="diagonal" lines up the columns by name. A shared column must have one data type in every frame:

operations.pymypy: no errors
1class Trades(Columns):
2 symbol: pl.String
3 price: pl.Float64
4 quantity: pl.Int64
5
6
7trades: TypedDataFrame[Trades]
8
9
10class Quotes(Columns):
11 symbol: pl.String
12 venue: pl.String
13
14
15quotes: TypedDataFrame[Quotes]
16diagonal = pl.concat([trades, quotes], how="diagonal")
17vertical = pl.concat([trades, trades])
Resultdiagonaldeclared
symbolStringpriceFloat64quantityInt64venueString
Revealed type
TypedDataFrame[TypedDict({'symbol': String, 'price': Float64, 'quantity': Int64, 'venue': String}, closed=True), None]
Resultvertical
DataFrame

A vertical concat stacks the frames by the position of their columns. A type does not hold the order of the columns, so the checker gives the plain frame. The setting disallow_vertical_concat makes each vertical concat an error. hstack, vstack, extend and merge_sorted are checked too.

fill_null and fill_nan give each column the data type that Polars gives after the fill. A fill value of another data type can widen a column: fill_null(0.5) turns an Int64 column into a Float64 column. The whole-frame reducers, for example sum() and mean(), give one row, with the data type that the reducer gives for each column.

See Operations for the full list.