Schemas

A schema is the columns of a frame, each with a name and a data type. A schema has no order. Declare it once, and use it in the type of a frame.

Declare a schema with a class

A Columns class lists the columns. Each field is a column, and its type is a Polars data type. Columns declares no column of its own. A subclass inherits the columns of its base:

schemas.pymypy: no errors

1import polars as pl

2

3from astrolabe import Columns, TypedDataFrame

4

5

6class Trades(Columns):

7 symbol: pl.String

8 price: pl.Float64

9 quantity: pl.Int64

10

11

12class Blotter(Trades):

13 venue: pl.String

14

15

16blotter: TypedDataFrame[Blotter]

Resultblotterdeclared

symbolStringpriceFloat64quantityInt64venueString

Revealed type

TypedDataFrame[TypedDict(Blotter, {'symbol': String, 'price': Float64, 'quantity': Int64, 'venue': String}, closed=True), None]

TypedDataFrame[Blotter] is a closed frame: it has exactly the four columns. To accept a frame with more columns, see Open frames.

TypedDataFrame with no argument names no column. The checker reads it as the plain frame pl.DataFrame, and does not track it. TypedDataFrame[()] is a frame with no column.

Declare a schema with a dict

A column name that is not a Python identifier cannot be a field. Declare such a schema with columns(), from a name and a dict:

schemas.pymypy: no errors

1import polars as pl

2

3from astrolabe import TypedDataFrame, columns

4

5Pivoted = columns("Pivoted", {"id": pl.Int64, "2024": pl.Float64, "v1|a": pl.Datetime("us")})

6

7pivoted: TypedDataFrame[Pivoted]

8picked = pivoted.select("id", "2024")

Resultpickeddeclared

idInt642024Float64

Revealed type

TypedDataFrame[TypedDict({'id': Int64, '2024': Float64}, closed=True), None]

Give the parameters of a data type

Some data types have parameters: the time unit of a Datetime, the precision of a Decimal, the categories of an Enum. A field is a type annotation, so it names these data types with a marker:

schemas.pymypy: no errors

1from typing import Literal

2

3import polars as pl

4

5from astrolabe import Columns, Datetime, Decimal, Enum, List, TypedDataFrame

6

7

8class Fills(Columns):

9 time: Datetime[Literal["us"], None]

10 cost: Decimal[Literal[10], Literal[2]]

11 lots: List[pl.Int64]

12 side: Enum[Literal["buy"], Literal["sell"]]

13

14

15fills: TypedDataFrame[Fills]

Resultfillsdeclared

timeDatetime[Literal['us'], None]costDecimal[Literal[10], Literal[2]]lotsList[Int64]sideEnum[tuple[Literal['buy'], Literal['sell']]]

Revealed type

TypedDataFrame[TypedDict(Fills, {'time': Datetime[Literal['us'], None], 'cost': Decimal[Literal[10], Literal[2]], 'lots': List[Int64], 'side': Enum[tuple[Literal['buy'], Literal['sell']]]}, closed=True), None]

The checker keeps the parameters. A Datetime in microseconds is a different data type from a Datetime in milliseconds. See Data types for each marker.

Keep a data type in a name

A name can hold a data type. Write it as you write it everywhere else in Polars:

schemas.pymypy: no errors

1import polars as pl

2

3from astrolabe import TypedDataFrame, columns

4

5Stamp = pl.Datetime("us")

6Money = pl.Decimal(20, 4)

7Trade = columns("Trade", {"time": Stamp, "cost": Money})

8

9trade: TypedDataFrame[Trade]

Resulttradedeclared

timeDatetime[Literal['us'], None]costDecimal[Literal[20], Literal[4]]

Revealed type

TypedDataFrame[TypedDict(Trade, {'time': Datetime[Literal['us'], None], 'cost': Decimal[Literal[20], Literal[4]]}, closed=True), None]

Every declaration that takes a data type as an argument reads such a name, with its parameters: columns(...), column(...) and other_columns(all_of=...). A name bound to another name resolves too.

A field is a type annotation, and mypy refuses a variable there. In a Columns class, name the data type with its marker: Stamped = Datetime[Literal["us"], None] is a type alias, and time: Stamped declares the field.

Write a generic schema

A schema can take a data type as a parameter. Each use of the schema gives the parameter:

schemas.pymypy: no errors

1import polars as pl

2

3from astrolabe import Columns, TypedDataFrame

4

5

6class Keyed[T: pl.DataType](Columns):

7 key: pl.String

8 value: T

9

10

11def non_empty[T: pl.DataType](df: TypedDataFrame[Keyed[T]]) -> TypedDataFrame[Keyed[T]]:

12 return df.filter(pl.col("key").str.len_chars() > 0)

13

14

15counts: TypedDataFrame[Keyed[pl.UInt32]]

16kept = non_empty(counts)

Resultkeptdeclared

keyStringvalueUInt32

Revealed type

TypedDataFrame[TypedDict(Keyed, {'key': String, 'value': UInt32}, closed=True), None]

The checker binds T to UInt32 at the call, so the result has the data type of the caller.

Rules of a schema

A Polars column is never optional, and the declared columns of a frame are complete. So a Columns class has two rules:

• Each field has a Polars data type, or a marker of one.
• The class does not use total=False, closed=False, Required[...] or NotRequired[...].

The checker reports a class that breaks a rule where a frame type uses the class:

schemas.pymypy: 2 errors

1import polars as pl

2

3from astrolabe import Columns, TypedDataFrame

4

5

6class Prices(Columns):

7 symbol: pl.String

8 price: float

↑error: Column "price" must be a polars dtype or an astrolabe dtype marker [columns-field-type]

9

10

11class Partial(Columns, total=False):

↑error: Columns subclass "Partial" must not use total=False, Required[...], or NotRequired[...]: every column is required [columns-not-total]

12 symbol: pl.String

13

14

15prices: TypedDataFrame[Prices]

16partial: TypedDataFrame[Partial]