Frames as types

A type describes a set of values. The type of a frame describes a set of frames: those with some columns, each with a data type.

Schema

A schema is the columns of a frame, each with a name and a data type. A schema has no order: two frames with the same columns in another order have the same schema. The order of the columns is a fact of the data, and no type keeps it.

Closed and open frames

A frame type names its declared columns. It can also have a remainder, which stands for the columns that it does not name.

• A closed frame has no remainder. Its type describes the frames with exactly the declared columns. TypedDataFrame[Trades] is closed.
• An open frame has a remainder. Its type describes every frame with at least the declared columns. TypedDataFrame[Trades, Rest] is open.

A closed frame is the type of a concrete frame in a program. An open frame is the type of a parameter that accepts many frames.

The remainder follows the caller

A remainder is a variable, like a type variable of a generic function. At each call, the checker puts the columns of the caller in its place. So the result of a function describes the real frame of each caller.

Below, add_notional accepts any frame with the columns of Trades. Rest stands for the other columns of the caller. The return type writes InferredRemainder for the remainder of the result, and infer(...) lets the checker compute it from the body. A Blotter frame has one more column, venue, and the result of the call keeps it:

notional.pymypy: no errors

1class Trades(Columns):

2 symbol: pl.String

3 price: pl.Float64

4 quantity: pl.Int64

5

6

7class WithNotional(Trades):

8 notional: pl.Float64

9

10

11Rest = other_columns()

12

13

14def add_notional(

15 df: TypedDataFrame[Trades, Rest],

16) -> TypedDataFrame[WithNotional, InferredRemainder]:

17 return infer(df.with_columns(notional=pl.col("price") * pl.col("quantity")))

18

19

20class Blotter(Trades):

21 venue: pl.String

22

23

24blotter: TypedDataFrame[Blotter]

25result = add_notional(blotter)

Resultresultdeclaredaddedfrom the remainder

symbolStringpriceFloat64quantityInt64notionalFloat64venueString

Revealed type

TypedDataFrame[TypedDict({'symbol': String, 'price': Float64, 'quantity': Int64, 'notional': Float64, 'venue': String}, closed=True), None]

The checker follows what each operation of the body does to the remainder. Here with_columns replaces a notional column of the caller, so the remainder of the result holds every column of the caller except notional. At the call, that remainder is venue, a column that the function never names.

A column with an unknown data type

Unchecked is the data type of a column whose name is known and whose data type is not. It keeps the rest of the schema exact, where the alternative is to give up on the whole frame. Each later operation accepts it, as mypy accepts a value of type Any.

Plain frames

A plain frame is a pl.DataFrame that the checker does not track. It is the type of a frame whose columns no code states, for example the result of a reader with no schema. The checker reports nothing about a plain frame, so code that does not use Astrolabe checks as before.