3. Write a function that accepts other columns

A pipeline puts its steps in functions. A function reads some columns, and a caller often has more. This chapter declares a function that accepts any frame with the columns that it reads.

Declare the columns that the function reads

A Columns class names columns and their data types, like a dataclass. other_columns() declares a remainder: the columns of a frame that its type does not name. Write it as the second argument of TypedDataFrame:

features.pymypy: no errors

1import polars as pl

2

3from astrolabe import Columns, TypedDataFrame, other_columns

4

5

6class Priced(Columns):

7 symbol: pl.String

8 price: pl.Float64

9 quantity: pl.Int64

10

11

12class Turnover(Columns):

13 symbol: pl.String

14 turnover: pl.Float64

15

16

17Rest = other_columns()

18

19

20def turnover(df: TypedDataFrame[Priced, Rest]) -> TypedDataFrame[Turnover]:

21 value = (pl.col("price") * pl.col("quantity")).sum()

22 return df.group_by("symbol").agg(value.alias("turnover"))

TypedDataFrame[Priced, Rest] accepts any frame with symbol, price and quantity, whatever other columns it has. A frame of this type is an open frame. A frame with no remainder, like TypedDataFrame[Turnover], is a closed frame: it has exactly its declared columns.

The checker checks each call

features.pymypy: 1 error

1class Priced(Columns):

2 symbol: pl.String

3 price: pl.Float64

4 quantity: pl.Int64

5

6

7class Turnover(Columns):

8 symbol: pl.String

9 turnover: pl.Float64

10

11

12Rest = other_columns()

13

14

15def turnover(df: TypedDataFrame[Priced, Rest]) -> TypedDataFrame[Turnover]:

16 value = (pl.col("price") * pl.col("quantity")).sum()

17 return df.group_by("symbol").agg(value.alias("turnover"))

18

19

20class Blotter(Priced):

21 venue: pl.String

22

23

24class Quotes(Columns):

25 symbol: pl.String

26 price: pl.Float64

27

28

29blotter: TypedDataFrame[Blotter]

30quotes: TypedDataFrame[Quotes]

31turnover(blotter)

32turnover(quotes)

↑error: Missing required column "quantity" [missing-column]

• blotter has the three columns and one more, venue. The call passes.
• quotes has no quantity. The checker reports the call, not the body of the function.

The body sees only the declared columns

A caller can pass a frame without venue, so the body cannot use it:

features.pymypy: 1 error

1class Priced(Columns):

2 symbol: pl.String

3 price: pl.Float64

4 quantity: pl.Int64

5

6

7Rest = other_columns()

8

9

10def by_venue(df: TypedDataFrame[Priced, Rest]) -> None:

11 df.group_by("venue").agg(pl.len())

↑error: The result of "group_by" has no known schema: the undeclared columns of the frame can hold the name, or not [implicitly-untyped]

↑note: set allow_unproven = "possible" in [tool.astrolabe] to allow this, or set [tool.astrolabe.allow_unproven_overrides] group_by = "possible"

The remainder Rest can hold venue or not, so the checker cannot give the result a schema. The note under the error names the level of the setting allow_unproven at which the line passes: see Make the checker stricter. To use venue, declare it in the class of the parameter. Then each caller must pass it.

Next: 4. Let the body decide the return type.