Proposal: KeyType and ElementType for TypedDicts (also, Map)

[tmke8]

Previously:

• `TypeVarDict` for DataFrames and other TypedDict-like containers (also called “key types”) #1387

The motivation is to provide a typing mechanism for dict-like containers (like pandas.DataFrame) to have key-wise type annotations – in the same way that TypedDict provides key-wise types for dict.

To this end, we introduce three new special forms which act as type operators on Mappings and TypedDicts: KeyType, ElementType and Map.

As a motivating example, here is how you would type-annotate pandas.DataFrame with the proposed mechanism – the most important part is the definition of __getitem__. (In the comments, I’m sometimes using the syntax TypedDict({"foo": int, "bar": str}) for “anonymous TypedDicts”.)

from typing import Any, ElementType, KeyType, Map, Mapping, TypedDict
import numpy as np
from numpy import typing as npt


class Series[T]: ...


class DataFrame[TD: Mapping[str, Any]]:  # we use `Mapping[str, Any]` as the bound because it is covariant

    # `Map` maps generic types over the types of a TypedDict.
    # So, if `TD` corresponds to `TypedDict({"col1": int})`,
    # then `data` has the type `TypedDict({"col1": Series[int]})`.
    data: Map[Series, TD]

    # here, `Map` basically strips off `NDArray` from the value types
    def __init__(self, values: Map[npt.NDArray, TD]):
        self.data = {k: Series(v) for k, v in values.items()}

    # using `KeyType` and `ElementType` to access entries of the TypedDict
    def __getitem__[K: KeyType[TD]](self, col: K) -> Series[ElementType[TD, K]]:
        return self.data[col]


df = DataFrame({
    "a": np.array([0, 3, 4], dtype=np.int64),
    "b": np.array([1.2, -0.9, 0.0], dtype=np.float64),
})
reveal_type(df)  # `DataFrame[TypedDict({"a": np.int64, "b": np.float64})]`
reveal_type(df["a"])  # `Series[np.int64]`
df["wrong key"]  # type error!


# a constructor for `DataFrame` with explicit `dtype`
def read_csv[TD: : Mapping[str, Any]](filepath: Path, dtype: Map[type, TD]) -> DataFrame[TD]: ...

df2 = read_csv(Path("d.csv"), {"c1": np.float64, "c2": np.int32})
reveal_type(df2)  # DataFrame[TypedDict({"c1": np.float64, "c2": np.int32})]

c1: Series[np.float64] = df2["c1"]

Now I’ll explain everything in more detail.

How do KeyType and ElementType work?

If TD is a TypedDict (or a TypeVar with bound TypedDict), then KeyType[TD] is a union of all key types. Consider:

from typing import KeyType, TypedDict

class MyDict(TypedDict):
    a: int
    b: str

type Keys = KeyType[MyDict]  # equivalent to `Literal["a", "b"]`
key: Keys = "a"  # OK
notakey: Keys = "c"  # type error

ElementType, on the other hand, represents the type of a specific element of a TypedDict:

from typing import ElementType, TypedDict

class MyDict(TypedDict):
    a: int
    b: str

type TypeofA = ElementType[MyDict, Literal["a"]]  # equivalent to `int`
type TypeofB = ElementType[MyDict, Literal["b"]]  # equivalent to `str`
type TypeofC = ElementType[MyDict, Literal["c"]]  # type error

If the second type argument to ElementType is a union, then the resulting type is also a union:

# ...
type ValueTypes = ElementType[MyDict, Literal["a", "b"]]  # `int | str`
x: ValueTypes = 0  #OK
y: ValueTypes = "foo"  #OK

This means we can combine KeyType and ElementType like this:

from typing import ElementType, TypedDict

Movie = TypedDict("Movie", {"name": str, "year": int})

def f(x: ElementType[Movie, KeyType[Movie]]) -> None:
    print(x)

Let’s unpack what’s happening here. First, we apply KeyType[] to MyDict. This results in Literal["a", "b"]. So, the definition of f above is equivalent to:

# ...
def f(x: ElementType[Movie, Literal["name", "year"]]) -> None:
    print(x)

Then, we can expand ElementType. It returns the union of all types that are associated with the given key types. The result is:

# ...
def f(x: str | int) -> None:
    print(x)

In general, ElementType[TD, KeyType[TD]] is the union of all element types in TD.

However, where it gets really interesting is when we the function generic with KeyType as the bound:

from typing import ElementType, KeyType, TypedDict

class MyDict(TypedDict):
    a: int
    b: str

def get[K: KeyType[MyDict]](key: K) -> ElementType[MyDict, K]:
    my_dict: MyDict = ...  # get data from somewhere
    return my_dict[key]

As discussed, KeyType[MyDict] is Literal["a", "b"], so the type variable K can either be Literal["a"] or Literal["b"]. Therefore, the above definition is equivalent to this overload:

from typing import Literal, overload

@overload
def get(key: Literal["a"]) -> int: ...
@overload
def get(key: Literal["b"]) -> str: ...
def get(key: Literal["a", "b"]) -> int | str:
    my_dict = ...  # get data from somewhere
    return my_dict[key]

We can also combine this with a type variable that has Mapping as the bound:

from collections.abc import Mapping
from typing import ElementType, KeyType, TypedDict, Unpack

class C[TD: Mapping[str, Any]]:
    def __init__(self, **kwargs: Unpack[TD]):
        self.vals = kwargs 

    def __setitem__[K: KeyType[TD]](self, index: K, value: ElementType[TD, K]) -> None:
        self.vals[index] = value 

MyTypes = TypedDict("MyTypes", {"a": str, "b": bool})

c: C[MyTypes] = ...
c["a"] = "foo"  # OK
c["a"] = 3  # type error!
c["b"] = True  # OK

Here, K: KeyType[TD] means that K is a key type of the arbitrary Mapping TD. This is equivalent to:

from typing import Literal, overload

class C:
    def __init__(self, **kwargs):
        self.vals = kwargs 

    @overload
    def __setitem__(self, index: Literal["a"], value: str) -> None: ...
    @overload
    def __setitem__(self, index: Literal["b"], value: bool) -> None: ...
    def __setitem__(self, index: Literal["a", "b"], value: bool | str) -> None: ...
        self.vals[index] = value 

c: C[{"a": str, "b": bool}] = ...
c["a"] = "foo"  # OK
c["a"] = 3  # type error!
c["b"] = True  # OK

Example: TypedMapping

Full example putting everything together:

from collections.abc import ItemsView, KeysView, Mapping, ValuesView
from typing import ElementType, KeyType, TypedDict, Unpack

class TypedMapping[TD: Mapping[str, Any]]:
    def __init__(self, **kwargs: Unpack[TD]): ...
    def __getitem__[K: KeyType[TD]](self, index: K) -> ElementType[TD, K]: ...
    def keys(self) -> KeysView[KeyType[TD]]: ...
    def values(self) -> ValuesView[ElementType[TD, KeyType[TD]]]: ...
    def items(self) -> ItemsView[KeyType[TD], ElementType[TD, KeyType[TD]]]: ...

def f(a: str, b: bool) -> None:
    m = TypedMapping(a=a, b=b)

    for v in m.values():
        reveal_type(v)  # Union[str, bool]

KeyType and ElementType on normal dict types and Mapping types

For a type like this:

type Prices = dict[str, int]

we can also use KeyType and ElementType to extract types:

KeyType[Prices] == str
ElementType[Prices, Literal["any arbitrary string"]] == int

It works the same for Mapping.

How does Map work?

To really make TypeVar with TypedDict bound useful, we introduce the special form Map as well. Map was originally introduced in this proto-PEP.

It works like this:

# map `list` onto the types in the TypedDict
Map[list, TypedDict({"a": str, "b": bool})] == TypedDict({"a": list[str], "b": list[bool]})

The first argument to Map has to be a generic type with exactly one TypeVar slot.

This is needed for example in the definition of read_csv:

def read_csv[TD: Mapping[str, Any]](filepath: Path, dtype: Map[type, TD]) -> DataFrame[TD]: ...

The dtype object that you pass in will look something like {"col1": np.int64} but that has type TypedDict({"col1": type[np.int64]}), and not type TypedDict({"col1": np.int64}) which is what we need in order to infer the correct type for the DataFrame.

So, the type[] needs to be stripped away somehow. That is what Map does: the dtype we pass in has type TypedDict({"col1": type[np.int64]}) which gets matched to Map[type, TD] which means that TD is inferred as TypedDict({"col1": np.int64}), just as we wanted.

Interaction with KeyType and ElementType

Map does not affect the keys. So we have

KeyType[Map[list, TD]] == KeyType[TD]

The types of the elements are wrapped in the generic type:

ElementType[Map[list, TD], K] == list[ElementType[TD, K]]

Map on normal dict types and Mapping types

For a type like this:

type Prices = Mapping[str, int]

Map still works:

Map[list, Prices] == dict[str, list[int]]

Aside on TypeVarTuple

The proto-PEP linked above defines Map to be used on TypeVarTuples like this:

def product[*Ts](*it: *Map[Iterable, Ts]) -> Iterable[Tuple[*Ts]]: ...

# generalization of
T1 = TypeVar("T1")
T2 = TypeVar("T2")
def product(it1: Iterable[T1], it2: Iterable[T2]) -> Iterable[Tuple[T1, T2]]: ...

# informal meaning:
Map[Iterable, (int, str)] == (Iterable[int], Iterable[str])

Interaction with other features of TypedDict

total=False and NotRequired

Not completely sure how to deal with fields from non-total TypedDicts and entries marked as NotRequired. I suppose the easiest solution is to drop those fields entirely in KeyType.

It should be allowed to map Required or NotRequired with Map over a TypedDict.

from typing import Map, NotRequired, TypedDict

Movie = TypedDict("Movie", {"name": str, "year": int})

type MovieNothingRequired = Map[NotRequired, Movie]
# equivalent to
# MovieNothingRequired = TypedDict(
#     "MovieNothingRequired",
#     {"name": NotRequired[str], "year": NotRequired[int]},
# )

readonly=True and ReadOnly

There is a PEP proposing to extend TypedDict with the ability to declare read-only fields. It would be nice to somehow honor this in dict-like containers that are derived from TypedDict via the mechanism shown for TypedMapping above, but no proposal has been made yet for this.

Who would use this?

Any library that has DataFrame-like objects:

• pandas
• polars
• structured arrays in numpy

Dict-wrappers like ModuleDict in PyTorch.

Relation to other PEPs

This proposal would synergize extremely well with a PEP that was proposing an inline syntax for TypedDict.

Survey of other programming languages

TypeScript

KeyType corresponds straightforwardly to the keyof operator in TypeScript:

type Point = { x: number; y: number };
type P = keyof Point;  // equivalent to `type P = "x" | "y"`

ValueType can also be expressed in TypeScript:

type Person = { age: number; name: string; alive: boolean };
type V = Person[keyof Person];  // equivalent to `type V = string | number | boolean`

This is possible because Person["age"], for example, is used in TypeScript to refer to the type of the field "age" in the type Person.

Generic types bound to the key types are also possible:

type Person = { age: number; name: string; alive: boolean };

let person: Person = {age: 3, name: "foo", alive: false};

function get<K extends keyof Person>(key: K): Person[K] {
  return person[key]
}

let y = get("age"); // OK
let x = get("bar"); // error

This syntax in TypeScript is very elegant but it is based on the Person[K] syntax (which returns the value type of the entry with key K), which does not seem to be possible in Python because Python already uses square brackets for generics (whereas TypeScript uses angled brackets for generics).

The functionality of Map can be recreated with mapped types in TypeScript (though note that mapped types are more powerful than the proposed Map operator):

type Person = { age: number; name: string; alive: boolean };

type WrappedPerson = {
  [Key in keyof Person]: Array<Person[Key]>;
};

Flow

KeyType[T] is $Keys<T> in Flow:

type Person = {age: number, name: string, alive: boolean};

const person: Person = {age: 24, name: "foo", alive: false};

function get<K: $Keys<Person>>(key: K): Person[K] {
  return person[key];
}

const age = get("age");  // OK
const bar = get("bar");  // error

type WrappedPerson = $ObjMap<Person, <V>(V) => Array<V>>;

Flow now uses the same syntax as TypeScript for accessing the type of a certain entry: Person[K], but before that, there was another, now deprecated mechanism for this: $ElementType<Person, K>. This proposal is directly inspired by the deprecated Flow syntax. Again, we unfortunately can’t use the Person[K] syntax because we use square brackets for generics.

$ObjMap in Flow is very similar to the proposed Map, though $ObjMap is more powerful because it accepts an arbitrary lambda to map the entry types to other types.

Prior art in typing of DataFrames

static_frame

static_frame is a Python library for statically typing data frames. It uses variadic generics to allow specifying the types of the columns:

from typing import Any
import numpy as np
from static_frame import Frame, Series, Index, IndexYearMonth

def process(f: Frame[   # type of the container
        Any,            # type of the index labels
        Index[np.str_], # type of the column labels
        np.int_,        # type of the first column
        np.str_,        # type of the second column
        np.float64,     # type of the third column
        ]) ->  -> dict[
                int,
                Series[                 # type of the container
                        IndexYearMonth, # type of the index labels
                        np.float64,     # type of the values
                        ],
                ]: ...

The downside of this approach is that when columns are accessed by column name (as opposed to column index), the type system cannot infer the column's type.

Pandera

Pandera allows validation of pandas DataFrames:

import pandas as pd
import pandera as pa
from pandera.typing import Index, DataFrame, Series  # these are needed because the pandas classes aren't generic

class InputSchema(pa.DataFrameModel):
    year: Series[int] = pa.Field(gt=2000, coerce=True)
    month: Series[int] = pa.Field(ge=1, le=12, coerce=True)
    day: Series[int] = pa.Field(ge=0, le=365, coerce=True)

class OutputSchema(InputSchema):
    revenue: Series[float]

@pa.check_types
def transform(df: DataFrame[InputSchema]) -> DataFrame[OutputSchema]:
    return df.assign(revenue=100.0)

df = pd.DataFrame({
    "year": ["2001", "2002", "2003"],
    "month": ["3", "6", "12"],
    "day": ["200", "156", "365"],
})

transform(df)

invalid_df = pd.DataFrame({
    "year": ["2001", "2002", "1999"],
    "month": ["3", "6", "12"],
    "day": ["200", "156", "365"],
})
transform(invalid_df)  # runtime error

This works for runtime type validation, but static type checkers can't understand it. With this PEP, it could be re-written as:

from typing import Annotated, TypedDict
import pandas as pd
import pandera as pa


class InputSchema(TypedDict):
    year: Annotated[int, pa.ValueRange(2000, None), pa.Coerce(True)]
    month: Annotated[int, pa.ValueRange(1, 12), pa.Coerce(True)]
    day: Annotated[int, pa.ValueRange(0, 365), pa.Coerce(True)]

class OutputSchema(InputSchema):
    revenue: float

@pa.check_types
def transform(df: pd.DataFrame[InputSchema]) -> pd.DataFrame[OutputSchema]:
    return df.assign(revenue=100.0)

# ...

[JelleZijlstra]

I like this idea and I'd be willing to sponsor a PEP for it.

Some early feedback:

• It bothers me that the names are Key and ValueType. Why not Key and Value, or KeyType and ValueType?
• I would suggest that this PEP should only define the behavior of Map on TypedDicts. A future PEP can extend it to TypeVarTuple, similar to how Unpack was initially (PEP 646) only defined for TypeVarTuple and later (PEP 692) extended to TypedDict.
• We probably do need Keys and ValueTypes as the feature would feel incomplete otherwise; we should be able to fully annotate a type like TypedMapping in your example. However, the distinction between Key and Keys may be confusing for users. Maybe there's a way to unify them.

[tmke8]

It bothers me that the names are Key and ValueType. Why not Key and Value, or KeyType and ValueType?

Yes, I basically couldn't decide which naming scheme to use and ended up with a hybrid. Which one do you suggest?

However, the distinction between Key and Keys may be confusing for users. Maybe there's a way to unify them.

Agreed. If anyone has suggestions?

[JelleZijlstra]

Yes, I basically couldn't decide which naming scheme to use and ended up with a hybrid. Which one do you suggest?

I would prefer KeyType and ValueType. I think Key and Value are a bit too short to be meaningful, considering that we'll expect people to mostly use from typing import Key, Value.

Agreed. If anyone has suggestions?

The idea that came to mind when reading the above was to drop the rule that Key and ValueType have to appear together in a signature. So when you just use Key[TD], it means "any Key of the TD", but if you use both in the same signature, it means the two have to match. That feels quite magical, though.

[tmke8]

Changed Key to KeyType 👍

The idea that came to mind when reading the above was to drop the rule that Key and ValueType have to appear together in a signature.

That's a good idea. You could actually think of it as if the overloads are still there, but they just amount to a union because only one argument varies. So:

class MyDict(TypedDict):
    a: int
    b: str

def f(x: ValueType[MyDict]) -> None:
    print(x)

would be unrolled to

@overload
def f(x: int) -> None: ...
@overload
def f(x: str) -> None: ...
def f(x: int | str) -> None:
    print(x)

which is equivalent to

def f(x: int | str) -> None:
    print(x)

Though there are maybe some problems. What if I want to express that my function takes a list of keys and a list of values, but they don't have to match?

class MyDict(TypedDict):
    a: int
    b: str

def f(keys: list[KeyType[MyDict]], values: list[ValueType[MyDict]]) -> None:
    pass # do something with the lists

This would get unrolled to

@overload
def f(keys: list[Literal["a"]], values: list[int]) -> None: ...
@overload
def f(keys: list[Literal["b"]], values: list[str]) -> None: ...
def f(keys: list[Literal["a"]] | list[Literal["b"]], values: list[int] | list[str]) -> None:
    print(x)

which is probably not what we wanted.

One solution I could think of for this is to define a type alias:

type AllKeys = KeyType[MyDict]

and then we say that the overload magic is not applied for type aliases?

[erictraut]

A few thoughts...

• I generally like the direction, and I think this is worthwhile pursuing.
• I feel like we're jumping too quickly to a detailed proposal. When exploring new language features, I find that it's important to 1) research how other programming languages have solved similar problems, 2) clearly write down the motivation for the feature and the capabilities it's intended to provide, 3) explore half a dozen alternative solutions — including significant variations — to get a feel for the pros/cons of each. I find that it's useful at that point to present several of the best variants to a few others to get their input. Only then do I whittle it down to a single proposal and get broader feedback. That may sound like a lot of work, but when I try to rush the process and skip these steps, I end up with a suboptimal solution. Perhaps you've already done some of these steps behind the scenes. If so, great. If not, I'd encourage you to go through a process similar to what I've outlined. One concrete approach that I find useful is to use slide software (PowerPoint, etc.) and put one variant on each slide along with a list of pros/cons. For example, here are the slides that I prepared for an early version of PEP 695.
• The concepts of KeyType, ValueType, and Map — along with some of the other concepts we've been discussing in the context of PEP 705 like ReadOnly — have strong parallels in TypeScript. If you haven't already done so, I encourage you get a good understanding of the Keyof Type Operator, the Typeof Type Operator, Indexed Access Types, Mapped Types, and perhaps even Conditional Types. These are all powerful concepts that have been designed to compose well with each other. We shouldn't reinvent the wheel if we don't have to.
• As you think through your design, consider how the concepts will compose with other features in the Python type system including unions, generics, type param default values (PEP 696), etc.

Here are a few specific reactions to the initial proposal:

• Like Jelle, I find that the magic associated with "both KeyType and ValueType must appear in the same signature" is problematic. That will be technically limiting, won't compose well with other typing features, and will be confusing for users. I encourage you to look for ways to eliminate this magic.
• The limitation of "The first argument to Map has to be a generic type with exactly one TypeVar slot" is also problematic. It's very limiting and won't compose well with other typing features like PEP 696. Here again, I encourage you to explore ways to eliminate this limitation.
• I don't think there's a need for a separate KeyTypes and ValueTypes (plural). The singular forms need to handle unions. TypeScript, for example, doesn't have a separate keyof or typeof operator to refer to "all of the keys" or "all of the values".

I spent some time this morning playing with some variant ideas. Here are some of my musings. Note that my example code builds on PEP 695 & 696 syntax and the experimental inlined TypedDict syntax discussed in this typing-sig thread.

Exploration 1: How can we eliminate the magic of automatically associating KeyType and ValueType? How can we make this more explicit rather than implicit? I think we need to allow ValueType to accept a type argument that specifies which key types it applies to.

Note: I switch terminology from KeyType to KeyOf, which I find more natural to read, but we shouldn't get hung up on names at this stage of the design process.

# KeyOf is a special form that represents a string literal type that is
# parameterized by a TypedDict type. It represents a key in the TypedDict.
class KeyOf[TD: TypedDict](LiteralString): ...

# ValueOf is a special form that represents the value type associated with
# a key in a TypedDict.
class ValueOf[TD: TypedDict, K: LiteralString]: ...

# Examples:
type TD1 = dict[{"a": int, "b": str}]
KeyOf[TD1] # Evaluates to Literal["a", "b"]
ValueOf[TD1, Literal["a"]] # Evaluates to int
ValueOf[TD1, Literal["a", "b"]] # Evaluates to int | str
ValueOf[TD1, KeyOf[TD1]] # Evaluates to int | str

Exploration 2: How could this be used in a generic function? A generic method?

# Generic function
def get[T: TypedDict, K: KeyOf[T]](d: T, k: K) -> ValueOf[T, K]:
    return d[k]

# Usage:
td1: TD1 = {"a": 1, "b": "hi"}
get(td1, "a") # Evaluates to int

k1: Literal["a", "b"]
get(td1, k1) # Evaluates to int | str

# Generic method that references a class-scoped TypeVar
class C[TD: TypedDict]:
    def __init__(self, **kwargs: Unpack[TD]):
        self.vals = kwargs 

    def __setitem__[K: KeyOf[TD]](self, index: K, value: ValueOf[TD, K]) -> None:
        self.vals[index] = value

type MyTypes = dict[{"foo": int, "bar": str, "baz": NotRequired[bool]}]

c = C[MyTypes](**{"foo": 0, "bar": ""})

Exploration 3: How could "mapping" work such that we don't have a "single type param" limitation?

# Variant 1: Use a type variable to represent the key type:
type Immutable[TD: TypedDict, K: KeyOf[TD] = KeyOf[TD]] = dict[K, ReadOnly[TD[K]]]

# Variant 2: A slightly different syntax:
type Immutable[TD: TypedDict, K: KeyOf[TD] = KeyOf[TD]] = dict[{K: ReadOnly[TD[K]]}]

# Variant 3: Using a dictionary comprehension in the type expression:
type Immutable[TD: TypedDict] = dict[{K: ReadOnly[ValueOf[TD, K]] for K in KeyOf[TD]}]

Of these, I prefer variant 3. While it feels a bit weird to use a comprehension in a type expression, I think it's pretty readable and makes good use of existing Python syntax and concepts. Variant 3 is also much more flexible.

type Partial[TD: TypedDict] = dict[{K: NotRequired[ValueOf[TD, K]] for K in KeyOf[TD]}]

type MakeList[TD: TypedDict] = dict[{K: list[ValueOf[TD, K]] for K in KeyOf[TD]}]

type MakeDict[TD: TypedDict] = dict[{K: dict[str, ValueOf[TD, K]] for K in KeyOf[TD]}]

Exploration 4: Could this mapping proposal extend to TypeVarTuple mapping?

def product[*Ts](*it: *tuple[Iterable[T] for T in Ts]) -> Iterable[tuple[*Ts]]: ...

You asked "What if I want to express that my function takes a list of keys and a list of values, but they don't have to match?"? That's easy with this proposal.

type MyDict = dict[{"a": int, "b": str}]

# Accepts any value of any key defined in MyDict
def f1(x: ValueOf[MyDict, KeyOf[MyDict]]) -> None:
    pass

f1(3) # OK
f1(3.0) # Error

# Accepts any key and any (unrelated) value
def f2(keys: list[KeyOf[MyDict]], values: list[ValueOf[MyDict, KeyOf[MyDict]]]) -> None:
    pass

f2(["a"], [3]) # OK
f2(["a", "b"], [""]) # OK
f2(["a", "c"], []) # Error
f2([], [3.0]) # Error

# Accepts a list of any keys and a list of related values
def f3[K: KeyOf[MyDict]](keys: list[K], values: list[ValueOf[MyDict, K]]) -> None:
    pass

f3(["a"], [3]) # OK
f3(["a", "b"], [3, ""]) # OK
f3(["a"], [""]) # Error

I hope that's useful. Please keep moving forward on this proposal! I think it has significant value. If/when we start to converge on a proposal, I can implement pieces of it in pyright. I find this useful for playing around with the idea in code before the spec gets cast in concrete.

[tmke8]

I looked at TypeScript and Flow and I agree that their way of doing it (which is also what you're proposing) is better. I updated the main text to reflect this. I went with the names from Flow (KeyType and ElementType) instead of KeyOf and ValueOf but I'm not at all attached to these names.

One thing that would be nice would be to support Required/NotRequired and ReadOnly (if the corresponding PEP is accepted) somehow. So, if we have this:

from typing import ElementType, KeyType, NotRequired, TypedDict

Movie = TypedDict("Movie", {"name": str, "year": NotRequired[int]})

def get[K: KeyType[Movie]](key: K) -> ElementType[Movie, K]: ...

then what is this going to do:

get("year")

?

[erictraut]

what is this going to do?

Good question. I think there are two options:

1. It evaluates to int
2. It evaluates to int | None

Option 2 is comparable to what TypeScript does in this situation (except that it uses undefined rather than None), but I don't know if that's appropriate for Python.

[tmke8]

Maybe we could use NoReturn/Never for this... something like, if "not_required" is a field in TD that is not required, then if ElementType[TD, Literal["not_required"]] appears as a function return type, it is Never. (As a function argument type, it's fine because NotRequired fields are always settable.) Though that means there is no way to retrieve this value even if it exists...

[JukkaL]

I wonder if this could be generalized to attributes of non-TypedDict types. For example, we could perhaps use something similar to add precise types to __getattr__ methods of proxy objects. If this seems plausible, I think it makes sense to keep this related use case in mind as a possible future extension when designing the feature, and to discuss it briefly in the proposal. I remember this being discussed some years ago, but I don't remember where (might have been offline). Perhaps @ilevkivskyi remembers it.

[JelleZijlstra]

I think Ivan presented a proposal like this at a typing meetup in the Bay Area. He called it "KeyType" if I recall correctly.

[ilevkivskyi]

Yes, I was thinking about this a while ago. I proposed something like this in the context of precise typing of Python numeric stack. There are two aspects there that need new dependent-type-like features to have precise typing:

• Dense data, such as e.g. NumPy's ndarray, needs an N = IntVar("N") (integer generics), i.e. a type variable that can take values of literal integers, and supports basic arithmetic ops, such as N + M (e.g. for vstack and hstack), N * M (e.g. for kron), and N // M (for slicing).
• Sparse data, such as e.g. pandas' DataFrame, needs a K = KeyType("K") (string generics), i.e. a type variable that can take values of literal strings. These variables need to support basically what mapped types in TypeScript support.

When thinking about the second one, I indeed noticed that it will be useful for various things beyond numeric stack, in particular for ORMs and proxy types. Note that for the latter we will need to support a syntax like T.K where T is a type variable, to express something like this:

class OptProxy(Generic[T]):
    def __init__(self, target: T) -> None:
        self.target = target
    def __getattr__(self, attr: K) -> Optional[T.K]:
        return getattr(self.target, attr, None)

[Gobot1234]

PEP 728 seems very related to all of this with its Fields and FieldNames special forms python/peps#3326

[ChuckJonas]

Curious if there has been any recent movement on this?

We have a pattern where we need to initialize a dict (or dataclass if possible) from a dict where the key matches the target structure and the value: V | Callable[[],V]:

class Fooy(TypeDict):
    some_str: string
    some_str2: string
    some_num: number
    some_bool: boolean

# this datastructure will be "reduced" to generated at instance of `Fooy`
init = {
   "some_str":  'from-lit',
   "some_str2": lambda: 'from-fn',
   "some_num":  lambda: 42,
   "some_bool": lambda: True
}

It's very important to have type safety, but currently doesn't seem possible in python.

In typescript this could be done very easily:

type Fooy = {
    someStr: string;
    someStr2: string;
    someNum: number;
    someBool: boolean;
}

type InitFunctions<T> = {[key in keyof T]: (()=> T[key]) | T[key]}

const init: InitFunctions<Fooy>  = {
  someStr: 'from-lit',
  someStr2: () => 'from-fn',
  someNum: () => 42,
  someBool: () => true
}

playground link

[erictraut]

Curious if there has been any recent movement on this?

https://discuss.python.org/t/introduce-partial-for-typeddict/45176/63

[Viicos]

Coming from this discussion, I wanted to discuss some experiments I did (mainly for Map). As Eric mentioned, it is important to understand how these concepts compose with other typing features. I'll also add that runtime support is an important aspect to take into account: will it require new syntax to be introduced (how easy will it be to backport the feature)? Will it require extra work for runtime typechecking libraries to understand the constructs?

There seem to be two ideas emerging from this thread.

Introducing a Map construct

Also mentioned as a solution for partial typed dictionaries.

Map would be a special form taking two arguments: an operator and a type. Depending on the type, a specific set of operators are allowed. Only typed dictionaries can be allowed as types at first, but this can easily be extended in the future. Similarly to the Annotated special form, the type can be parametrized (but not the operator (?)), so that the following works:

from typing import Annotated, NotRequired, TypeAlias

Partial: TypeAlias = Map[NotRequired, T]

PartialSomeTD = Partial[SomeTD]

At runtime, we could do the following (but this needs a better specification):

If the type is a type variable, Map[..., T] is kept as an instance of _MapAlias (see implementation below). Whenever it is fully parametrized (e.g. Map[..., SomeTD], it evaluates to a new TypedDict class). Alternatively, it is also kept as an instance of _MapAlias, but we provide an API to help runtime tools getting the actual type (see implementation below).

Type checkers will need to special case the meaning of each type and each operator. For typed dictionaries, we allow Required, NotRequired and ReadOnly to be used as operators, and the meaning is "apply operator[...] to every field annotation of the typed dictionary".

This Map special form can easily be extended for other constructs:

• for type variable tuples, Map[list, TS] means "apply list[...] to every element of TS". (the allowed operators would have to be properly specified: dict doesn't make sense, what about generics with constrained/bound type variables, e.g. os.PathLike?).
• dataclasses or classes having the @dataclass_transform() decorator applied.

Benefits/drawbacks

Benefits

• The runtime implementation would be less of an issue. Runtime type checkers could easily inspect the constructs (see implementation below).
• Can easily be extended for dataclasses and other constructs.

Drawbacks

• Type checkers need to special case every type and operator. This gives less freedom to end users to implement their own constructs.
• Less powerful: While it could make sense for Map[list, SomeTD] (and other generic types with a single type variable) to mean "apply list[...] to every field annotation of SomeTD", it doesn't for Map[dict, SomeTD] (and other generic types with more than one type variable).
• Does not fulfill use cases such as def get[T: TypedDict, K: KeyOf[T]](d: T, k: K) -> ValueOf[T, K]: ..., Pick, Omit.

Introducing the concept of KeyOf and typed dictionaries comprehension

Eric provided an example of such a typed dictionary comprehension:

type Immutable[TD: TypedDict] = dict[{K: ReadOnly[ValueOf[TD, K]] for K in KeyOf[TD]}]

My main concern here (apart from the inlined typed dict discussion on the mypy issue, we could imagine dict[{...}] is a construct returning a new TypedDict class object) is how this evaluates at runtime. Ideally, {K: ReadOnly[ValueOf[TD, K]] for K in KeyOf[TD]} would evaluate to {"a": ReadOnly[int]} (if TD were to be defined with a single a: int field).

The runtime value of KeyOf[TD] would also have to be properly specified. Is it returning Literal["a"]?

Taking another example which should be valid:

type TDWithA = dict[{K: ValueOf[TD, K] for K in Literal['a']}]

This would require making Literal iterable in some way ¹.

If a specific typed dictionary comprehension syntax was introduced (I don't have any ideas on how it could look yet, if it should use existing Python syntax or introduce a new one -- like PEP 646 did), the runtime behavior would be less of an issue. KeyOf and ValueOf could just be special forms that do nothing at runtime. It would also make it possible to be lazily evaluated/parametrized (currently, the Immutable example eagerly evaluates, but TD is a still a type variable that needs to be replaced at some point).

If such as new syntax is introduced, might be good getting inspiration from TS mapped types as mentioned by Eric.

I have to put a lot more thinking into this, there're many ways this could be implemented and it is challenging as many use cases could be fulfilled by this, beyond typed dictionaries.

Benefits/drawbacks

Benefits

• Once type checkers understand this typed dictionary comprehension and how KeyOf/ValueOf plays together, no need to introduce special casing.
• Fulfills use cases such as def get[T: TypedDict, K: KeyOf[T]](d: T, k: K) -> ValueOf[T, K]: ..., Pick, Omit. Is generally more powerful.
• As mentioned in this thread, KeyOf/ValueOf could even be extended for proxy types.

Drawbacks

• Cannot really be extended for dataclasses and other constructs (but for instance, supporting Pick/Omit for dataclasses could be done in a different way, leveraging Annotated, etc).
• The runtime implementation is more of an issue. Runtime type checkers could have trouble inspecting the constructs.

Draft implementation of Map

Although jumping into the implementation isn't the priority, this is still useful to show how it will work with runtime type checkers:

from typing import _GenericAlias

class _MapAlias(_GenericAlias, _root=True):
    # Probably need to implement more things on this class:
    # `copy_with`, etc
    def __init__(self, operator, type):
        super().__init__(type, type, name="Map")
        self.__operator__ = operator

    def __repr__(self):
        return "typing.Map[{}, {}]".format(
            _type_repr(self.__operator__),
            _type_repr(self.__origin__),
        )

    # The runtime API we talked above:
    def apply(self):
        return self.__origin__.__apply_map__(self.__operator__)

class Map:
    __slots__ = ()

    def __new__(cls, *args, **kwargs):
        raise TypeError("Type Map cannot be instantiated.")

    def __class_getitem__(cls, params):
        if not isinstance(params, tuple):
            params = (params,)
        return cls._class_getitem_inner(cls, *params)

    def _class_getitem_inner(cls, *params):
        if len(params) < 2:
            raise TypeError("Map[...] should be used "
                            "with at least two arguments.")
        return _MapAlias(*params)

Usage:

class TD(TypedDict):
    a: int

T = TypeVar("T")

Partial = Map[NotRequired, T]
#> typing.Map[NotRequired, ~T]
Partial[TD]
# Either:
#> typing.Map[NotRequired, TD]
# or a new TypedDict class:
# TypedDict("TBD", {"a": NotRequired[int]})

If Partial[TD] still returns an instance of _MapAlias, then runtime type checkers could call the apply method. Every type supporting Map (typed dictionaries here, but as said above this could apply to other things as well) should define a __apply_map__(self, operator) method, returning what would be the result of the application of the operator to the type. In our case, TypedDict("TBD", {"a": NotRequired[int]}) would be returned.

Whether parametrizing Partial with a concrete typed dictionary should return a new typed dictionary class or still evaluate to _MapAlias depends on if we want to support subclassing:

class SubTD(Map[NotRequired, TD]):
    b: str

Well, the _TypedDictMeta metaclass could manually call apply for every base, but if Map were to be used for dataclasses or dataclass-like libraries, this is different.

Footnotes

1. It already is, but not how we expect it to be. ↩