Generic NewType?

[Daenyth]

I'd like to be able to write code like;

from typing import *
SortedList = NewType('SortedList', List)
A = TypeVar('A')
def my_sorted(items: List[A]) -> SortedList[A]:
    ...

Currently I get error: "SortedList" expects no type arguments, but 1 given

[gvanrossum]

Hm, that may actually be reasonable. After all NewType is an optimized version of subclassing that erases the distinction at runtime -- and generics are also erased at runtime, so that may be a reasonable match.

[ilevkivskyi]

NewType is an optimized version of subclassing

Because of this, at runtime SortedList is a function that returns its argument. Practically any way of making NewType subscriptable will make it much slower.

[Daenyth]

Ahh, I see. Could it not be done like:

class NewType(?):
    def __init__(self, typename, type):
        ?
    def __getitem__(self, key):
        (type things)
    def __call__(self, arg):
        return arg

Would that be such a large overhead?

It seems like potentially it could be made somewhat opt-in. By that I mean that if you do NewType('a', t) for a t that is not Generic, return the identity function as it already does, but in the case that t is generic, return the subscriptable object?

[Daenyth]

That would parallel fairly reasonably with expectations people may have from similar behavior in other languages. Value types in scala are usually erased, for example, except in some cases. Similarly for newtypes in haskell, as I understand it. And with traits in rust.

I don't think it's that bad of a drawback, considering that the drawback only occurs when supporting something that currently can't be done at all. As long as it's documented it seems reasonable, at least.

[ilevkivskyi]

@Daenyth

Would that be such a large overhead?

Approximately 30x slower:

>>> timeit('class C: ...', number=10000)
0.11837321519851685
>>> timeit('def f(x): return x', number=10000)
0.00439511239528656
>>> 

[ilevkivskyi]

@Daenyth

something that currently can't be done at all.

What about normal subclassing? Your example can be just this

class SortedList(List[T]):
    def __init__(self, lst: List[T]) -> None:
        ...

[Daenyth]

Well, "not at all" is definitely an exaggeration. The drawback to the wrapper class is that I either have to implement proxy methods for every single list interface or expose a .value accessor.

[carljm]

We have a slightly different use case for this. We want to be able to do:

T = TypeVar('T')

IGID = NewType('IGID', (int, Generic[T]))

user_id: IGID[User] = IGID(3)

Subclassing is not an option here due to the runtime overhead; these need to stay real ints at runtime.

Without the generic, we either lose the ability to distinguish different types of IGIDs in the type system, or we have to create a separate type (e.g. UserID = NewType(...)) for every object type with an IGID (and we have many).

This use case does require two new features: a) passing a tuple of types to NewType for multiple "inheritance", and b) supporting indexing the runtime NewType.

@Daenyth it's not a wrapper class, it's a subclass, and the list constructor accepts a list already, so add a super().__init__(lst) and the subclassing solution works without any extra proxy methods or accessors.

@ilevkivskyi

Approximately 30x slower:

The 30x overhead you timed would be paid only once per process, at NewType creation. The more likely critical cost is the one that you pay every time the type is used, which is more like 2-3x difference:

>>> timeit.timeit('c(1)', 'class C:\n    def __call__(self, arg): return arg\n\nc = C()') 
0.18850951734930277 
>>> timeit.timeit('f(1)', 'def f(arg): return arg')                                                                                         
0.08798552677035332 

That's still probably enough of a cost that we will just go with the "lots of separate types" solution for our case instead.

[stereobutter]

I'd also like to use generic NewType in a manner inspired by phantom types to make APIs/ libraries I write type safe without introducing a lot of actual subclasses (for that I would have to implement/ pass through lots of methods). Take for example:

from typing import NewType, List, NoReturn, TypeVar

ListOfInts = List[int]

NonEmptyListOfInts = NewType('NonEmptyListOfInts', ListOfInts)


def prove_sequence_of_ints_is_nonempty(seq: ListOfInts) -> NonEmptyListOfInts: 
    if len(seq) > 0:
        return NonEmptyListOfInts(seq)
    else:
        raise ValueError('Sequence is empty')


def foo(seq: NonEmptyListOfInts) -> NoReturn: pass


a = [1, 2, 3]

b = prove_sequence_of_ints_is_nonempty(a)

foo(a) # Argument 1 to "foo" has incompatible type "List[int]"; expected "NonEmptyListOfInts"
foo(b)

This mostly works as expected (except for methods mutating b like b.pop() not causing mypy downgrade b to List[int] like c = a + b # c has type List[int] would do) but becomes tedious quite quickly because one has to introduce lots of type aliases like ListOfInts due to NewType not supporting generics.

[adaamz]

No update on this?

I basically want to typehint multiprocessing queue which is probably impossible now?

from logging import Handler
from multiprocessing import Queue as MPQueue
from queue import Queue
from typing import Union


DEFAULT_LIMIT = 10000
QueueValueType = Handler
QueueType = Queue[QueueValueType]  # error here
ProcessQueueType = MPQueue[QueueValueType]  # or here
QueueTypes = Union[QueueType, ProcessQueueType]

local_queue: QueueType = Queue(maxsize=DEFAULT_LIMIT)
process_queue: ProcessQueueType = MPQueue(maxsize=DEFAULT_LIMIT)

is OK for mypy, but Python raises error

Traceback (most recent call last):
  File "main.py", line 9, in <module>
    QueueType = Queue[QueueValueType]
TypeError: 'type' object is not subscriptable

[jancespivo]

Hi @adaamz

QueueType: TypeAlias = "Queue[QueueValueType]"
ProcessQueueType: TypeAlias = "MPQueue[QueueValueType]"

might help. See https://peps.python.org/pep-0613/

[eltoder]

This issue was opened many years ago. In recent versions NewType is already a class, which makes generic NewType easy to support (at least on the runtime side). We need two bits of syntax: to declare generic parameters for a NewType and to pass arguments for them. I chose [] for both of them as the most intuitive, but there's some flexibility for the former. The examples from previous comments now look like this:

T = TypeVar('T')

SortedList = NewType[T]('SortedList', list[T])

def my_sorted(items: list[T]) -> SortedList[T]:
    return SortedList[T](sorted(items))


IGID = NewType[T]('IGID', int)

class User:
    pass

user_id: IGID[User] = IGID(3)

My proof-of-concept implementation is here. It can be copy-pasted into typing.py.

@ilevkivskyi @JelleZijlstra What do you think? Does this need a PEP or can this be submitted as a PR?

[ilevkivskyi]

Not that I am final authority on this kind of questions, but I think this should probably be a PEP. Also PoC implementation should include not just typing part but also implementation in one of static type checkers (e.g. in mypy).

[alexchandel]

This is necessary for newtyping existing instances of builtin Python types like dict_view into like a newtype of Set[T].

[olejorgenb]

I was thinking about the same thing (SortedList). A shame we can't express this yet(?). Though using if for a SortedList would not be very safe without somehow blocking mutation? But SortedTuple would work very well

[erictraut]

Support for a generic NewType would require a modification to the typing spec, so the mypy issue tracker probably isn't the best place to be discussing this. If you would like to propose a change or addition to the Python typing spec, the Python typing forum is a good place to start that discussion.

[ManiMozaffar]

Following the discussion here, I created a proposition in Python typing forum.
https://discuss.python.org/t/generic-newtype/61234