When using Class as a Decorator, on decorated Instance Methods or Class Methods, MyPy reports 'Missing positional argument "self" [or "cls"] in call to "__call__" of "Decorator"'. Type Checks Okay for Static Methods and Normal Functions.

[mikelane]

Bug Report

I'm trying to use a class as a decorator to inject auth data into function calls, but I'm having a tough time getting Mypy to work correctly. Everything works correctly for a non-class method or a staticmethod, but the type checking fails for an instance method or a classmethod. The failure is Missing positional argument "self" in call to "call" of "Auth" [call-arg] for instance methods and Missing positional argument "cls" in call to "call" of "Auth" [call-arg]. These failures happen even when the UserId is passed like it should be.

I guess the implicit self and cls values aren't being recognized by mypy as being passed to the Auth.call method when the functions are called in their normal way. Note in the examples When I explicitly add a positional parameter as the first parameter to the instance method and to the classmethod which is the instance or class respectively, mypy stops complaining about the missing positional parameter and instead checks the type of the passed parameter. But obviously forcing users to call python methods in some weird way just to satisfy mypy isn't going to fly.

To Reproduce

1. Create a decorator class as shown in the code below
2. Decorate an instance method or a classmethod of another class with the decorator
3. Note the Erroneous type errors about the missing positional argument "self" for the decorated instance method or "cls" for the decorated class method.
4. Note that a decorated static method or regular function will type check the parameters normally.

import functools 
from typing import (
    Callable,
    Generic,
    NewType,
    TypeVar,
    ParamSpec,
)

UserId = NewType('UserId', str)
T = TypeVar('T')
P = ParamSpec('P')


class Auth(Generic[P, T]):
    def __init__(self, wrapped: Callable[P, T]) -> None:
        functools.update_wrapper(self, wrapped)
        self.wrapped = wrapped
        self.token = self.login()
    
    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> T:
        return self.wrapped(*args, **kwargs)
        
    def login(self) -> dict[str, str]:
        return {'accessToken': 'wow', 'tokenType': 'Bearer'}
        

def with_auth(wrapped: Callable[P, T]) -> Auth[P, T]:
    """This method is an experiment to see if I could hack around this issue by
    doing some double-wrapping shenanigans. Didn't work. Examples of using
    the Auth decorator and this Auth decorator decorator are shown below.
    """
    @Auth
    def decorated(*args: P.args, **kwargs: P.kwargs) -> T:
        return wrapped(*args, **kwargs)
    return decorated

class Client:
    @with_auth
    def one(self, user_id: UserId) -> str:
        return f'Making call with token {self.one.token}'
        
    @classmethod
    @Auth
    def two(cls, user_id: UserId) -> str:
        return f'Making call with token {cls.two.token}'
        
    @staticmethod
    @Auth
    def three(user_id: UserId) -> str:
        return f'Making call with token {Client.three.token}'

@Auth
def four(user_id: UserId) -> str:
    return f'Making call with token {four.token}'
    
@with_auth
def five(user_id: UserId) -> str:
    return f'Making call with token {five.token}'

# Sanity check. Succeeds as expected
UserId('asdf')
# Sanity check. Fails Successfully: Argument 1 to "UserId" has incompatible type "int"; expected "str"  [arg-type]
UserId(42)

client = Client()

'''
This should type check without error but doesn't: Missing positional 
argument "self" in call to "__call__" of "Auth"  [call-arg]
'''
client.one(user_id=UserId('asdf'))

'''
However, these calls fail successfully: Argument "user_id" to "__call__" of "Auth" has 
incompatible type "str"; expected "UserId". But note that I had to put `client` or `Client()`
in as the first positional argument. I shouldn't have to do that to get type checks to pass.
This is the bug, I think.
'''
client.one(client, user_id=UserId('asdf'))
Client.one(Client(), user_id=UserId('asdf'))

'''
This gives the wrong error. It fails with Missing positional argument "self" in call 
to "__call__" of "Auth"  [call-arg] instead of an error about "str" and "UserId" being incompatible types.
'''
client.one(user_id='asdf')

'''
These fail for the correct reason, but again note that I added the positional argument `client` and `Client()`
'''
client.one(client, user_id='asdf')
Client.one(Client(), user_id='asdf')

'''
Fails for the wrong reason: Missing positional argument "self" in call to "__call__" of "Auth"  [call-arg]
'''
client.one(user_id=42)

'''
These give the right errors, but again I had to add the positional arguments.
'''
client.one(client, user_id=42)
Client.one(Client(), user_id=42)

'''
More of the same examples as above, this time with a class method.
'''
# Should succeed but doesn't: Missing positional argument "cls" in call to "__call__" of "Auth"  [call-arg]
Client.two(user_id=UserId('asdf'))
# Fails for the wrong reason: Missing positional argument "cls" in call to "__call__" of "Auth"  [call-arg]
Client.two(user_id='asdf')
# Fails for the wrong reason: Missing positional argument "cls" in call to "__call__" of "Auth"  [call-arg]
Client.two(user_id=42)

'''
The static method works like I'd expect it to work.
'''
# Succeeds as expected.
Client.three(user_id=UserId('asdf'))
# Fails Successfully: Argument "user_id" to "__call__" of "Auth" has incompatible type "str"; expected "UserId"  [arg-type]
Client.three(user_id='asdf')
# Fails Successfully: Argument "user_id" to "__call__" of "Auth" has incompatible type "int"; expected "UserId"  [arg-type]
Client.three(user_id=42)

'''
A regular method works fine, too.
'''
# Succeeds as expected.
four(user_id=UserId('asdf'))
# Fails Successfully: Argument "user_id" to "__call__" of "Auth" has incompatible type "str"; expected "UserId"  [arg-type]
four(user_id='asdf')
# Fails Successfully: Argument "user_id" to "__call__" of "Auth" has incompatible type "int"; expected "UserId"  [arg-type]
four(user_id=42)

'''
Also the decorator decorator experiment works fine with normal functions, too.
'''
# Succeeds as expected
five(user_id=UserId('asdf'))
# Fails Successfully: Argument "user_id" to "__call__" of "Auth" has incompatible type "str"; expected "UserId"  [arg-type]
five(user_id='asdf')
# Fails Successfully: Argument "user_id" to "__call__" of "Auth" has incompatible type "int"; expected "UserId"  [arg-type]
five(user_id=42)

Expected Behavior

I expected calls to all decorated methods to type check the parameters that are used in the normal calls to the methods.

Actual Behavior

I found that calls like the following to instance methods:

client = Client()  # Assuming Client is a class with decorated methods
user = auth.get_user(user_id=UserId('asdf1234'))  # Should type check okay, but actually says that `self` is missing
user = Auth.get_other_data(user_id=UserId('asdf1234')) # Should type check okay, but acutally says that 'cls' is missing

Your Environment

• Mypy version used: 0.971
• Mypy command-line flags: Either no flags or --strict
• Mypy configuration options from mypy.ini (and other config files): Used mypy playground with no flags and then strict and local with no flags and then strict.
• Python version used: 3.10, 3.11
• Operating system and version: Mypy playground and MacOS 12.4

[mikelane]

gist: https://gist.github.com/8926e3364dfd4edad6e5303839decbfb
playground: https://mypy-play.net/?mypy=latest&python=3.11&flags=strict&gist=8926e3364dfd4edad6e5303839decbfb

[erictraut]

This is (unfortunately) how ParamSpec works when used for a decorator that is applied to a method. The self or cls parameter is captured as part of the ParamSpec. For details, refer to PEP 612.

One (admittedly ugly) workaround is to define two different versions of the decorator — one for methods and one for normal functions. The version for methods needs to use a Concatenate to strip away the self or cls parameter from the captured signature.

def with_auth_method(wrapped: Callable[Concatenate[Any, P], T]) -> Auth[P, T]: ...

[mikelane]

@erictraut, thanks for the info. Is there a way to do what I'm trying to do without using ParamSpec? Maybe with Protocols?

[erictraut]

You could use a TypeVar bound to a callable. It's not ideal because it involves a cast, but I think all of the solutions here have tradeoffs.

C = TypeVar("C", bound=Callable[..., Any])

def with_auth(wrapped: C) -> C:
    @Auth
    def decorated(*args: Any, **kwargs: Any) -> C:
        return wrapped(*args, **kwargs)

    return cast(C, decorated)

[mikelane]

Hmm. With that, the decorator removes the ability to type check decorated methods. :-\

I changed it up a bit (so I'm not doing that decorating the decorator bit), but it's got the same functionality as your example. But when I run this through mypy in the mypy playground with --strict and all the other errors and warnings enabled (except for --disallow-any-expr and --disallow-any-explicit), there is only

import functools 
from typing import (
    cast,
    Any,
    Callable,
    Generic,
    NewType,
    TypeVar,
)

UserId = NewType('UserId', str)
F = TypeVar('F', bound=Callable[..., Any])


class Auth:
    def login(self) -> dict[str, str]:
        return {'accessToken': 'wow', 'tokenType': 'Bearer'}


class WithAuth(Generic[F]):
    def __init__(self, wrapped: F) -> None:
        functools.update_wrapper(self, wrapped)
        self.wrapped = wrapped
        self.auth = Auth()
    
    def __call__(self, *args: Any, **kwargs: Any) -> Any:
        self.token = self.auth.login()
        return cast(F, self.wrapped(*args, **kwargs))
        

class Client:
    @WithAuth
    def one(self, user_id: UserId) -> str:
        return f'Making call with token {self.one.token}'
        
    @classmethod
    @WithAuth
    def two(cls, user_id: UserId) -> str:
        return f'Making call with token {cls.two.token}'
        
    @staticmethod
    @WithAuth
    def three(user_id: UserId) -> str:
        return f'Making call with token {Client.three.token}'

@WithAuth
def four(user_id: UserId) -> str:
    return f'Making call with token {four.token}'
    
@WithAuth
def five(user_id: UserId) -> str:
    return f'Making call with token {five.token}'

# Succeeds as expected.
UserId('asdf')
# Fails Successfully: Argument 1 to "UserId" has incompatible type "int"; expected "str"  [arg-type]
UserId(42)

client = Client()


# Every call here checks successfully with mypy, but only those which cast the str to a UserId should pass
client.one(user_id=UserId('asdf'))
client.one(user_id='asdf')
client.one(user_id=42)

Client.two(user_id=UserId('asdf'))
Client.two(user_id='asdf')
Client.two(user_id=42)

Client.three(user_id=UserId('asdf'))
Client.three(user_id='asdf')
Client.three(user_id=42)

four(user_id=UserId('asdf'))
four(user_id='asdf')
four(user_id=42)

five(user_id=UserId('asdf'))
five(user_id='asdf')
five(user_id=42)

[bjoernpollex-sc]

So the example here is a bit complex, but I think I've run into the same issue, with a simpler example:

from typing import Protocol, ParamSpec

P = ParamSpec('P')


class Thing(Protocol[P]):
    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> str:
        ...


def decorate(f: Thing[P]) -> Thing[P]:
    return f


class Foo:
    @decorate
    def bar(self, x: int) -> str:
        return str(x)


reveal_type(Foo().bar)
Foo().bar(5)

Here, mypy reports the following:

error: Missing positional argument "x" in call to "__call__" of "Thing"  [call-arg]
error: Argument 1 to "__call__" of "Thing" has incompatible type "int"; expected "Foo"  [arg-type]

@erictraut Above you mention that "This is (unfortunately) how ParamSpec works when used for a decorator that is applied to a method." - however, when I replace the Protocol with a Callable in the example above, this type-checks just fine, so it seems there is a way to handle self?

When I make Thing a generic class (so class Thing(Generic[P])) I get the same behavior as with Protocol. I wonder if there is a viable workaround for this? Ultimately what I'm trying to achive is to write type-hints for a decorator that adds attributes to the decorated function (similar to what's discussed in #2087), but I want it to work for methods too.

[erictraut]

@bjoernpollex-sc, Callable[P, str] is different from Thing[P]. Callable retains implicit attributes about the captured captured function (such as whether it's an unbound instance or class method, etc.). When this function is captured by some other class parameterized by a ParamSpec, these attributes are lost. There's no way to know whether these implicit attributes should be applied when the ParamSpec is used, such as in the Thing.__call__ method.

If you use Callable directly (as opposed to using a callback protocol Thing), it will type check without errors.

def decorate(f: Callable[P, str]) -> Callable[P, str]:
    return f

[bjoernpollex-sc]

@erictraut Thanks for the explanation. Do you know if there is any work in progress to adress this? Specifically, what is the intended way to type-hint decorators that add attributes to the decorated functions?

[tombulled]

I'm encountering this exact same issue, and also getting an "Incompatible types in assignment" error when trying to instantiate a class that should conform to my protocol. Here's a simple reproduction of the issue:
app.py

from typing import Callable, Protocol, TypeVar
from typing_extensions import ParamSpec

PS = ParamSpec("PS")
RT = TypeVar("RT", covariant=True)


class MyCallable(Protocol[PS, RT]):
    def __call__(self, *args: PS.args, **kwargs: PS.kwargs) -> RT:
        ...


def decorate(func: Callable[PS, RT], /) -> MyCallable[PS, RT]:
    return func


class Animal(Protocol):
    @decorate
    def speak(self) -> None:
        ...


class Dog:
    def speak(self) -> None:
        print("Woof!")


dog: Animal = Dog()

dog.speak()

$ mypy app.py
app.py:28: error: Incompatible types in assignment (expression has type "Dog", variable has type "Animal")  [assignment]
app:28: note: Following member(s) of "Dog" have conflicts:
app:28: note:     speak: expected "MyCallable[[Animal], None]", got "Callable[[], None]"
app:30: error: Missing positional argument "self" in call to "__call__" of "MyCallable"  [call-arg]
Found 2 errors in 1 file (checked 1 source file)

I appear to have a very similar use-case to @bjoernpollex-sc, as I would like the MyCallable protocol to contain an attribute that will be added to the callable by decorate(...). As such, having decorate(...) return a Callable[PS, RT] is infeasible.

[erictraut]

I think mypy is working correctly here according to PEP 612. I don't see any actionable bugs in any of the above code samples. I think this issue can be closed.