os.open doesn't accept Union[bytes, Text]

[euresti]

The following code:

from typing import Text, Union
import os

def test(path):
    # type: (Union[bytes, Text]) -> None
    f = os.open(path, os.O_RDONLY)

Passes with --py2 but fails in python3 as follows:

error: Type argument 1 of "open" has incompatible value "Union[bytes, str]"

@JukkaL

[gvanrossum]

FWIW if we change os.open() to take Union[str, bytes] the error goes away, but I'm sort of thinking that passing that union to something expecting AnyStr should work...

[JukkaL]

Yes, Union[str, bytes] should probably be sometimes valid when AnyStr is expected (but it isn't right now). Example:

def f(x: AnyStr) -> AnyStr:
    return x

def g(x: Union[str, bytes]) -> None:
    f(x)  # should be fine, return type Union[str, bytes]

However, List[Union[str, bytes]] should not be compatible with List[AnyStr] because the latter is uniform but the prior could have a mix of str and bytes. Also, this shouldn't be valid, following similar reasoning:

def f2(x: AnyStr, y: AnyStr) -> AnyStr:
    return x + y

def g2(x: Union[str, bytes], y: Union[str, bytes]) -> None:
    f2(x, y)  # error, could be a mix of str and bytes

Here's how we could support these (and other things not involving type variables, such as overloads):

• If caller has arguments with union types, expand all the combinations of the union types into non-union types and test the call with each of these expansions. The call is valid if all combinations type check okay.

The call to f2 in the second example would be checked 4 times:

• f2(<str>, <str>)
• f2(<str>, <bytes>)
• f2(<bytes>, <str>)
• f2(<bytes>, <bytes>)

The obvious problem with this is that this could be really slow if there are many union types in a call. An implementation could plausibly be clever and only expand those unions where it seems like a useful thing to do.

[gnprice]

I think the right fix to @euresti's original issue is that the type of os.open's parameter should not be AnyStr -- it should be Union[bytes, str]. In general, AnyStr doesn't provide any additional value when it appears only once in a type signature, and it introduces complexity because its semantics are somewhat complicated.

It's possible we should also handle the interaction of AnyStr with Union differently in general. I think that ought to be a separate issue if we want to pursue it further.

[gnprice]

Closing in favor of the typeshed issue -- @JukkaL , I haven't fully absorbed your comment, but if you think there's more to discuss please do make an issue for it.

Thanks for reporting this, David!

[gvanrossum]

This issue wasn't actually closed, and I don't think it should be just yet. I still think that a function with exactly one AnyStr parameter (not occurring in a container) should be supported -- as @matthiaskramm mentioned it is a pretty convenient notation. That said I agree we should just fix os.open() -- and all other os functions that have just one string argument and that don't return a string.

[gnprice]

I don't want to ban using AnyStr just once in a function signature, but I think the semantics as currently written in PEP 484 are pretty complicated (details in that typeshed thread). Do you think what we're currently doing differs from those semantics? Or is there another fix you think we can make here?

[JukkaL]

Increasing priority to normal since issues with union types are frequently causing trouble for users. This may also affect code that uses --strict-optional.

[ilevkivskyi]

Currently the original example passes in both Python 2 and Python 3 modes. So I am closing this since we have another issue for tracking union maths.