Broadcasting && with missings

[cstjean]

#39594 implemented .&& and .||, which is very cool, but unfortunately it does not support missing values on 1.7.

julia> @. [true, missing, true] && [true, false, false]
ERROR: TypeError: non-boolean (Missing) used in boolean context

I know that && technically does not work with missing either, but if the spirit of that PR is to solve #5187 by providing a & with better operator precedence, then missing support is natural...

@mbauman

[jakobnissen]

That would make .&& behave differently from simply broadcasting &&, which would be very confusing and might end up causing more problems than it solves. For example, suppose someone sees some code that does a .&& b and replaces it with a for loop, then suddenly find the code fails and can't understand why.

[cstjean]

It's already different.

julia> false .&& error("how come this is executed")
ERROR: how come this is executed
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] top-level scope
   @ REPL[9]:1

For that matter,

julia> missing .< 3 .< 5
missing

julia> missing < 3 < 5
ERROR: TypeError: non-boolean (Missing) used in boolean context
Stacktrace:
 [1] top-level scope
   @ REPL[11]:1

Fundamentally, broadcasting is a special operation, so there's bound to be semantic differences...

[oscardssmith]

those are some serious WATs.

[cstjean]

🤷‍♂️ Is that also WAT-worthy?

julia> true && missing
missing

julia> missing && true
ERROR: TypeError: non-boolean (Missing) used in boolean context

&& is a control flow operator. It can't support missing in first position because its semantic is to evaluate the right operand conditionally. .&& is a broadcasting operator that must evaluate both operands, and thus can support missing just fine. #5187 was an issue about pragmatic syntax:

But when behaving as their element-wise boolean counterparts to || and &&, it is surprising that A < 1 || A > 2 must be written differently if A is an array: (A .< 1) .| (A .> 2)

And likewise I would say:

When it comes to .|| and .&&, it is surprising and inconvenient that A .< 1 .|| B .> 2 must be written differently if A contains missing: (A .< 1) .| (B .> 2).

We use broadcasting on missings extensively, and the (a < b) & (c < d) issue was painful enough that years ago we implemented #39594 at the macro level.

If .&& doesn't work with missings, then it can't be used in packages whose users may have missing data. That's really too bad.

[pdeffebach]

I think current behavior is okay, though the below is probably a bug and should be handled.

julia> false .&& error("how come this is executed")
ERROR: how come this is executed
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] top-level scope
   @ REPL[9]:1

Let me plug my package MissingsAsFalse.jl as a way to help with this. Though I dont think it supports broadcasting and it would be a great PR.

[mbauman]

.&& is a broadcasting operator that must evaluate both operands, and thus can support missing just fine.

This is not true. Your example false .&& error("how come this is executed") throws an error because error itself is not broadcasted. Compare:

julia> false .&& error.("how come this is executed")
false

While the false .&& error("how come this is executed") case looks sketchy, it really cannot behave any differently than it does. It's syntactically the same as false .&& f(). Julia needs to broadcast && over the result of f(). It can't skip evaluating f() because — even though Julia could statically know that every element should be false — it doesn't know what shape it should be. E.g.,

julia> false .&& trues(1, 4)
1×4 BitMatrix:
 0  0  0  0

[cstjean]

Oh! Thank you for the explanation. I had misconceptions about what the broadcasting machinery was capable of, but it makes sense in retrospect. Doesn't that mean that julia> @. [false,false] ? [1,2] : [3,4] could be supported as well? It'd be interesting for some operations. At the moment, it's an error.

So... I'm guessing we should close this? missing .&& error.("a") can't decide whether error should be called, and thus must be an error in the general case?

[cstjean]

Still a bummer that #5187 is only solved for non-missing data.