support unicode whitespace in strip and split?

[stevengj]

Three of the functions in util.jl — split, lstrip, and rstrip — use

const _default_delims = [' ','\t','\n','\v','\f','\r']

as their list of "space" characters, which is obviously only the ASCII space characters. As discussed on the mailing list, this leads to unexpected behavior (contrary to the documentation) when splitting a string that uses other Unicode space characters.

Three options:

• Change the documentation of split and strip to say that they only handle ASCII spaces by default.
• Change _default_delims to filter(isspace, Char(0):Char(0x10FFFF)) or similar. Howeve, this will slow down these functions somewhat, because there are currently 23 Unicode space characters vs. only 6 ASCII space characters. (We could also use a Set, but I don't think that is much faster for such a small list.)
• Define a special immutable _Spaces; end collection type, and _default_delims = _Spaces(), where in(c::Char, ::_Spaces) = isspace(c) and we make sure isspace(c) is carefully optimized.

I'm inclined to favor the third option at the moment.

[stevengj]

I just checked, and isspace(c) is already 72% faster than c in _ascii_default_delims, while c in _ascii_default_delims is 70% faster than the Set version of c in _unicode_default_delims and 140% faster than the Array version of c in _ascii_default_delims.

[nalimilan]

I was thinking along the same lines. I have also found two other possibilities:

• Use the regex r"\s", which is very easy with the current code and should work fine. Likely not great for performance.
• Add split(s, p) taking a predicate, which would be exported for general use, and make split(s) = split(s, isspace). It should be easy to refactor the existing code so that split(s, c::Char) would call split(s, p).

I'd favour either your last solution, or my last solution (which are quite similar). Mine makes sense only if exporting split(s, p) would be useful.

[tkelman]

Wouldn't a predicate version make more sense with the predicate first? That would allow do notation.

[nalimilan]

@tkelman Yeah, probably.

[stevengj]

@nalimilan, I agree that we should have split(predicate::Function, s::AbstractString). However, we should also optimize the common case of splitting on a collection of characters, and especially on spaces, and since Function arguments aren't currently inlined etc. I think we should still implement specialized types for those cases. Maybe specialized callable types, similar to how reduce is optimized for + etc.

[nalimilan]

Making Spaces callable and passing it to split(p, s) sounds like a good solution indeed.

[stevengj]

(The regex test is 50x slower than isspace.)

[iagobaapellaniz]

I found that

julia> "\U2297" == "\otimes[TAB]"
true

while

julia> r"\U2297" == r"\otimes[TAB]"
false

julia> Regex("\U2297") == r"\otimes[TAB]"
true

Is this a bug? Should it consider that \U is special?
Should I open a new issue? Feel free to do that for me if that's the case...

[stevengj]

@i-apellaniz, no, it's not a bug (you can't use \U escapes in a regex AFAIK) and please don't post questions on a completely unrelated issue.

[iagobaapellaniz]

Sorry, I'll open a new issue, since I think this must be further investigated. Sorry again.

[laborg]

This has been fixed in #27309