CI/TST: update pyarrow pin in windows-3.8 env / fix failing parquet test on windows

[jorisvandenbossche]

Closes #45344

[jorisvandenbossche]

So updating pyarrow from 1.0 to 2.0 seems to highlight some other bugs ..

(edit: the xfails for this have been updated in #45257 in the meantime)

[jbrockmendel]

So updating pyarrow from 1.0 to 2.0 seems to highlight some other bugs ..

Something like this seems to happen frequently. I've got an under-utilized ubuntu machine that i could set up with 'tox' to run the tests daily(ish) with a thorough(ish) matrix of pyarrow versions. Would that be useful? If this is windows-specific then never mind.

[jreback]

closing in favor of #45354 which is already merged & backported

[jorisvandenbossche]

closing in favor of #45354 which is already merged & backported

#45354 is only xfailing the test, while this PR is an initial attempt to actually fix #45344 (which is still open)

Something like this seems to happen frequently. I've got an under-utilized ubuntu machine that i could set up with 'tox' to run the tests daily(ish) with a thorough(ish) matrix of pyarrow versions. Would that be useful? If this is windows-specific then never mind.

I don't think it is windows-specific (it's a fsspec / pyarrow incompatible version combination). But I also don't think it's necessarily needed for you to set up separate testing against more versions of pyarrow (or at least it should not be needed, we should "simply" do this better on our CI). I would first try to diversify the version in our own CI matrix (eg we can already reduce the number of builds that pins against pyarrow=1.0.1).

[jreback]

this looks good thanks @jorisvandenbossche

[jreback]

@meeseeksdev backport 1.4.x

[lumberbot-app]

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

git checkout 1.4.x
git pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -m1 91f12c44d94b4d399e3f2321edb7c35c91998242

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #45364: CI/TST: update pyarrow pin in windows-3.8 env / fix failing parquet test on windows'

4. Push to a named branch:

git push YOURFORK 1.4.x:auto-backport-of-pr-45364-on-1.4.x

5. Create a PR against branch 1.4.x, I would have named this PR:

"Backport PR #45364 on branch 1.4.x (CI/TST: update pyarrow pin in windows-3.8 env / fix failing parquet test on windows)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

[lumberbot-app]

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

git checkout 1.4.x
git pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -m1 91f12c44d94b4d399e3f2321edb7c35c91998242

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #45364: CI/TST: update pyarrow pin in windows-3.8 env / fix failing parquet test on windows'

4. Push to a named branch:

git push YOURFORK 1.4.x:auto-backport-of-pr-45364-on-1.4.x

5. Create a PR against branch 1.4.x, I would have named this PR:

"Backport PR #45364 on branch 1.4.x (CI/TST: update pyarrow pin in windows-3.8 env / fix failing parquet test on windows)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

[jreback]

@jorisvandenbossche if you wouldn't mind pushing the manual backport here

[jorisvandenbossche]

-> #45450