CI: Inconsistencies in the CI tests

[datapythonista]

Not sure if anyone know exactly what we are testing, but after removing Python 3.5 and some other old stuff, I think we've got lots of builds testing almost the same things, and the lack of a clear structure is preventing us from finding inconsistencies easily.

Some things that seem wrong:

• I don't think we're running tests with network marker in any build (we run not slow and not network and slow)
• We just run the slow in allowed failures (not sure if that's intentional)
• py36_locale_slow is not actually running slow tests: https://github.com/pandas-dev/pandas/blob/master/ci/azure/posix.yml#L32
• We only test in Mac with 2 year old versions of numpy and dateutil (never with recent ones)
• In windows we also test only with old numpy versions 1.14 and 1.15, never recent ones

To fix those, what I think it'd make more sense is:

• Remove from the dependencies files the CI system (travis, azure) and the OS (windows...), I don't think those add value
• Get a dependencies file with all the dependencies without specifying versions (conda should give the latest) and test that on Linux, Windows and Mac
• Get a dependencies file with the minimum supported versions (similar to azure-36-minimum_versions.yaml, but including all dependencies, the current just got a small set). Test it in Linux (possibly in Windows and Mac if important, but I don't think so)
• Use the same dependencies files to test the slow and network tests
• For the locales, use couple of the previous builds
• For everything that we need to test that is not covered in the previous builds, create a dependencies file explaining exactly what we want to test with that (e.g. 32 bits, numpydev)

Not sure I'm missing something, but I think this should make things much simpler, test more things, and we can even save couple of builds (we have 12 now)

CC: @jreback @TomAugspurger

[jreback]

how about slow incremental changes
rather than sweeping proposals which tend to break things in the CI

[datapythonista]

Absolutely, that makes a lot of sense.

Are you ok with the general idea of the proposal, to move incrementally in that direction?

[alimcmaster1]

don't think we're running tests with network marker in any build (we run not slow and not network and > slow)

Are you sure? I thpught we run them in the py36_locale_slow_old_np build? ( I remember as them failing in this PR I worked on - but I might be wrong)

[datapythonista]

the pattern for that build is slow, if the test is also marked as slow, then we don't need to have not slow and not network everywhere

[datapythonista]

This is what we've got now:

Name	CI	OS	Python	NumPy	regular/slow	locale	services
azure-36-32bits	Azure	Linux	3.6	1.14	regular	en
azure-36-locale	Azure	Linux	3.6	1.15	regular	it
azure-36-locale_slow	Azure	Linux	3.6	1.14	slow	zh
azure-36-minimum_versions	Azure	Linux	3.6	1.13.3	regular	en
azure-37-locale	Azure	Linux	3.7	latest	regular	zh
azure-37-numpydev	Azure	Linux	3.7	dev	regular	en
Azure-macos-36	Azure	Mac	3.6	1.14	regular	en
Azure-windows-36	Azure	Windows	3.6	1.15	regular	en
Azure-windows-37	Azure	Windows	3.7	1.14	regular	en
travis-36-cov	Travis	Linux	3.6	1.15	regular	en	sql
travis-36-locale	Travis	Linux	3.6	latest	regular	zh	sql
travis-36-slow	Travis	Linux	3.6	latest	slow	en	sql
Travis-37	Travis	Linux	3.7	latest	regular	en
Travis-38	Travis	Linux	3.8	latest	regular	en

[alimcmaster1]

the pattern for that build is slow, if the test is also marked as slow, then we don't need to have not slow and not network everywhere

Ahh gotcha. Looks like we only have 1 test with the mark pytest.mark.network - let me know if there is a build you want me to add them to/separate build? - so we at least have them running :)

[mroeschke]

Summarizing the work of the linked PRs that can close this issue

• Linux, MacOS, Windows now share the same dependency file of unpinned dependencies
• Covers PY38, PY39, PY310
• Linux covers all tests (matrix over pytest markers: not single_cpu and single_cpu)
• Linux runs the "special" builds (downtream deps, min versions, locale, datamanager, numpy dev)

For further inconsistencies, let's open individual issues.