Skip to content

General improvements to the tests and CI Environment #1461

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 20 commits into from
Sep 15, 2020
Merged

General improvements to the tests and CI Environment #1461

merged 20 commits into from
Sep 15, 2020

Conversation

Giotino
Copy link
Collaborator

@Giotino Giotino commented Sep 11, 2020

As can be seen here: https://ci.nodejs.org/job/citgm-smoker-nobuild/935/ the fact that the Got tests pass on "an ubuntu release" doesn't mean that it works everywhere.

I added some OS: Ubuntu Trusty (14.04), Ubuntu Xenial (16.04), Ubuntu Bionic (18.04), Ubuntu Focal (20.04), Windows 1809 (newer versions are not supported), macOS 10.14, macOS 10.15.

Whenever we decide to switch to Github Actions these are the available OS.

Virtual environment YAML workflow label
Windows Server 2019 windows-latest or windows-2019
Ubuntu 20.04 ubuntu-20.04
Ubuntu 18.04 ubuntu-latest or ubuntu-18.04
Ubuntu 16.04 ubuntu-16.04
macOS Catalina 10.15 macos-latest or macos-10.15

@Giotino Giotino marked this pull request as draft September 11, 2020 09:21
@Giotino
Copy link
Collaborator Author

Giotino commented Sep 11, 2020

With the latest commit (9fa1d1c) I copy-and-pasted (and ported to Typescript) https://github.com/lukechilds/create-test-server .
With my modifications it doesn't create a new HTTPS certificate each time (which takes time), in fact it doesn't even support HTTPS.

For tests regarding HTTPS createHttpsTestServer (already present in the repo) should be used, and it's already used by all the https.ts tests.
I had to disable 5 tests (4 marked as failing, 1 commented) because they need to be ported to the new HTTPS Server, I'm working on it.

Tests are now much faster and shouldn't timeout on slower systems.

How is this related to this PR?

Looking at https://ci.nodejs.org/job/citgm-smoker-nobuild/nodes=osx1014/935/testReport/junit/(root)/citgm/got_v11_6_1/ (macOS 10.14) the tests timed out, as you can see there are some tests which took up to a minute and a half to run.

It could be due to the "different" openssl present on macOS (might be a slower release), or to the fact that the hardware beneath the OS is a Mac mini (a common setup for a macOS server).

@Giotino Giotino changed the title More OS in Travis CI Build General improvements to the Test and CI Environment Sep 11, 2020
@Giotino
Copy link
Collaborator Author

Giotino commented Sep 11, 2020

It seems that Tavis CI Ubuntu Focal VM does not resolve IPv6 hostnames. I disabled that test for Focal on Travis CI.

process.env.TRAVIS_DIST !== 'focal'

@szmarczak
Copy link
Collaborator

  1. Can we use GitHub Actions instead of Travis?
  2. It'd be best if this PR was split up into two: 1) CI improvements, 2) test improvements.

@Giotino
Copy link
Collaborator Author

Giotino commented Sep 11, 2020

  1. Can we use GitHub Actions instead of Travis?

Yes, we are going have less platforms to run the tests, but that shouldn't be a problem.

  1. It'd be best if this PR was split up into two: 1) CI improvements, 2) test improvements.

I opened this PR to make the master branch able to run on CITGM.
The fixed tests and the speed-up due to the new HTTP Test Server were aiming to a successful run on CIRGM.
The increased CI targets were needed for us to look at issues on other platforms.


I would like to have this PR on master as soon as possible, then I'm going to open two new PRs with the aim of improving the Test Environment (as you can see the new modifications are a little messy, I would like to clean it up in the new PR) and the CI Environment (aka. migration/trial on Gitlab Action).

Issue pointed out by you, such as:

  • The meaning of true in withHttpsServer({}, true)
  • "Maybe let's do withHttpAndHttpsServer?"
  • Why clock?.runAll();
  • GitHub Actions

Are going to be addressed in the new PRs.

Let me know what do you think about this.
@sindresorhus @szmarczak

Of course the incredibly-overdue PR #1272 is going to be closed in favor of the newer one.

@sindresorhus
Copy link
Owner

I don't really see the point of testing on so many Linux combinations. It's unlikely to make a difference but will make builds slower. This is especially true for macOS. I think we should limit it to latest macOS and Windows, and also Ubuntu Bionic (18.04) and Ubuntu Focal (20.04).

@sindresorhus sindresorhus changed the title General improvements to the Test and CI Environment General improvements to the tests and CI Environment Sep 13, 2020
@sindresorhus sindresorhus marked this pull request as ready for review September 15, 2020 11:28
@sindresorhus sindresorhus merged commit 38ba09c into sindresorhus:master Sep 15, 2020
@sindresorhus
Copy link
Owner

Merging this as there was a request to get it merged fast. But please address the feedback given here in the follow-up PR.

@Giotino
Copy link
Collaborator Author

Giotino commented Sep 15, 2020

Merging this as there was a request to get it merged fast. But please address the feedback given here in the follow-up PR.

Sure, I'm already preparing it, a draft is coming soon.

@Giotino Giotino deleted the travis-multiple-os branch September 15, 2020 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants