Windows Support

[fkorotkov]

Are there any plans to get official Windows Server images? Preferably based of microsoft/windowsservercore image?

[sfackler]

We'd definitely like to have them - the only blocker is figuring out how to make them actually work. There's an attempt here for example: alexcrichton@494237c

[nelsonjchen]

Is there a list of unresolved issues with that attempt?

[nelsonjchen]

🤷‍♂️

I guess a big one is that the build tools are kinda launched in a forked process on that branch and don't specify --wait, causing a premature killing of the intermediate build step and no VS stuff being installed. I'm saying this in case anyone wants to go ahead of me and bang on it. This was something I noticed by looking at examples of Windows Containers in GitHub installing the VS Build Tools.

I have a branch in my fork that definitely hacky, still doesn't work, and still a WIP. Need to rebase and it's kinda annoying that generated "artifacts" are in this repo.

[nelsonjchen]

My branch works, I can do that nifty "Compile your app inside the Docker container" but with Windows thing with the whole --rm thing at the bottom of the README but I don't have it backported to the generate.sh stuff yet. I think I'll go check out how the python Docker image works since they have a windows target there and see what they do.

[sfackler]

Awesome!

I cargo-culted a bunch of the setup here from the python and golang images, so they'd be good places to look for Windows setup as well.

[nelsonjchen]

By the way, if anyone is developing this on Windows 10 and not Windows Server:

If you're on 10 and 1809, replace the docker.exe and dockerd.exe executables from the latest builds of docker from the tip. Microsoft recently sent in a PR and licensing approval to remove Docker's artificial disablement of --isolation=process on Windows client SKUs from the daemon. It's not in Docker Edge yet as of this comment. This means faster development without the Hyper-V VM and visibility with Procmon and so on. Only supported for dev/test.

[nelsonjchen]

I've decided to drop nanoserver from my efforts. It'll probably be great for those multi-stage builds where someone compiles something with Rust from the bigger windowsservercore-based images and copies it to nanoserver for running/runtime but it's probably inappropriate for a build tool. I mean, do you see an alpine image for rust? Though, I guess there's debian slim and a hypothetical alpine version would kinda be oddly supported in a weird way. But this is Microsoft country, and it takes years for them to trim more fat.

[sfackler]

I mean, do you see an alpine image for rust?

The only reason there's no alpine image is that there isn't an x86_64-unknown-linux-musl rustc build yet. #10

[nelsonjchen]

Oh! I didn't see that.

[nelsonjchen]

There are some straggling issues left I guess.

• The MSVC variant probably can't be redistributed. IANAL. Technically you need to have a license to Visual Studio to use the build tools distributions. I think the output from this for this project might just be a Dockerfile licensed users can use and no more. I guess a more freely redistributable version would have to be Mingw based.
• Unfortunately, there does not appear to be a way to pin a MSVC or build tools 2017 version. For reproducibility reasons, this is worrying. You can pin the latest patched minor versions of N-1 minor versions 2017 tools but you can't pin the latest version or any of the patch releases. This really isn't great for reproducibility of builds. It's a bit more worrying if the limit for what can be distributed for these kinds of users are a Dockerfile.

[nelsonjchen]

The GNU variant was predictably crazy easy to implement and definitely not encumbered.

[nelsonjchen]

I have to take a break from this for now. If someone wants to take a crack at getting this upstreamed, have at it. There are MSVC and GNU Dockerfiles in here:

https://github.com/nelsonjchen/docker-rust/tree/windows

I am not sure how to get this integrated in with that generate-stackbrew-library.sh. What's the play here? Run that in WSL with Docker for Windows configured in a mode that supports both Linux and Windows containers exposed to the Docker client inside WSL? I just don't know.

[sfackler]

That just generates the metadata blob that goes over to the docker-library/official-images repo - it doesn't need to run in the same OS as the docker image or anything.

[nelsonjchen]

Oops, I meant update.sh, it depends on something called bashbrew and I'm not familiar with that yet.

[sfackler]

Similarly, update.sh can create a Dockerfile for a Windows image without being run on Windows.

Bashbrew is here: https://github.com/docker-library/official-images/tree/master/bashbrew

[nelsonjchen]

Bashbrew explodes now in WSL with some nil dereference error. Not sure why. Anyway, update.sh still seems to work. I've updated it to handle some new axes.

• ltsc2016, aka latest of microsoft/windowsservercore.
• 1709
• 1803
• ltsc2019, aka "1809".

Along with the msvc and gnu axes. This makes 8 combinations.

Unfortunately, only the ltsc2016 targets build on Appveyor:

https://ci.appveyor.com/project/nelsonjchen/docker-rust/builds/20890075

For Windows containers where the kernel doesn't match the image, nested virtualization is required. These jobs fail to build because Appveyor does not have nested virtualization. Unlike Linux containers, where there's a Linus to scream "WE DO NOT BREAK USERSPACE! Seriously. How hard is this rule to understand? We particularly don't break user space with TOTAL CRAP.", this is not the case for Windows. The userspace is designed to match the kernel.

When the images match, Docker will run the container with process isolation much like on Linux systems. When they don't, Docker will launch a "nano" sized boot of a cut-down Windows VM with a matching kernel version. There's actually about a small but noticeable delay when Docker does this. On client Windows systems, this is the norm and is enforced since client Windows kernels can vary a lot. Only recently in a currently unreleased Docker version did some Microsoftie take off this limitation in a PR. Windows 10 Client 1809 and up can run images matching the kernel with process isolation, though it's branded as for dev and test only. While not currently available and somewhat useless, I added the 1709 and 1803 variants as a small test of building images for users who are running client Windows versions that don't happen to have a LTSC server match, like right now with whatever will come after 1809 and post Windows Server 2019.

Another issue is that I think Appveyor only provides building one job at a time. I could have sworn they used to provide more concurrency in the past for OSS projects. 🤷‍♂️ I might have been confusing it with some other CI service . Right now, these images take about 16 minutes for the GNU variant and 32 minutes for the MSVC variant. Pretend the nested virtualization wasn't an issue. To check all these, it would take about 192 minutes on Appveyor. This list will grow a bit as more 10-year supported LTSC releases of Windows happen.

I have a better, cheaper, and faster proposal. Could we use Azure Pipelines? They provide 10 concurrent Windows (or any OS including Mac or Linux) VMs for free for OSS projects. They currently do not support nested virtualization but this pull request from a Microsoftie is inside this repo they use for generating the images they are using for their service is switching the instance type they are using for building to something that supports nested virtualization. It's un-merged but it's safe to say they're thinking of introducing support for it. Barring that, Azure Pipelines is most likely to have available the in-between non-LTSC builds of Windows such as 1803 right now. Using their service to shotgun build and test all these windows images at once would be great.

[nelsonjchen]

We might have more combinations in the future as well. Like #14 which could double or triple the amount of images.

[sfackler]

I don't have strong preferences on the CI setup - but presumably the other official image repos that have Windows images should have something to base the work here off of?

[nelsonjchen]

Definitely, there's some things to learn from them.

Using the go and python Windows images as reference, I've found the following.

• Due to Appveyor limitations, only ltsc2016 is supported and tested. When the Python library's update.sh is run, it skips over 1709 when building the appveyor test matrix.
• Go is a bit more liberal with nanoserver images and tests 1803 in addition thanks to TravisCI's support for 1803-based images.
  • It actually builds on public CI systems against:
    • nanoserver:sac2016
    • nanoserver:1709
      • Not built in official Jenkins image builder. See below.
    • nanoserver:1803
      • Not built in official Jenkins image builder. See below.
    • windowsservercore:ltsc2016
    • windowsservercore:1709
    • windowsservercore:1803
  • They don't test 1709 variants as they aren't available to test with process isolation on any public cloud.
  • They test 1803 variants on TravisCI.
• Judging from the time it took to test a Python commit with three image targets, I think they only run one job at a time on Appveyor and aren't paying for more in the docker-library organization. The Go project's Travis Jobs seem to be nicely parallelized even with Windows.
  • Two CI systems for Go. A slow Windows CI system in Python and omitting of 1803. Ugh! I'm totally going to implement Azure Pipelines for the Windows builds into my branch.
• The official images workflow use Appveyor and TravisCI for public testing but not for deploy. Images are built by a Docker consultancy's Jenkins instance which are then pushed to become the official images.
  • Jenkins config: https://github.com/docker-library/oi-janky-groovy
  • The multiarch pipeline jobs are used to build the official images for uh ... multiarch: https://doi-janky.infosiftr.net/job/multiarch/
    • Go: https://doi-janky.infosiftr.net/job/multiarch/job/windows-amd64/job/golang/
      • nanoserver:sac2016
      • windowsservercore:ltsc2016
      • windowsservercore:1709
      • windowsservercore:1803
    • Python: https://doi-janky.infosiftr.net/job/multiarch/job/windows-amd64/job/python/
      • windowsservercore:ltsc2016
      • windowsservercore:1709
    • The Rust jobs for Linux and other non-Windows arches can be found in here too!
  • I don't know the process of getting into these pipelines. Obviously, I think we need to build the working images first.
• DockerFile Structure
  • Golang and Python
    • <Language Version>/<os>/<winvariant>-<winversion>
  • I like this 3 level max directory depth thing. Let's do that. But I'm not sure how to fit the whole "GNU" or "MSVC" stuff in here. We don't want to deploy the MSVC versions but they should exist here in the repo for reference.
    • Perhaps <Language Version>/<os>/<winvariant>-<winversion>-<rustbuild> is a good scheme?

---

In the meantime, I've also discovered that simply rebasing some of the GNU images atop of nanoserver images simply do not work. nanoserver:sac2016 fails because the powershell module to set TLS 1.2 compatibility simply doesn't exist. Later nanoserver images explode with failures due to some system calls that I guess rustup makes that simply don't work. Cool, nanoserver is definitely a no-go.

So here's my TODO list:

• Try the <Language Version>/<os>/<winvariant>-<winversion>-<rustbuild> scheme which is similar to the current structure in Go and Python
• Replace Appveyor and implement Azure Pipelines in update.sh with initial testing for the axes of windowsservercore:{ltsc2016,1803} and {gnu,msvc}.

Maybe after all this it might be PR ready. We'll see.

[nelsonjchen]

Alright! I got it refactored to that scheme.

I got a hardcoded azure-pipelines.yml. I'll need to make update.sh generate parts of it or some file it references for templating. I'm a little torn on making a template to be included or do that awk stuff.

https://dev.azure.com/nelsonjchen/docker-rust/_build/results?buildId=15

In the meantime, while hardcoding it, I think there's some issue with the windowsservercore_ltsc2016_msvc image. This isn't related to Rust and it's more likely to be something related to the process isolation mode, ltsc2016, azure, and the VS build tools. This builds fine in hyper-v isolation on my Windows 10 machine. Meanwhile, windowsservercore_ltsc2016_gnu, windowsservercore_1803_gnu, and windowsservercore_1803_msvc run just fine 😮. I'll go read some documentation and posts and see if windowsservercore_ltsc2016_msvc+process isolation + azure just isn't meant to be or something. It works on Appveyor.

[gilescope]

Just a quick check to see if there's been any movement on this front as time has gone by.

[IceCodeNew]

Have we made any progress on this issue these years?

[gilescope]

I think the main issue is no one seems to use windows containers. If they did there might be more movement, but the use case is lacking at the moment.

[yodaldevoid]

As someone who uses Windows containers, there's dozens of us! Dozens!

Jokes out of the way, for my job I am currently creating Windows containers for Rust. I would rather not have to make them myself, but seeing how long this issue has been around and how long the beta issue has been around, I don't think that will be changing any time soon.

[sfackler]

I mean, the only thing blocking this issue is someone getting a Dockerfile that works. If you have those, then feel free to open a PR.

[yodaldevoid]

I was under the impression that there was more needed than just some dockerfiles, but if that is truly all that's needed I'll submit a PR ASAP.

[lippertmarkus]

@yodaldevoid I see you made some progress in https://github.com/yodaldevoid/rust-windows. Can we expect a PR from you?

[yodaldevoid]

Welp, it seems ASAP was two months. Sorry about that, world's been a bit crazy.

Anywho, I've opened #71 with what I've got so far. I'll probably tomorrow to throw together an Azure, Appveyor, or GitHub Actions pipeline from other's previous work. I don't really have any personal stake in getting the GNU toolchains working, but maybe I'll get inspired.

[lippertmarkus]

Thanks a lot!

[lippertmarkus]

For anyone coming here who just wants Windows support to be able to use a compile stage in their Dockerfile for creating a Windows container image:

You can also cross-build a Windows container image on Linux for a Rust application:
Example Dockerfile: https://github.com/lippertmarkus/cross-building-windows-and-linux-multi-arch-images/blob/main/windows-examples/rust/Dockerfile
How to build with buildkit: https://lippertmarkus.com/2021/11/30/win-multiarch-img-lin/#tldr