Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Ruby 3.1 and 3.2 #533

Merged
merged 10 commits into from
Sep 26, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 55 additions & 34 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,30 +39,47 @@ jobs:
docker exec cimg_ruby bash -c './test/bin/setup_tinytds_db.sh'

- run:
name: compile openssl library
name: bundle install gems
command: |
docker exec cimg_ruby bash -c 'sudo -E ./test/bin/install-openssl.sh'
docker exec cimg_ruby bash -c 'bundle install'

- run:
name: compile freetds library
name: Write used versions into file
command: |
docker exec cimg_ruby bash -c 'sudo -E ./test/bin/install-freetds.sh'
docker exec cimg_ruby bash -c 'bundle exec rake ports:version_file'

- restore_cache:
name: restore ports cache
keys:
- ports-<< parameters.ruby_version >>-{{ checksum ".ports_versions" }}
- ports-<< parameters.ruby_version >>-

- run:
name: bundle install gems
name: compile ports
command: |
docker exec cimg_ruby bash -c 'bundle install'
docker exec cimg_ruby bash -c 'bundle exec rake ports'

- run:
name: build gem
command: |
docker exec cimg_ruby bash -c 'bundle exec rake build'

- run:
name: Fix permissions on ports directory
command: |
docker exec cimg_ruby bash -c 'sudo chown -R $(id -u):$(id -g) ports'

- run:
name: test gem
command: |
docker exec cimg_ruby bash -c 'bundle exec rake test'

- save_cache:
name: save ports cache
paths:
- ./ports
key: ports-<< parameters.ruby_version >>-{{ checksum ".ports_versions" }}

- store_test_results:
path: test/reports

Expand Down Expand Up @@ -197,53 +214,49 @@ jobs:
path: test/reports

cross_compile_gem:
machine:
image: ubuntu-2004:current
parameters:
platform:
description: "Platform to compile the gem resources"
type: string

docker:
- image: "ghcr.io/rake-compiler/rake-compiler-dock-image:1.3.0-mri-<< parameters.platform >>"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switching the base image here is a little surprising, can you share more why this is needed? Also, rather than pinning to 1.3.0, is there a tag for the latest release?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switching the base image here is a little surprising, can you share more why this is needed?

It's been a while since I implemented this. But basically, we previously had this one Ubuntu job which compiled the ports for x86-mingw32, x64-mingw32. Now with this PR, we need to introduce another platform with x64-mingw-ucrt.

To speed to process up, I instead opt to run a separate job for each architecture. I do not recall how much time it saves, but I assume we talk about ~10 minutes per pipeline run.

Also, rather than pinning to 1.3.0, is there a tag for the latest release?

Looking into GitHub packages, there is a snapshot version available, which might be unstable. Then 1.3.0 is already the latest release.

I think it makes sense to pin this version to align it with the rake-compiler-dock gem.


steps:
- ruby/install:
version: '2.7'
- checkout
- restore_cache:
name: restore gem cache
keys:
- v1-bundle-{{ .Branch }}-{{ checksum "tiny_tds.gemspec" }}
- v1-bundle-{{ .Branch }}-
- v1-bundle-

- run:
name: bundle install gems
command: |
bundle install --path vendor/bundle

- save_cache:
name: save gem cache
paths:
- ./vendor/bundle
key: v1-bundle-{{ .Branch }}-{{ checksum "tiny_tds.gemspec" }}
bundle install

- run:
name: Write used versions for ports into file
command: |
bundle exec rake ports:version_file
rake ports:version_file[<< parameters.platform >>]

- restore_cache:
name: restore ports cache
keys:
- ports-{{ checksum ".ports_versions" }}
- ports-
- ports-win-{{ checksum ".ports_versions" }}
- ports-win-

- run:
name: Build gems
name: Build gem
command: |
bundle exec rake gem
bundle exec rake gem:native
rake gem:for_platform[<< parameters.platform >>]

- run:
name: Move gems into separate directory before caching
command: |
mkdir -p artifacts/gems
mv pkg/*.gem artifacts/gems
mkdir -p artifacts-<< parameters.platform >>/gems
mv pkg/*.gem artifacts-<< parameters.platform >>/gems

- run:
name: Remove non-native gem to avoid conflict in workspace
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean there's unexpected shared state between the CI jobs? Addressing that might be a better approach than "delete everything that's not me" 🤔

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a rather undesired side-effect of gem build: Each build yields its non-native version of the gem, on which CircleCI reports a conflict of resources. I did some research and there seems to be not a way to surpress the generation of the non-native gem.

command: |
gemVersion=$(cat VERSION | tr -d "[:space:]")
rm -rf artifacts-<< parameters.platform >>/gems/tiny_tds-$gemVersion.gem

- store_artifacts:
path: artifacts/gems
Expand All @@ -252,18 +265,24 @@ jobs:
name: save ports cache
paths:
- ./ports
key: ports-{{ checksum ".ports_versions" }}
key: ports-win-{{ checksum ".ports_versions" }}

- persist_to_workspace:
name: save gems into workspace
root: artifacts
root: artifacts-<< parameters.platform >>
paths:
- gems

workflows:
test_supported_ruby_versions:
jobs:
- cross_compile_gem
- cross_compile_gem:
matrix:
parameters:
platform:
- "x86-mingw32"
- "x64-mingw32"
- "x64-mingw-ucrt"
- test_windows:
requires:
- cross_compile_gem
Expand All @@ -275,5 +294,7 @@ workflows:
- '2.6'
- '2.7'
- '3.0'
- '3.1'
- '3.2'
- test_linux:
matrix: *ruby_versions
29 changes: 19 additions & 10 deletions Rakefile
Original file line number Diff line number Diff line change
Expand Up @@ -6,20 +6,33 @@ require 'rake/extensiontask'
require_relative './ext/tiny_tds/extconsts'

SPEC = Gem::Specification.load(File.expand_path('../tiny_tds.gemspec', __FILE__))

ruby_cc_ucrt_versions = "3.2.0:3.1.0".freeze
ruby_cc_mingw32_versions = "3.0.0:2.7.0:2.6.0:2.5.0:2.4.0".freeze

GEM_PLATFORM_HOSTS = {
'x86-mingw32' => 'i686-w64-mingw32',
'x64-mingw32' => 'x86_64-w64-mingw32'
'x86-mingw32' => {
host: 'i686-w64-mingw32',
ruby_versions: ruby_cc_mingw32_versions
},
'x64-mingw32' => {
host: 'x86_64-w64-mingw32',
ruby_versions: ruby_cc_mingw32_versions
},
'x64-mingw-ucrt' => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this commit is also introducing a ucrt based build, can you update the commit message with an additional explanation on this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a reference to the commit in cae55bf.

host: 'x86_64-w64-mingw32',
ruby_versions: ruby_cc_ucrt_versions
},
}
RUBY_CC_VERSION="3.0.0:2.7.0:2.6.0:2.5.0:2.4.0".freeze

# Add our project specific files to clean for a rebuild
CLEAN.include FileList["{ext,lib}/**/*.{so,#{RbConfig::CONFIG['DLEXT']},o}"],
FileList["exe/*"]
FileList["exe/*"]

# Clobber all our temp files and ports files including .install files
# and archives
CLOBBER.include FileList["tmp/**/*"],
FileList["ports/**/*"].exclude(%r{^ports/archives})
FileList["ports/**/*"].exclude(%r{^ports/archives})

Dir['tasks/*.rake'].sort.each { |f| load f }

Expand All @@ -33,15 +46,11 @@ Rake::ExtensionTask.new('tiny_tds', SPEC) do |ext|
# The fat binary gem doesn't depend on the freetds package, since it bundles the library.
spec.metadata.delete('msys2_mingw_dependencies')

platform_host_map = GEM_PLATFORM_HOSTS
gemplat = spec.platform.to_s
host = platform_host_map[gemplat]

# We don't need the sources in a fat binary gem
spec.files = spec.files.reject { |f| f =~ %r{^ports\/archives/} }

# Make sure to include the ports binaries and libraries
spec.files += FileList["ports/#{host}/**/**/{bin,lib}/*"].exclude do |f|
spec.files += FileList["ports/#{spec.platform.to_s}/**/**/{bin,lib}/*"].exclude do |f|
File.directory? f
end

Expand Down
6 changes: 4 additions & 2 deletions ext/tiny_tds/extconf.rb
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,11 @@ def do_help
do_help if arg_config('--help')

# Make sure to check the ports path for the configured host
host = RbConfig::CONFIG['host']
architecture = RbConfig::CONFIG['arch']
architecture = "x86-mingw32" if architecture == "i386-mingw32"

project_dir = File.expand_path("../../..", __FILE__)
freetds_ports_dir = File.join(project_dir, 'ports', host, 'freetds', FREETDS_VERSION)
freetds_ports_dir = File.join(project_dir, 'ports', architecture, 'freetds', FREETDS_VERSION)
Comment on lines -24 to +28
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't completely understand the swap from "host" to "architecture", other than that I generally agree the latter makes sense. Is switching this matching rake compiler patterns, or is it something bespoke for this project's config?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not really on us, but rather on rake-compiler-dock.

I edited the description, so it's hopefully more clear.

freetds_ports_dir = File.expand_path(freetds_ports_dir)

# Add all the special path searching from the original tiny_tds build
Expand Down
7 changes: 1 addition & 6 deletions lib/tiny_tds/gem.rb
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,7 @@ def ports_lib_paths
end

def ports_host
h = RbConfig::CONFIG['host']

# Our fat binary builds with a i686-w64-mingw32 toolchain
# but ruby for windows x32-mingw32 reports i686-pc-mingw32
# so correct the host here
h.gsub('i686-pc-mingw32', 'i686-w64-mingw32')
RbConfig::CONFIG["arch"]
end
end
end
Expand Down
16 changes: 13 additions & 3 deletions tasks/native_gem.rake
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,19 @@ task 'gem:native' => ['ports:cross'] do
require 'rake_compiler_dock'

# make sure to install our bundle
sh "bundle package --all" # Avoid repeated downloads of gems by using gem files from the host.
sh "bundle package --all" # Avoid repeated downloads of gems by using gem files from the host.

GEM_PLATFORM_HOSTS.keys.each do |plat|
RakeCompilerDock.sh "bundle --local && RUBY_CC_VERSION=#{RUBY_CC_VERSION} rake native:#{plat} gem", platform: plat
GEM_PLATFORM_HOSTS.each do |plat, meta|
RakeCompilerDock.sh "bundle --local && RUBY_CC_VERSION=#{meta[:ruby_versions]} rake native:#{plat} gem", platform: plat
end
end

# assumes you are in a container provided by Rake compiler
# if not, use the task above
task 'gem:for_platform', [:gem_platform] do |_task, args|
args.with_defaults(gem_platform: RbConfig::CONFIG["arch"])

sh "bundle install"
Rake::Task["ports:compile"].invoke(GEM_PLATFORM_HOSTS[args.gem_platform][:host], args.gem_platform)
sh "RUBY_CC_VERSION=#{GEM_PLATFORM_HOSTS[args.gem_platform][:ruby_versions]} rake native:#{args.gem_platform} gem"
end
36 changes: 22 additions & 14 deletions tasks/ports.rake
Original file line number Diff line number Diff line change
Expand Up @@ -14,32 +14,36 @@ namespace :ports do
}

directory "ports"
CLEAN.include "ports/*mingw32*"
CLEAN.include "ports/*mingw*"
CLEAN.include "ports/*.installed"

task :openssl, [:host] do |task, args|
args.with_defaults(host: RbConfig::CONFIG['host'])
task :openssl, [:host, :gem_platform] do |_task, args|
args.with_defaults(host: RbConfig::CONFIG['host'], gem_platform: RbConfig::CONFIG["arch"])

libraries_to_compile[:openssl].files = [OPENSSL_SOURCE_URI]
libraries_to_compile[:openssl].host = args.host
libraries_to_compile[:openssl].gem_platform = args.gem_platform

libraries_to_compile[:openssl].cook
libraries_to_compile[:openssl].activate
end

task :libiconv, [:host] do |task, args|
args.with_defaults(host: RbConfig::CONFIG['host'])
task :libiconv, [:host, :gem_platform] do |_task, args|
args.with_defaults(host: RbConfig::CONFIG['host'], gem_platform: RbConfig::CONFIG["arch"])

libraries_to_compile[:libiconv].files = [ICONV_SOURCE_URI]
libraries_to_compile[:libiconv].host = args.host
libraries_to_compile[:libiconv].gem_platform = args.gem_platform
libraries_to_compile[:libiconv].cook
libraries_to_compile[:libiconv].activate
end

task :freetds, [:host] do |task, args|
args.with_defaults(host: RbConfig::CONFIG['host'])
task :freetds, [:host, :gem_platform] do |_task, args|
args.with_defaults(host: RbConfig::CONFIG['host'], gem_platform: RbConfig::CONFIG["arch"])

libraries_to_compile[:freetds].files = [FREETDS_SOURCE_URI]
libraries_to_compile[:freetds].host = args.host
libraries_to_compile[:freetds].gem_platform = args.gem_platform

if libraries_to_compile[:openssl]
# freetds doesn't have an option that will provide an rpath
Expand All @@ -59,13 +63,13 @@ namespace :ports do
libraries_to_compile[:freetds].activate
end

task :compile, [:host] do |task, args|
args.with_defaults(host: RbConfig::CONFIG['host'])
task :compile, [:host, :gem_platform] do |_task, args|
args.with_defaults(host: RbConfig::CONFIG['host'], gem_platform: RbConfig::CONFIG["arch"])

puts "Compiling ports for #{args.host}..."
puts "Compiling ports for #{args.host} (Ruby platform #{args.gem_platform}) ..."

libraries_to_compile.keys.each do |lib|
Rake::Task["ports:#{lib}"].invoke(args.host)
Rake::Task["ports:#{lib}"].invoke(args.host, args.gem_platform)
end
end

Expand All @@ -74,22 +78,26 @@ namespace :ports do
require 'rake_compiler_dock'

# build the ports for all our cross compile hosts
GEM_PLATFORM_HOSTS.each do |gem_platform, host|
GEM_PLATFORM_HOSTS.each do |gem_platform, meta|
# make sure to install our bundle
build = ['bundle']
build << "rake ports:compile[#{host}] MAKE='make -j`nproc`'"
build << "RUBY_CC_VERSION=#{meta[:ruby_versions]} rake ports:compile[#{meta[:host]},#{gem_platform}] MAKE='make -j`nproc`'"
RakeCompilerDock.sh build.join(' && '), platform: gem_platform
end
end

desc "Notes the actual versions for the compiled ports into a file"
task "version_file" do
task "version_file", [:gem_platform] do |_task, args|
args.with_defaults(gem_platform: RbConfig::CONFIG["arch"])

ports_version = {}

libraries_to_compile.each do |library, library_recipe|
ports_version[library] = library_recipe.version
end

ports_version[:platform] = args.gem_platform

File.open(".ports_versions", "w") do |f|
f.write ports_version
end
Expand Down
Loading