Introduce the Scala version into the codegen cache namespace

[bpholt]

See #1589 for a detailed description of the problem demonstrated by this scripted test.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[kubukoz]

Possibly useful info from debug logs in CachedTask.scala:

[info] [info] [smithy4s] Arguments changed between smithy4s codegen invocations, diff:
[info] [info] => Diff (- obtained, + expected)
[info] [info]    }, {
[info] [info] -    "file": "file:///private/var/folders/f_/cbwjhbrs2jjg59qn23y6nhvm0000gn/T/sbt_6b76b52e/target/scala-2.13/src_managed/main/smithy/",
[info] [info] -    "hash": "20d077cdbffc38efb1059d82b28b142a50407635"
[info] [info] +    "file": "file:///private/var/folders/f_/cbwjhbrs2jjg59qn23y6nhvm0000gn/T/sbt_6b76b52e/target/scala-3.3.3/src_managed/main/smithy/",
[info] [info] +    "hash": "feb8b807c42e2f15de7316cb1a25a4f04d7a0bc3"
[info] [info]    }, {
[info] [info] -    "file": "file:///private/var/folders/f_/cbwjhbrs2jjg59qn23y6nhvm0000gn/T/sbt_6b76b52e/target/scala-2.13/src_managed/main/smithy/generated-metadata.smithy",
[info] [info] -    "hash": "3f5399412b33a6f5acbbbb2951d4a44b293fb189"
[info] [info] +    "file": "file:///private/var/folders/f_/cbwjhbrs2jjg59qn23y6nhvm0000gn/T/sbt_6b76b52e/target/scala-3.3.3/src_managed/main/smithy/generated-metadata.smithy",
[info] [info] +    "hash": "a5796d45118398985624fe2ae6225929155b59ba"
[info] [info]    }],
[info] [info] -  "output": "/private/var/folders/f_/cbwjhbrs2jjg59qn23y6nhvm0000gn/T/sbt_6b76b52e/target/scala-2.13/src_managed/main/smithy4s",
[info] [info] -  "resourceOutput": "/private/var/folders/f_/cbwjhbrs2jjg59qn23y6nhvm0000gn/T/sbt_6b76b52e/target/scala-2.13/resource_managed/main",
[info] [info] +  "output": "/private/var/folders/f_/cbwjhbrs2jjg59qn23y6nhvm0000gn/T/sbt_6b76b52e/target/scala-3.3.3/src_managed/main/smithy4s",
[info] [info] +  "resourceOutput": "/private/var/folders/f_/cbwjhbrs2jjg59qn23y6nhvm0000gn/T/sbt_6b76b52e/target/scala-3.3.3/resource_managed/main",
[info] [info]    "skip": [],
[info] [info]    "localJars": [{
[info] [info] -    "file": "file:///Users/kubukoz/.ivy2/local/com.disneystreaming.smithy4s/smithy4s-core_2.13/dev-SNAPSHOT/jars/smithy4s-core_2.13.jar",
[info] [info] -    "hash": "c5046ed9fba96f34f2488970dd12513ba6619d91"
[info] [info] +    "file": "file:///Users/kubukoz/.ivy2/local/com.disneystreaming.smithy4s/smithy4s-core_3/dev-SNAPSHOT/jars/smithy4s-core_3.jar",
[info] [info] +    "hash": "a91463542795c76f118dd25281e122c91bec5121"
[info] [info]    }, {
[info] [info]    }, {
[info] [info] -    "file": "file:///Users/kubukoz/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-collection-compat_2.13/2.11.0/scala-collection-compat_2.13-2.11.0.jar",
[info] [info] -    "hash": "a1d1a4591c59ac01f041762692d81879accd849b"
[info] [info] +    "file": "file:///Users/kubukoz/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-collection-compat_3/2.11.0/scala-collection-compat_3-2.11.0.jar",
[info] [info] +    "hash": "dccc0fe9416577811b2c43a042040d5eb62bfa6e"
[info] [info]    }, {
[info] [info] -    "file": "file:///Users/kubukoz/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.15/scala-library-2.13.15.jar",
[info] [info] -    "hash": "ed6f1d58968b16c5f9067d5cac032d952552de58"
[info] [info] +    "file": "file:///Users/kubukoz/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar",
[info] [info] +    "hash": "3c51da898af4ebfac356fcde11388afb2ae8ee82"
[info] [info]    }, {
[info] [info] +    "file": "file:///Users/kubukoz/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.3.3/scala3-library_3-3.3.3.jar",
[info] [info] +    "hash": "8717a89f7030fde424b4ba44608fb2583b5ed159"
[info] [info] +  }, {
[info] [info]      "file": "file:///Users/kubukoz/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/software/amazon/smithy/smithy-jmespath/1.49.0/smithy-jmespath-1.49.0.jar",

Looks like the codegen treats the re-generation after the Scala version switch as a cache miss rather than a completely different job.

[bpholt]

The scripted codegen-plugin/cross-builds now passes for me locally, so I'm making progress. However, in my actual use case, I have an sbt subproject that defines a ProjectionTransformer used during codegen, and when the state of the repo is such that the version contains a timestamp (which happens when there is uncommitted code, at least with the project settings I'm using), this also blows up the cache.

The preprocessor subproject is registered with the codegen project using smithy4sAllDependenciesAsJars (or smithy4sInternalDependenciesAsJars, I've tried it both ways).

The preprocessor is rebuilt when switching Scala versions, meaning the jar's name isn't stable when it contains a timestamp. This goes into CodegenArgs.localJars, which is the input to the cache on the actual codegen task.

So I think I need to figure out how to avoid running the proprocessor project's Compile/packageBin task every time, or somehow introduce a cache there to avoid its version changing. Or maybe there's a way to use that jar's hash instead of the hash of the jar's filename (assuming the jar itself is actually stable). I'm not sure yet.

[Baccata]

So I think I need to figure out how to avoid running the proprocessor project's Compile/packageBin task every time

Considering you have NoPublishSettings enabled on the project that contains the preprocessor, you should be able to pin its version without risk, and stabilise the cache.

Or maybe there's a way to use that jar's hash instead of the hash of the jar's filename

We already have a way, but it's also including the filename. I wouldn't want to use only the hash, as the filenames may be significant and I don't really want to open the door to other unforeseen problems to mitigate an issue that's frankly rather niche.

[bpholt]

Considering you have NoPublishSettings enabled on the project that contains the preprocessor, you should be able to pin its version without risk, and stabilise the cache.

Yes, of course, that's a great idea. Thanks for catching that! I tested this in my library locally and it appears to work with the changes I made in this PR.

So, I think the changes I made are sufficient to solve the problem and can be reviewed for inclusion. I'll update the PR accordingly.