Benchmark: Enable 9.6, 9.8 #4118

soulomoon · 2024-03-05T17:21:02Z

~~ Enable 4 version to do bench seems to be taking too long, considering picking two: highest and lowest ~~

Fix bench for newer ghc versions
The following have been done:

No longer use the implicit-hie to generate the hie.yaml for the bench examples and in favor of using "cradle:\n cabal:\n", seems to be working with modern cabal.
upgrade benchmark to use 9.6, 9.8 (The latest two we support for now).
upgrade bench examples to Cabal version: 3.10.2.1, lsp-types version: 2.1.1.0
fix minor error that *.hp files duplicates its extension name

michaelpj · 2024-03-06T10:45:38Z

"Two highest" would also be reasonable, as those are the ones we are going to support for longest. But "highest and lowest" captures the most difference, so is maybe indeed the most interesting. If we do this we should write it down.

Due to its error behaviour on lsp-types-3.1.1.0, In favor of using basic configuration for hie.yaml.

soulomoon · 2024-03-06T23:56:50Z

Some thing problematic with the hie.yaml generated by implicit-hie for lsp-types-2.1.1.0.
Instead of generate the hie.yaml with implicite-hie, fill hie.yaml simply with "cradle:\n cabal:\n" seems
to be working fow more examples in the bench nowadays. And so I did.

Bench for 4 versions of ghc: Total duration
51m 28s.
Acceptable? What do you guys think.

If not I'll chunk it to only run for newest and oldest and add the comment as @michaelpj suggest.

jhrcek

Great work!
I'm not really sure which option is the best (2 highest vs. newest+oldest) but having all 4 seems too much.

From maintenance point of view 2 highest sounds easiest to maintain.
When we add a new ghc (e.g. 9.10)

2 highest will require dropping the older one and adding a new one (e.g. drop 9.6, add 9.10)
newest+oldest will require updating the oldest to one higher (9.2->9.4) + newest to one higher (9.8->9.10)

So I would vote for "2 highest", but don't feel too strongly about it.

jhrcek · 2024-03-06T04:04:34Z

haskell-language-server.cabal

@@ -1835,8 +1835,6 @@ test-suite wrapper-test
 benchmark benchmark
    import:           defaults, warnings
    -- Depends on shake-bench which is unbuildable after this point
-    if impl(ghc >= 9.5)


The comment above belongs to this if, so I think it should be removed too.

jhrcek · 2024-03-06T04:05:06Z

shake-bench/shake-bench.cabal

@@ -17,8 +17,6 @@ source-repository head

 library
  -- Depends on Chart which is unbuildable after this point
-  if impl(ghc >= 9.5)


Is this comment still true? Is chart now buildable?

Not true, we can build chart, I forget to delete them.

fendor

LGTM, thanks for fixing up the benchmark!

ghcide-bench/src/Experiments.hs

michaelpj · 2024-03-07T08:49:14Z

I think just do two. 51 minutes is a lot, and the benchmark job does actually run quite a bit, so we're just burning CPU cycles for not much benefit.

Co-authored-by: fendor <[email protected]>

soulomoon · 2024-03-07T10:40:44Z

After hearing you guys opinion.
I decided we keep just two latest versions.
Benchmark is expensive. I believe @jhrcek "maintenance reason" (lacking man power) and @michaelpj "support longest reason" (make our development more forward looking) does defeat the need to capture the most different(by choosing the lowest and highest)

jhrcek

Thanks! LGTM

@wz1000

## 2.9.0.0 - Bindists for GHC 9.10.1 by @wz1000, @jhrcek, @michaelpj - More hls-graph reliability improvements by @soulomoon - Refactoring of test suite runners by @soulomoon - Fixes in multiple home units support by @wz1000 ### Pull Requests - Fix quadratic memory usage in GetLocatedImports ([#4318](haskell/haskell-language-server#4318)) by @mpickering - Bump stack configs + CI to 9.6.5 and 9.8.2 ([#4316](haskell/haskell-language-server#4316)) by @jhrcek - Add support for Fourmolu 0.16 ([#4314](haskell/haskell-language-server#4314)) by @ brandonchinn178 - Code action to remove redundant record field import (fixes #4220) ([#4308](haskell/haskell-language-server#4308)) by @battermann - Use restricted monad for plugins (#4057) ([#4304](haskell/haskell-language-server#4304)) by @awjchen - 4301 we need to implement utility to wait for all runnning keys in hls graph done ([#4302](haskell/haskell-language-server#4302)) by @soulomoon - Call useWithStale instead of useWithStaleFast when calling ParseCabalFields ([#4294](haskell/haskell-language-server#4294)) by @VeryMilkyJoe - test: add test documenting #806 ([#4292](haskell/haskell-language-server#4292)) by @develop7 - ghcide: drop ghc-check and ghc-paths dependency ([#4291](haskell/haskell-language-server#4291)) by @wz1000 - Limit number of valid hole fits to 10 ([#4288](haskell/haskell-language-server#4288)) by @akshaymankar - Add common stanza to completion data ([#4286](haskell/haskell-language-server#4286)) by @VeryMilkyJoe - FindImports: ThisPkg means some home unit, not "this" unit ([#4284](haskell/haskell-language-server#4284)) by @wz1000 - Remove redudant absolutization in session loader ([#4280](haskell/haskell-language-server#4280)) by @soulomoon - Bump to new lsp versions ([#4279](haskell/haskell-language-server#4279)) by @michaelpj - Put more test code into pre-commit ([#4275](haskell/haskell-language-server#4275)) by @soulomoon - Delete library ghcide test utils ([#4274](haskell/haskell-language-server#4274)) by @soulomoon - Delete testUtil from ghcide-tests ([#4272](haskell/haskell-language-server#4272)) by @soulomoon - CI change, only run bench on performance label ([#4271](haskell/haskell-language-server#4271)) by @soulomoon - Migrate WatchedFileTests ([#4269](haskell/haskell-language-server#4269)) by @soulomoon - Migrate UnitTests ([#4268](haskell/haskell-language-server#4268)) by @soulomoon - Migrate SafeTests ([#4267](haskell/haskell-language-server#4267)) by @soulomoon - Migrate SymlinkTests ([#4266](haskell/haskell-language-server#4266)) by @soulomoon - Remove unused and outdated CHANGELOG files ([#4264](haskell/haskell-language-server#4264)) by @fendor - Enable cabal flaky test ([#4263](haskell/haskell-language-server#4263)) by @soulomoon - Migrate RootUriTests ([#4261](haskell/haskell-language-server#4261)) by @soulomoon - Migrate PreprocessorTests ([#4260](haskell/haskell-language-server#4260)) by @soulomoon - Migrate PluginSimpleTests ([#4259](haskell/haskell-language-server#4259)) by @soulomoon - Migrate ClientSettingsTests ([#4258](haskell/haskell-language-server#4258)) by @soulomoon - Unify critical session running in hls ([#4256](haskell/haskell-language-server#4256)) by @soulomoon - Bump cachix/cachix-action from 14 to 15 ([#4255](haskell/haskell-language-server#4255)) by @dependabot[bot] - Bump haskell-actions/setup from 2.7.2 to 2.7.3 ([#4254](haskell/haskell-language-server#4254)) by @dependabot[bot] - Bump haskell-actions/setup from 2.7.2 to 2.7.3 in /.github/actions/setup-build ([#4253](haskell/haskell-language-server#4253)) by @dependabot[bot] - Shorter file names completion ([#4252](haskell/haskell-language-server#4252)) by @VenInf - Fix progress start delay ([#4249](haskell/haskell-language-server#4249)) by @michaelpj - Bump cachix/install-nix-action from 26 to 27 ([#4245](haskell/haskell-language-server#4245)) by @dependabot[bot] - Bump haskell-actions/setup from 2.7.1 to 2.7.2 ([#4244](haskell/haskell-language-server#4244)) by @dependabot[bot] - Bump haskell-actions/setup from 2.7.1 to 2.7.2 in /.github/actions/setup-build ([#4243](haskell/haskell-language-server#4243)) by @dependabot[bot] - Enable test for #717 ([#4241](haskell/haskell-language-server#4241)) by @soulomoon - Remove Pepe from CODEOWNERS ([#4239](haskell/haskell-language-server#4239)) by @michaelpj - Fix resultBuilt(dirty mechanism) in hls-graph ([#4238](haskell/haskell-language-server#4238)) by @soulomoon - Support for 9.10 ([#4233](haskell/haskell-language-server#4233)) by @wz1000 - Refactor hls-test-util and reduce getCurrentDirectory after initilization ([#4231](haskell/haskell-language-server#4231)) by @soulomoon - [Migrate BootTests] part of #4173 Migrate ghcide tests to hls test utils ([#4227](haskell/haskell-language-server#4227)) by @soulomoon - Actually enable pedantic flag in ci flags job ([#4224](haskell/haskell-language-server#4224)) by @jhrcek - Cleanup cabal files, ghc compat code, fix ghc warnings ([#4222](haskell/haskell-language-server#4222)) by @jhrcek - Another attempt at using the lsp API for some progress reporting ([#4218](haskell/haskell-language-server#4218)) by @michaelpj - [Migrate diagnosticTests] part of #4173 Migrate ghcide tests to hls test utils ([#4207](haskell/haskell-language-server#4207)) by @soulomoon - Prepare release 2.8.0.0 ([#4191](haskell/haskell-language-server#4191)) by @wz1000 - Stabilize the build system by correctly house keeping the dirtykeys and rule values [flaky test #4185 #4093] ([#4190](haskell/haskell-language-server#4190)) by @soulomoon - hls-cabal-plugin: refactor context search to use `readFields` ([#4186](haskell/haskell-language-server#4186)) by @fendor - 3944 extend the properties api to better support nested configuration ([#3952](haskell/haskell-language-server#3952)) by @soulomoon ## 2.8.0.0 - Bindists for GHC 9.6.5 - New hls-notes plugin (#4126, @jvanbruegge) - Floskell, hlint and stylish-haskell plugins enabled for GHC 9.8 - Improvements for hls-graph increasing robustness (#4087, @soulomoon) - Improvements to multi-component support (#4096, #4109, #4179, @wz1000, @fendor) ### Pull Requests - Bump haskell-actions/setup from 2.7.0 to 2.7.1 ([#4189](haskell/haskell-language-server#4189)) by @dependabot[bot] - Bump haskell-actions/setup from 2.7.0 to 2.7.1 in /.github/actions/setup-build ([#4188](haskell/haskell-language-server#4188)) by @dependabot[bot] - Fix ghcdie-tests CI ([#4184](haskell/haskell-language-server#4184)) by @soulomoon - Fix ghc and hlint warnings, fix formatting ([#4181](haskell/haskell-language-server#4181)) by @jhrcek - Allow users to specify whether to use `cabal`'s multi-repl feature ([#4179](haskell/haskell-language-server#4179)) by @fendor - Improve parsing of import suggestions extending multiple multiline imports (fixes #4175) ([#4177](haskell/haskell-language-server#4177)) by @jhrcek - move ghcide-tests to haskell-language-server.cabal and make it depend on hls-test-utils ([#4176](haskell/haskell-language-server#4176)) by @soulomoon - enable ThreadId for when testing ([#4174](haskell/haskell-language-server#4174)) by @soulomoon - Drop Legacy Logger from Codebase ([#4171](haskell/haskell-language-server#4171)) by @fendor - get rid of the `unsafeInterleaveIO` at start up ([#4167](haskell/haskell-language-server#4167)) by @soulomoon - Remove EKG ([#4163](haskell/haskell-language-server#4163)) by @michaelpj - Mark plugins as not buildable if the flag is disabled ([#4160](haskell/haskell-language-server#4160)) by @michaelpj - Fix references to old CPP names in tests, update tests ([#4159](haskell/haskell-language-server#4159)) by @jhrcek - Bump haskell-actions/setup from 2.6.3 to 2.7.0 ([#4158](haskell/haskell-language-server#4158)) by @dependabot[bot] - Bump haskell-actions/setup from 2.6.3 to 2.7.0 in /.github/actions/setup-build ([#4157](haskell/haskell-language-server#4157)) by @dependabot[bot] - Remove dead code in ghcide and hls-graph for priority ([#4151](haskell/haskell-language-server#4151)) by @soulomoon - Bump haskell-actions/setup from 2.6.2 to 2.6.3 in /.github/actions/setup-build ([#4150](haskell/haskell-language-server#4150)) by @dependabot[bot] - Bump haskell-actions/setup from 2.6.2 to 2.6.3 ([#4149](haskell/haskell-language-server#4149)) by @dependabot[bot] - Run ExceptionTests in temporary directory ([#4146](haskell/haskell-language-server#4146)) by @fendor - hls-eval-plugin: Replicate #4139 ([#4140](haskell/haskell-language-server#4140)) by @mattapet - Update comment in refactor tests ([#4138](haskell/haskell-language-server#4138)) by @jhrcek - Update contact info in docs ([#4137](haskell/haskell-language-server#4137)) by @jhrcek - hls-notes-plugin: Do not error if no note is under the cursor ([#4136](haskell/haskell-language-server#4136)) by @jvanbruegge - improve logging in semantic tokens rule ([#4135](haskell/haskell-language-server#4135)) by @soulomoon - Bump softprops/action-gh-release from 1 to 2 ([#4133](haskell/haskell-language-server#4133)) by @dependabot[bot] - Bump cachix/install-nix-action from 25 to 26 ([#4132](haskell/haskell-language-server#4132)) by @dependabot[bot] - Use Set.member instead of Foldable.elem ([#4128](haskell/haskell-language-server#4128)) by @jhrcek - hls-notes-plugin: Initial implementation ([#4126](haskell/haskell-language-server#4126)) by @jvanbruegge - Enable floskell and hlint plugins for ghc 9.8 ([#4125](haskell/haskell-language-server#4125)) by @jhrcek - Integrate stylish-haskell into hls executable with ghc 9.8 ([#4124](haskell/haskell-language-server#4124)) by @jhrcek - Reduce usage of partial functions ([#4123](haskell/haskell-language-server#4123)) by @jhrcek - Benchmark: Enable 9.6, 9.8 ([#4118](haskell/haskell-language-server#4118)) by @soulomoon - Bump haskell-actions/setup from 2.6.1 to 2.6.2 in /.github/actions/setup-build ([#4116](haskell/haskell-language-server#4116)) by @dependabot[bot] - Bump haskell-actions/setup from 2.6.1 to 2.6.2 ([#4115](haskell/haskell-language-server#4115)) by @dependabot[bot] - eval: more robust way to extract comments from ParsedModule ([#4113](haskell/haskell-language-server#4113)) by @jhrcek - Improve isolation of build artefacts of test runs ([#4112](haskell/haskell-language-server#4112)) by @fendor - Improve handling of nonsense rename attempts ([#4111](haskell/haskell-language-server#4111)) by @jhrcek - Exit with non-zero exitcode if wrapper fails to launch ([#4110](haskell/haskell-language-server#4110)) by @fendor - Replace checkHomeUnitsClosed with a faster implementation ([#4109](haskell/haskell-language-server#4109)) by @wz1000 - Don't distribute gifs or plugin readmes ([#4107](haskell/haskell-language-server#4107)) by @fendor - Remove locale workaround for Module name that conatins non-ascii characters ([#4106](haskell/haskell-language-server#4106)) by @fendor - Track extra-source-files of plugins more accurately ([#4105](haskell/haskell-language-server#4105)) by @fendor - remove non-ascii name ([#4103](haskell/haskell-language-server#4103)) by @soulomoon - Add cabal-gild as a cabal file formatter plugin ([#4101](haskell/haskell-language-server#4101)) by @fendor - Remove more workarounds for GHCs < 9.2 (#4092) ([#4098](haskell/haskell-language-server#4098)) by @jhrcek - session-loader: Don't loop forever when we don't find a file in any multi component ([#4096](haskell/haskell-language-server#4096)) by @wz1000 - Prepare release 2.7.0.0 ([#4095](haskell/haskell-language-server#4095)) by @fendor - Remove more workarounds for GHCs < 9.0 ([#4092](haskell/haskell-language-server#4092)) by @jhrcek - Fix hls-graph: phantom dependencies invoke in branching deps (resolve #3423) ([#4087](haskell/haskell-language-server#4087)) by @soulomoon - Rename only if the current module compiles (#3799) ([#3848](haskell/haskell-language-server#3848)) by @sgillespie - Reintroduce ghc-lib flag for hlint plugin ([#3757](haskell/haskell-language-server#3757)) by @RaoulHC

soulomoon added 4 commits March 6, 2024 01:11

enable 9.6 9.8 for bench pipeline

43d2476

update bench example

c8ff612

update lsp-types example

eba48a3

fix bench

2d25b07

soulomoon added 5 commits March 7, 2024 00:03

use known component for bench

24f86ed

remove usage of implicit-hie:gen-hie in bench,

1ff52c1

Due to its error behaviour on lsp-types-3.1.1.0, In favor of using basic configuration for hie.yaml.

update bench config

b5604d9

update

6bfe01a

fix cradle edit

e26b2b7

soulomoon marked this pull request as ready for review March 6, 2024 23:57

soulomoon requested review from michaelpj, fendor and pepeiborra as code owners March 6, 2024 23:57

soulomoon marked this pull request as draft March 6, 2024 23:58

soulomoon marked this pull request as ready for review March 6, 2024 23:59

soulomoon changed the title ~~Pipeline: Enable 9.6 9.8 for bench~~ Pipeline: Enable 9.6, 9.8 for bench Mar 7, 2024

soulomoon changed the title ~~Pipeline: Enable 9.6, 9.8 for bench~~ Benchmark: Enable 9.6, 9.8 Mar 7, 2024

jhrcek reviewed Mar 7, 2024

View reviewed changes

fendor approved these changes Mar 7, 2024

View reviewed changes

ghcide-bench/src/Experiments.hs Outdated Show resolved Hide resolved

soulomoon and others added 3 commits March 7, 2024 16:57

Update Experiments.hs

b3fbab5

Co-authored-by: fendor <[email protected]>

cleanup

90c115d

pick two latest

081a335

update doc

30e51a1

soulomoon enabled auto-merge (squash) March 7, 2024 10:53

Merge branch 'master' into update-hls-bench

d773307

jhrcek approved these changes Mar 7, 2024

View reviewed changes

soulomoon merged commit 8a8f59b into haskell:master Mar 7, 2024
39 checks passed

chenrui333 mentioned this pull request May 8, 2024

haskell-language-server 2.8.0.0 Homebrew/homebrew-core#171165

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Benchmark: Enable 9.6, 9.8 #4118

Benchmark: Enable 9.6, 9.8 #4118

soulomoon commented Mar 5, 2024 •

edited

Loading

michaelpj commented Mar 6, 2024

soulomoon commented Mar 6, 2024 •

edited

Loading

jhrcek left a comment •

edited

Loading

jhrcek Mar 6, 2024

soulomoon Mar 7, 2024 •

edited

Loading

jhrcek Mar 6, 2024

soulomoon Mar 7, 2024 •

edited

Loading

fendor left a comment

michaelpj commented Mar 7, 2024

soulomoon commented Mar 7, 2024 •

edited

Loading

jhrcek left a comment

Benchmark: Enable 9.6, 9.8 #4118

Benchmark: Enable 9.6, 9.8 #4118

Conversation

soulomoon commented Mar 5, 2024 • edited Loading

michaelpj commented Mar 6, 2024

soulomoon commented Mar 6, 2024 • edited Loading

jhrcek left a comment • edited Loading

Choose a reason for hiding this comment

jhrcek Mar 6, 2024

Choose a reason for hiding this comment

soulomoon Mar 7, 2024 • edited Loading

Choose a reason for hiding this comment

jhrcek Mar 6, 2024

Choose a reason for hiding this comment

soulomoon Mar 7, 2024 • edited Loading

Choose a reason for hiding this comment

fendor left a comment

Choose a reason for hiding this comment

michaelpj commented Mar 7, 2024

soulomoon commented Mar 7, 2024 • edited Loading

jhrcek left a comment

Choose a reason for hiding this comment

soulomoon commented Mar 5, 2024 •

edited

Loading

soulomoon commented Mar 6, 2024 •

edited

Loading

jhrcek left a comment •

edited

Loading

soulomoon Mar 7, 2024 •

edited

Loading

soulomoon Mar 7, 2024 •

edited

Loading

soulomoon commented Mar 7, 2024 •

edited

Loading