Benchmark work estimate

[upsj]

This PR adds work estimates to the executor Operations, implements them for a few Dense kernels and outputs them in benchmark loggers.

Related to #1784

[sonarqubecloud]

SonarCloud Quality Gate failed.

2 Bugs
0 Vulnerabilities
0 Security Hotspots
31 Code Smells

2.3% Coverage
6.5% Duplication

[codecov]

Codecov Report

Merging #802 (2e1aa39) into develop (621f991) will increase coverage by 3.68%.
The diff coverage is 21.05%.

❗ Current head 2e1aa39 differs from pull request most recent head dea9c36. Consider uploading reports for the commit dea9c36 to get more accurate results

@@             Coverage Diff             @@
##           develop     #802      +/-   ##
===========================================
+ Coverage    90.50%   94.19%   +3.68%     
===========================================
  Files          505      401     -104     
  Lines        43856    32156   -11700     
===========================================
- Hits         39693    30289    -9404     
+ Misses        4163     1867    -2296     

Impacted Files	Coverage Δ
core/matrix/dense_kernels.hpp	0.00% <0.00%> (ø)
include/ginkgo/core/base/executor.hpp	78.64% <0.00%> (+5.96%)	⬆️
core/matrix/dense.cpp	99.51% <100.00%> (+6.16%)	⬆️
core/factorization/ilu.cpp	0.00% <0.00%> (-100.00%)	⬇️
reference/factorization/ic_kernels.cpp	0.00% <0.00%> (-100.00%)	⬇️
reference/factorization/ilu_kernels.cpp	0.00% <0.00%> (-100.00%)	⬇️
include/ginkgo/core/factorization/ilu.hpp	0.00% <0.00%> (-100.00%)	⬇️
core/factorization/ic.cpp	0.00% <0.00%> (-96.88%)	⬇️
include/ginkgo/core/factorization/ic.hpp	0.00% <0.00%> (-93.34%)	⬇️
omp/matrix/fbcsr_kernels.cpp	0.00% <0.00%> (-57.54%)	⬇️
... and 404 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[tcojean]

I think one limit with this technic if I understand properly is that it requires two consistencies:

1. All executors will have a similar implementation so that they have the same work estimates
2. Every operation is tied to only one kernel.

I'm not sure about 1. for now, I think most of our algorithms have roughly the same order of magnitude of work between executors, but that means a design like the CSR SpMV (classical, imbalance, ...) with strategies would be a no go, and instead we would need an operation for each strategy and switch strategy at the core/algorithm level (which is maybe the best thing to do anyway).

I don't think it's a downside, I thought I should just mention it.

[ginkgo-bot]

Note: This PR changes the Ginkgo ABI:

Functions changes summary: 0 Removed, 0 Changed, 253 Added functions
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable

For details check the full ABI diff under Artifacts here

[pratikvn]

I think instead of artificially classifying operations into likely compute bound and likely memory bound (which is very much dependent on hardware), IMO a better approach would be to just calculate the work and memory complexities of the operations and register them for each operation. We can then have a roofline estimator (which could take in the hardware properties), to estimate whether the operation is memory-bound or compute bound.

[upsj]

I don't think the classification is particularly artificial, let me formulate it as

• compute-bound means the compute complexity grows asymptotically faster than the memory complexity
• memory-bound means the memory complexity grows asymptotically at least as fast as the compute complexity

There are many kernels where it doesn't make sense to talk of FLOPS, or that don't allow for a nice closed-form expression of their memory footprint/compute complexity, which is why I want to leave the option open to either not annotate kernels at all, or to annotate them with custom metrics.

[yhmtsai]

general question: how different between this and the profiler result?
I think they provide the roofline model analysis which might get more detail about the kernel actual action.
We can only calculate in best case (all memory in once) or worst case (no cache), which might not be accurate as what kernels happens in GPUs.

[upsj]

Yes, there are more precise models or exact performance counters. This is aiming mostly to provide a rootline-like approximation to the performance, to quickly highlight kernels that are significantly below expected performance, enabling users to highlight possible optimization opportunities. We can use this framework to capture such information on an application level without the need for executing with a profiler, which requires additional tooling for analysis.

The BLAS 1/2 and solver kernels should be pretty accurate, only the SpMVs undercount accesses to the input vector, which should ideally be served from cache though. IIRC, the footprints are equivalent to what we used in our ACM TOMS paper to report achieved bandwidths for different SpMVs and solvers.