Using Uprof to estimate the Genoa memory bandwidth

[meta1360]

I am using uProf tool (AMDuProfSys and AMDuProfPcm) to get memory bandwith on a Genoa CPU which has 12 memory channels. I am running Spack2 stream application and while it is expected to be ~70GB/s these tools are giving me only around 20 GB/s. can you help me please to know where I am doing something wrong? my command is like this

./AMDuProfSys.py collect --config df,umc -C 0 -o output taskset -c 0 ./stream

[jychang48]

Hi @meta1360 , thanks for reaching out. We’re going to try and get one of the uProf experts to help you

[meta1360]

Hi Justin, I appreciate your help.

…

-Meysam

On Fri, Jul 12, 2024 at 8:57 PM Justin Chang ***@***.***> wrote: Hi @meta1360 <https://github.com/meta1360> , thanks for reaching out. We’re going to try and get one of the uProf experts to help you — Reply to this email directly, view it on GitHub <#18 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACYNRJ2SKPSCL45NLBUGWULZMCQTNAVCNFSM6AAAAABKZXIHHKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMBTG4YDKNQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[meta1360]

Hi @jychang48
Could you find somebody from the uProf team with whom I can discuss my issue?
Thanks for all your help and attention.
-Meysam

[meta1360]

Hi @jychang48
I haven't heard from you, couldn't you find some members of UProf team? Should I be still waiting?

thanks.

[meta1360]

Hi Justin, Any Luck in finding somebody to discuss my issues in uProf?

…

On Fri, Jul 12, 2024 at 9:11 PM Meysam Taassori ***@***.***> wrote: Hi Justin, I appreciate your help. -Meysam On Fri, Jul 12, 2024 at 8:57 PM Justin Chang ***@***.***> wrote: > Hi @meta1360 <https://github.com/meta1360> , thanks for reaching out. > We’re going to try and get one of the uProf experts to help you > > — > Reply to this email directly, view it on GitHub > <#18 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACYNRJ2SKPSCL45NLBUGWULZMCQTNAVCNFSM6AAAAABKZXIHHKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMBTG4YDKNQ> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[gsitaram]

Hi @meta1360,
Thank you for reaching out to us and providing the command you used. From the command, I see that you are trying to collect memory bandwidth from core 0 only. This invocation will not use all memory channels on the socket, so the achieved memory bandwidth will be low. Could you try using all cores for this test, as shown in the command below?

./AMDuProfPcm -m memory -a -d 60 -o /tmp/pcmdata.csv

Here, we are collecting memory bandwidth information from all channels using -a to specify collection from all cores of the socket for a duration of 60 seconds.
Apologies for the delay in this response and I hope this clarifies the issue you faced. For more details, the AMDuProf User Guide seems to be a good reference.

[meta1360]

Hi Gina, Thanks for your help and reply. I have to bind the command you suggested with an app because I want to measure the memory bandwidth when an app is running the app is a simple Stream AMD Spack ./AMDuProfPcm -r -m memory -C -a -o out2.csv -- ./stream_triad the stream reports ~700G but Uprof measures it as 500G? Besides that, I am using the perf command as follows perf -e <event> to get the memory accesses and I am using this document of AMD ***@***.***/ for event number, As mentioned in this document, I am using remote(local)_processor_write(read)_data_beats_cs(0-11) to sum up all these counters to get the total memory accesses, but this time the number of memory accesses is way (~10x or 15x) more than Stream expected values. Especially, working with these counters remote(local)_processor_ write(read)_data_beats_cs(0-11) are so vital for me is there a way to get an acceptable estimation out of these counters? if not, what is your suggestion? Your prompt reply would be so much appreciated.

…

On Mon, Jul 22, 2024 at 8:09 PM Gina Sitaraman ***@***.***> wrote: Hi @meta1360 <https://github.com/meta1360>, Thank you for reaching out to us and providing the command you used. From the command, I see that you are trying to collect memory bandwidth from core 0 only. This invocation will not use all memory channels on the socket, so the achieved memory bandwidth will be low. Could you try using all cores for this test, as shown in the command below? ./AMDuProfPcm -m memory -a -d 60 -o /tmp/pcmdata.csv Here, we are collecting memory bandwidth information from all channels using -a to specify collection from all cores of the socket for a duration of 60 seconds. Apologies for the delay in this response and I hope this clarifies the issue you faced. For more details, the AMDuProf User Guide <https://www.amd.com/content/dam/amd/en/documents/developer/version-4-2-documents/uprof/uprof-user-guide-v4.2.pdf> seems to be a good reference. — Reply to this email directly, view it on GitHub <#18 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACYNRJYQPYGEVNIA7AHNIX3ZNXCOPAVCNFSM6AAAAABKZXIHHKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMJSGEZDQMY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[gsitaram]

I have requested help from the uProf team now for further details about the hardware counters. Let's wait to hear from them.

[meta1360]

Thanks for your help, I appreciate it.

…

-Meysam

On Tue, Jul 23, 2024 at 7:21 AM Gina Sitaraman ***@***.***> wrote: I have requested help from the uProf team now for further details about the hardware counters. Let's wait to hear from them. — Reply to this email directly, view it on GitHub <#18 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACYNRJ5RNPRCSQHZ72NECSLZNZREZAVCNFSM6AAAAABKZXIHHKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMJSG4ZTOOI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[meta1360]

Any Update?

[gsitaram]

Hi @meta1360, the AMDuProf team investigated this and determined that running Stream for a longer duration can help AMDuProfPcm report memory bandwidth that is closer to the expected bandwidth. This is because setup and teardown in Stream where much bandwidth is not consumed is also included in the calculation and this lowers the average. This is expected behavior.
The roofline option in AMDuProfPcm, on the other hand, reports more accurately because it filters out samples with below average cpu utilization or memory bandwidth.

Please try running stream with NTIMES = 500 or so, and this should give you the expected bandwidth.

[meta1360]

No problem at all, thanks for taking a look at the command.
I am just wondering why I am getting such a low number on each channel by collecting the mem_trans_total counters? Doesn't it look strange to you?

[gsitaram]

Yes, it's puzzling. I have requested help, but in the meantime, would try to reproduce your observation. We'll get back to you.

[meta1360]

I am using perf stat command as well
perf stat -a -e amd_iommu_0/mem_trans_total/ -e amd_iommu_1/mem_trans_total/ -e amd_iommu_2/mem_trans_total/ -e amd_iommu_3/mem_trans_total/ -- numactl --cpunodebind 0 --membind 0 -C 0-31 stream.5G

This is what C code (stream) gives me
Function Rate MB/s time
10 Triad: 5943.3 20.190970

---

and this is what counters capture as the data transferred to/from memory.
Performance counter stats for 'system wide':

             0      amd_iommu_0/mem_trans_total/
             0      amd_iommu_1/mem_trans_total/
           278      amd_iommu_2/mem_trans_total/
        70,958      amd_iommu_3/mem_trans_total/

  28.175127837 seconds time elapsed

[sham1810]

Hi @gsitaram Thanks for your reply and your suggestion. I am running Triad kernel in the Stream app (C = A * cst + B) for one trial (DNTIMES=1) on three 5000M-entry arrays each entry is 8 bytes. I am running on a Milan processor with 16 cores (2 threads SMT) based on the time that C code measures, the entire kernel takes ~20 sec -- I think this duration is long enough. Based on the entire data transferred (5G * 8 (byte) * 3 (array) = 120GB, we can calculate the bandwidth as 120GB/20sec = 6GB/s. The following command gives me a total 60GB/s bandwidth. ./AMDuProfPcm -r -m memory -a -C -o ~/counterVerification/Milan/Milan_out/20Sec.csv -- numactl --cpunodebind 0 --membind 0 -C 0-31 /spack2/stream

@meta1360 : AMDuProfPcm needs to collect enough samples to determine the memory bandwidth accurately, increasing the duration (array size or number of runs NTIMES) will help with that.
Please try
./AMDuProfPcm -m memory -a -d <duration> -o <output file> without "-C" to get the bandwidth in timeseries mode and share the output here.

[gsitaram]

@meta1360, did you get a chance to try @sham1810 's suggestion? We think running for a larger duration would help you see the right values in counters. We would like to close this issue if this works for you.

[meta1360]

Hi @gsitaram Gina Sitaraman,
Thanks for following up on this issue.
Yes I got it resolved, I had to follow this suggestion
https://github.com/RRZE-HPC/likwid/wiki/TutorialLikwidPerf
to bring up a kernel module named amd_uncore.ko.
Would you please close the issue?
Thanks,