Builds should ignore/filter march and mtune flags?

[gdevenyi]

If I'm building a DYNAMIC_ARCH build, many different files are compiled with different arch flags based on their targets, however if I specify march in CFLAGS during configure, I end up with that in the flags, and when I get to a kernel that uses features my hardware doesn't support, the compile fails.

I ran into this deep inside a cmake superbuild where I want the rest of the code to use -march but cmake passes that onwards into the built-in openblas integration and breaks the build.

[brada4]

The final CFLAGS/FFLAGS parameters are combined per-makefile from CCOMMON_OPT FCOMMON_OPT and COMMON_OPT flags. OpenBLAS will not build with CFLAGS set.

Might be that OpenBLAS is first in your build to do some testing, showing invalid CFLAGS, and rest of superbuild does not work either.

[martin-frbg]

I doubt there is an "any size fits all" solution to this - in a DYNAMIC_ARCH build, the CFLAGS will (mainly) determine what the common code will be built with, and there are probably several legitimate reasons for wanting to influence that.

[gdevenyi]

@brada4 Our superbuild uses the CMAKE configuration, which accepts, and passes through CFLAGS/CXXFLAGS, maybe this is not as intended?

The CFLAGS are not invalid, this is simply -march=native, which breaks the DYNAMIC_ARCH build for CPUs newer than the local system CPU. I build the superbuild in parallel, and everything else works fine (up until OpenBLAS dependent things anyways).

Maybe this is specific to the cmake configuration which I should've mentioned to start...

[martin-frbg]

Can you be a bit more specific about the instruction subset that your build host does not support, and that fail to compile only with the -march=native ? (Is it AVX512 ?)

[brada4]

OpenBLAS build uses variable names CFLAGS and FFLAGS internally. What you set before it interferes with normal operation. In addition it alters those to build included reference LAPACK.
Just unset them mmmmOK?

[gdevenyi]

Yes @martin-frbg in my case its the AVX512 versions failing on my system because -march=native doesn't support it. Purging this setting allows those to build.

...thanks @brada4. That's the bug I'm reporting.

[martin-frbg]

cmake/system.cmake should already add -march=skylake-avx512 as needed - however no attempt is made at removing any pre-existing -march flags (is there a CMAKE command for this ?) and I am not sure how the compiler would react to seeing both "-march=native -march=skylake-avx512"

[gdevenyi]

I guess based on my experience here, it looks like -march=native overrides the other arch settings:

> gcc -march=native -march=skylake-avx512 -E -v - </dev/null 2>&1 | grep cc1
 /usr/lib/gcc/x86_64-linux-gnu/8/cc1 -E -quiet -v -imultiarch x86_64-linux-gnu - -march=broadwell -mmmx -mno-3dnow -msse -msse2 -msse3 -mssse3 -mno-sse4a -mcx16 -msahf -mmovbe -maes -mno-sha -mpclmul -mpopcnt -mabm -mno-lwp -mfma -mno-fma4 -mno-xop -mbmi -mno-sgx -mbmi2 -mno-pconfig -mno-wbnoinvd -mno-tbm -mavx -mavx2 -msse4.2 -msse4.1 -mlzcnt -mrtm -mhle -mrdrnd -mf16c -mfsgsbase -mrdseed -mprfchw -madx -mfxsr -mxsave -mxsaveopt -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mno-prefetchwt1 -mno-clflushopt -mno-xsavec -mno-xsaves -mno-avx512dq -mno-avx512bw -mno-avx512vl -mno-avx512ifma -mno-avx512vbmi -mno-avx5124fmaps -mno-avx5124vnniw -mno-clwb -mno-mwaitx -mno-clzero -mno-pku -mno-rdpid -mno-gfni -mno-shstk -mno-avx512vbmi2 -mno-avx512vnni -mno-vaes -mno-vpclmulqdq -mno-avx512bitalg -mno-movdiri -mno-movdir64b --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=25600 -mtune=broadwell -fstack-protector-strong -Wformat -Wformat-security

^^ I see a bunch of "no AVX" settings which is where I'm running into the compiler failure.

cmake doesn't have a standardized way to remove items from a variable but maybe a regex works. I'm looking for examples.

[gdevenyi]

Looks like a REGEX replace on the C/CXX flags can remove, something like:
https://cmake.org/cmake/help/latest/command/string.html#regular-expressions

string(REGEX REPLACE "-march=[A-Za-z0-9_]+"
       "" CMAKE_C_FLAGS_RELEASE
       "${CMAKE_C_FLAGS_RELEASE}")

^Not tested yet.

[martin-frbg]

From what I saw it does not appear to be that easy from "within" a cmake project. The general recommendation appears to be to either append a corresponding -no-something flag if available (which is obviously not the case here) or to move the "special" files to a separate project and be sure to adjust the CFLAGS before building it (which trivially circumvents the issue).

[martin-frbg]

Hmm. Stripping out the "-march=native" like you suggested does "sort of" work, but only if I replace it very early on, e.g. in the "if (NOT NO_AVX512)" part of the DYNAMIC_ARCH option handling in arch.cmake where it will affect all parts of the build. When I try to limit it to only the SkylakeX kernel part, no matter if I try to do it in system.cmake or in kernel/CMakeLists.txt, I still end up with the original "C_FLAGS=-march=native" in kernel/CMakeFiles/kernel.SKYLAKEX.dir/flags.make

[gdevenyi]

My understanding of dynamic arch is that it should be stripped from all parts of the build, otherwise the build isn't really dynamic anyway. So this sounds good to me.

[martin-frbg]

Closing as fixed in 0.3.7

[gdevenyi]

Wonderful thanks!