version up : OpenBLAS 0.2.15. Add make-ext-options to specify more make options.

component/openblas/buildout.cfg

@@ -4,22 +4,25 @@ parts =
 
 [openblas]
 recipe = slapos.recipe.cmmi
-# OpenBLAS 0.2.13 and 0.2.14 does not build on Broadwell
-# CPU (Detecting CPU failed). But version 0.2.15 (which
-# does not exist yet) will fix the issue. Until then you
-# can add in options :
-# TARGET=HASWELL
-url = http://github.com/xianyi/OpenBLAS/archive/v0.2.14.tar.gz
-md5sum = 53cda7f420e1ba0ea55de536b24c9701
+url = http://github.com/xianyi/OpenBLAS/archive/v0.2.15.tar.gz
+md5sum = b1190f3d3471685f17cfd1ec1d252ac9
 configure-command = true
+
+# You can specify more options with openblas:make-ext-options parameter.
+# Example :
+# * to build generic binary that supports multiple architecture in one binary
+#   DYNAMIC_ARCH=1
+# * to build on not-yet-supported platform
+#   TARGET=GENERIC
+make-ext-options =
+
 make-options =
   PREFIX="${buildout:parts-directory}/${:_buildout_section_name_}"
   BINARY="$(uname -m | grep -q x86_64 && echo 64 || echo 32)"
   NO_STATIC=1
   USE_OPENMP=1
   USE_THREAD=1
-# to build generic binary that supports multiple architecture in one binary
-# DYNAMIC_ARCH=1
+  ${:make-ext-options}
 environment =
   PATH=${gcc-fortran:location}/bin:%(PATH)s
   LD_LIBRARY_PATH=${gcc-fortran:location}/lib:${gcc-fortran:location}/lib64

[Ivan Tyagov]

For Wendelin where we have people trying to use it with different architectures I believe we should set TARGET=GENERIC This will assure us that we do not have to maintain silly instructions like this one here: http://www.wendelin.io/P-WIA.Wendelin.Standalone, see Note 1: openblast compile will fail if CPU type of the VM is unknown

For projects for which we know the architecture of deployed we can override it if needed.

[Kazuhiko Shiozaki]

Can you try 'qemu-kvm -cpu host' with the latest OpenBLAS to see if it can detect a good target automatically without TARGET= parameter ?

[Ivan Tyagov]

Yes, I can try but not this week.

[Kazuhiko Shiozaki]

Another possibility I'm wondering is building with 'DYNAMIC_ARCH=1' without 'TARGET=...' can work on KVM, in both compilation and usage. Ideally, the best architecture is chosen and hopefully GENERIC is chosen as fallback, but I am not sure if it is possible.

[Ivan Tyagov]

So here's some results:

1. kvm with following CPU (/usr/bin/qemu-system-x86_64 -machine accel=kvm -m 4096 -hda /mnt/hd1/vm/debian8.1_neo.img -net nic,vlan=0,model=virtio,macaddr=00:16:35:AF:94:4B -net tap,vlan=0,ifname=tap0,script=no,downscript=no -vnc :1 -no-reboot -enable-kvm -smp 4) and not set TARGET at all ========================= processor : 3 vendor_id : GenuineIntel cpu family : 6 model : 6 model name : QEMU Virtual CPU version 2.4.0 stepping : 3 microcode : 0x1 cpu MHz : 3299.996 cache size : 4096 KB physical id : 3 siblings : 1 core id : 0 cpu cores : 1 apicid : 3 initial apicid : 3 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl pni cx16 x2apic popcnt hypervisor lahf_lm abm bogomips : 6599.99 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual ================================== Build: fails

2. Same but set TARGET = GENERIC Build: OK

My conclusion, use TARGET = GENERIC by default and commit this is OK I didn't try 'DYNAMIC_ARCH=1'

[Kazuhiko Shiozaki]

First, '-cpu host' is required when we build without TARGET parameter. https://github.com/xianyi/OpenBLAS/wiki/faq#qemu

I am not sure if there is any bad side-effect of using '-cpu host' for generic purpose. (I believe it's safe though).

My preference is always using '-cpu host' in kvm and building OpenBLAS without target by default, that will make (mostly) everyone happy by default.

For DYNAMIC_ARCH=1, unless we use binary cache, we compile the software to use it on the same environment and I don't think it is good to use it by default. It will take much more time to build and use more diskspace.

[Ivan Tyagov]

I'm trying now '-cpu host' in kvm and without set TARGET.

So far seems working, passed openblast build. So question is what's acceptable trade-off:

1. TARGET = GENERIC, no optimization and less if any build issues
2. Let openblast detect first and if fails recommend use kvm option '-cpu host' which we must maintain in documentation.

Let's follow #2 as you suggest and see how many people complain (almost everybody who tried to use hit this bug before as not always able to control the VM they have). If required we can either override for Wendelin or fix for generic later.

CPU:

processor : 3 vendor_id : GenuineIntel cpu family : 6 model : 60 model name : Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz stepping : 3 microcode : 0x1 cpu MHz : 3299.996 cache size : 6144 KB physical id : 3 siblings : 1 core id : 0 cpu cores : 1 apicid : 3 initial apicid : 3 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm arat xsaveopt fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid bogomips : 6599.99 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management:

[Ivan Tyagov]

Can you please commit it?

[Kirill Smelkov]

@kazuhiko, please add reviewd-by @Tyagov and reference to merge-request (nexedi/slapos!42) and apply the patch to master. If you need help with this I can assist and teach you how to do.

Thanks beforehand,
Kirill

[Kazuhiko Shiozaki]

Thanks for your feedbacks. I just pushed d09b1946 so that :

• First it tries build with auto detected target (or specifying build-ext-options)
• If it fails, it tries build with TARGET=GENERIC

What do you think ?

[Ivan Tyagov]

Seems fine, I try build now.