[package] gfortran/10.2: Missing files causing execution error

[uilianries]

First appearance on #4681

There are some missing files in gfortran which are necessary its execution.

Package and Environment Details (include every applicable attribute)

• Package Name/Version: gfortran/10.2
• Operating System+version: Linux Ubuntu 18.04
• Compiler+version: GCC 10
• Docker image: conanio/gcc10
• Conan version: conan 1.33.1
• Python version: Python 3.8.2

Conan profile (output of conan profile show default or conan profile show <profile> if custom profile is in use)

Put your profile output here

Steps to reproduce (Include if Applicable)

• Add Fortran as project in your CMake file:

  add_project(foobar C++ Fortran)

• Add gfortran/10.2 as requirement

• Run any Conan command which invokes build step.

Logs (Include/Attach if Applicable)

Click to expand log

himalaya/4.1.1: 
gfortran: fatal error: cannot execute ‘cc1’: execvp: No such file or directory
compilation terminated.
CMake Error at /opt/pyenv/versions/3.7.5/lib/python3.7/site-packages/cmake/data/share/cmake-3.18/Modules/CMakeTestFortranCompiler.cmake:51 (message):
  The Fortran compiler

    "/home/conan/w/BuildSingleReference/.conan/data/gfortran/10.2/_/_/package/4db1be536558d833e52e862fd84d64d75c2b3656/bin/gfortran"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /home/conan/w/BuildSingleReference/.conan/data/himalaya/4.1.1/_/_/build/b08e4f558500f42873bab1a4177938467ada1451/build_subfolder/CMakeFiles/CMakeTmp
    
    Run Build Command(s):/usr/bin/make cmTC_1476c/fast && /usr/bin/make  -f CMakeFiles/cmTC_1476c.dir/build.make CMakeFiles/cmTC_1476c.dir/build
    make[1]: Entering directory '/home/conan/w/BuildSingleReference/.conan/data/himalaya/4.1.1/_/_/build/b08e4f558500f42873bab1a4177938467ada1451/build_subfolder/CMakeFiles/CMakeTmp'
    Building Fortran object CMakeFiles/cmTC_1476c.dir/testFortranCompiler.f.o
    /home/conan/w/BuildSingleReference/.conan/data/gfortran/10.2/_/_/package/4db1be536558d833e52e862fd84d64d75c2b3656/bin/gfortran    -c /home/conan/w/BuildSingleReference/.conan/data/himalaya/4.1.1/_/_/build/b08e4f558500f42873bab1a4177938467ada1451/build_subfolder/CMakeFiles/CMakeTmp/testFortranCompiler.f -o CMakeFiles/cmTC_1476c.dir/testFortranCompiler.f.o
    gfortran: fatal error: cannot execute ‘f951’: execvp: No such file or directory
    compilation terminated.
    make[1]: *** [CMakeFiles/cmTC_1476c.dir/build.make:85: CMakeFiles/cmTC_1476c.dir/testFortranCompiler.f.o] Error 1
    make[1]: Leaving directory '/home/conan/w/BuildSingleReference/.conan/data/himalaya/4.1.1/_/_/build/b08e4f558500f42873bab1a4177938467ada1451/build_subfolder/CMakeFiles/CMakeTmp'
    make: *** [Makefile:140: cmTC_1476c/fast] Error 2

[SpaceIm]

Yes f951 is the actual compiler, it should be packaged with gfortran executable.

Problem in this recipe: the "Linux" archive is not there anymore. And surprinsingly the recipe tries to download it even though I'm on Windows.

[SpaceIm]

This recipe is broken in another way: f951 has dependencies, and we probably don't want to put its vendored dependencies in lib folder of this package...

On Macos:

spaceim:~/.conan/data/gfortran/10.2/_/_/package/46f53f156846659bf39ad6675fa0ee8156e859fe/bin$ objdump -p f951

f951:	file format Mach-O 64-bit x86-64

Mach header
      magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
MH_MAGIC_64  X86_64        ALL  0x00     EXECUTE    20       3008   NOUNDEFS DYLDLINK TWOLEVEL WEAK_DEFINES BINDS_TO_WEAK
Load command 0
      cmd LC_SEGMENT_64
  cmdsize 72
  segname __PAGEZERO
   vmaddr 0x0000000000000000
   vmsize 0x0000000100000000
  fileoff 0
 filesize 0
  maxprot ---
 initprot ---
   nsects 0
    flags (none)
Load command 1
      cmd LC_SEGMENT_64
  cmdsize 712
  segname __TEXT
   vmaddr 0x0000000100000000
   vmsize 0x0000000001894000
  fileoff 0
 filesize 25772032
  maxprot r-x
 initprot r-x
   nsects 8
    flags (none)
Section
  sectname __text
   segname __TEXT
      addr 0x0000000100001370
      size 0x00000000010ff294
    offset 4976
     align 2^4 (16)
    reloff 0
    nreloc 0
      type S_REGULAR
attributes PURE_INSTRUCTIONS SOME_INSTRUCTIONS
 reserved1 0
 reserved2 0
Section
  sectname __text_cold
   segname __TEXT
      addr 0x0000000101100604
      size 0x000000000004aa2a
    offset 17827332
     align 2^1 (2)
    reloff 0
    nreloc 0
      type S_REGULAR
attributes PURE_INSTRUCTIONS SOME_INSTRUCTIONS
 reserved1 0
 reserved2 0
Section
  sectname __text_startup
   segname __TEXT
      addr 0x000000010114b030
      size 0x0000000000007a4e
    offset 18133040
     align 2^4 (16)
    reloff 0
    nreloc 0
      type S_REGULAR
attributes PURE_INSTRUCTIONS SOME_INSTRUCTIONS
 reserved1 0
 reserved2 0
Section
  sectname __stubs
   segname __TEXT
      addr 0x0000000101152a7e
      size 0x0000000000001ff2
    offset 18164350
     align 2^1 (2)
    reloff 0
    nreloc 0
      type S_SYMBOL_STUBS
attributes PURE_INSTRUCTIONS SOME_INSTRUCTIONS
 reserved1 0 (index into indirect symbol table)
 reserved2 6 (size of stubs)
Section
  sectname __stub_helper
   segname __TEXT
      addr 0x0000000101154a70
      size 0x0000000000000e70
    offset 18172528
     align 2^2 (4)
    reloff 0
    nreloc 0
      type S_REGULAR
attributes PURE_INSTRUCTIONS SOME_INSTRUCTIONS
 reserved1 0
 reserved2 0
Section
  sectname __cstring
   segname __TEXT
      addr 0x00000001011558e0
      size 0x000000000016451a
    offset 18176224
     align 2^3 (8)
    reloff 0
    nreloc 0
      type S_CSTRING_LITERALS
attributes (none)
 reserved1 0
 reserved2 0
Section
  sectname __const
   segname __TEXT
      addr 0x00000001012b9e00
      size 0x000000000022b3f5
    offset 19635712
     align 2^5 (32)
    reloff 0
    nreloc 0
      type S_REGULAR
attributes (none)
 reserved1 0
 reserved2 0
Section
  sectname __eh_frame
   segname __TEXT
      addr 0x00000001014e51f8
      size 0x00000000003aee00
    offset 21910008
     align 2^3 (8)
    reloff 0
    nreloc 0
      type S_REGULAR
attributes (none)
 reserved1 0
 reserved2 0
Load command 2
      cmd LC_SEGMENT_64
  cmdsize 1512
  segname __DATA
   vmaddr 0x0000000101894000
   vmsize 0x00000000402c4000
  fileoff 25772032
 filesize 1474560
  maxprot rw-
 initprot rw-
   nsects 18
    flags (none)
Section
  sectname __nl_symbol_ptr
   segname __DATA
      addr 0x0000000101894000
      size 0x0000000000000008
    offset 25772032
     align 2^3 (8)
    reloff 0
    nreloc 0
      type S_NON_LAZY_SYMBOL_POINTERS
attributes (none)
 reserved1 1363 (index into indirect symbol table)
 reserved2 0
Section
  sectname __got
   segname __DATA
      addr 0x0000000101894008
      size 0x0000000000001f10
    offset 25772040
     align 2^3 (8)
    reloff 0
    nreloc 0
      type S_NON_LAZY_SYMBOL_POINTERS
attributes (none)
 reserved1 1364 (index into indirect symbol table)
 reserved2 0
Section
  sectname __la_symbol_ptr
   segname __DATA
      addr 0x0000000101895f18
      size 0x0000000000002a98
    offset 25779992
     align 2^3 (8)
    reloff 0
    nreloc 0
      type S_LAZY_SYMBOL_POINTERS
attributes (none)
 reserved1 2358 (index into indirect symbol table)
 reserved2 0
Section
  sectname __mod_init_func
   segname __DATA
      addr 0x00000001018989b0
      size 0x0000000000000168
    offset 25790896
     align 2^3 (8)
    reloff 0
    nreloc 0
      type S_MOD_INIT_FUNC_POINTERS
attributes (none)
 reserved1 0
 reserved2 0
Section
  sectname __const
   segname __DATA
      addr 0x0000000101898b20
      size 0x0000000000156308
    offset 25791264
     align 2^5 (32)
    reloff 0
    nreloc 0
      type S_REGULAR
attributes (none)
 reserved1 0
 reserved2 0
Section
  sectname __data
   segname __DATA
      addr 0x00000001019eee40
      size 0x000000000000c540
    offset 27192896
     align 2^5 (32)
    reloff 0
    nreloc 0
      type S_REGULAR
attributes (none)
 reserved1 0
 reserved2 0
Section
  sectname __static_data
   segname __DATA
      addr 0x00000001019fb380
      size 0x00000000000000ad
    offset 27243392
     align 2^0 (1)
    reloff 0
    nreloc 0
      type S_REGULAR
attributes (none)
 reserved1 0
 reserved2 0
Section
  sectname __gcc_except_tab
   segname __DATA
      addr 0x00000001019fb430
      size 0x00000000000003c8
    offset 27243568
     align 2^3 (8)
    reloff 0
    nreloc 0
      type S_REGULAR
attributes (none)
 reserved1 0
 reserved2 0
Section
  sectname __bss3
   segname __DATA
      addr 0x00000001019fb7f8
      size 0x0000000000001fe8
    offset 0
     align 2^3 (8)
    reloff 0
    nreloc 0
      type S_ZEROFILL
attributes (none)
 reserved1 0
 reserved2 0
Section
  sectname __bss5
   segname __DATA
      addr 0x00000001019fd7e0
      size 0x00000000000441c0
    offset 0
     align 2^5 (32)
    reloff 0
    nreloc 0
      type S_ZEROFILL
attributes (none)
 reserved1 0
 reserved2 0
Section
  sectname __pu_bss3
   segname __DATA
      addr 0x0000000101a419a0
      size 0x0000000000000b10
    offset 0
     align 2^3 (8)
    reloff 0
    nreloc 0
      type S_ZEROFILL
attributes (none)
 reserved1 0
 reserved2 0
Section
  sectname __bss2
   segname __DATA
      addr 0x0000000101a424b0
      size 0x00000000000006e0
    offset 0
     align 2^2 (4)
    reloff 0
    nreloc 0
      type S_ZEROFILL
attributes (none)
 reserved1 0
 reserved2 0
Section
  sectname __pu_bss5
   segname __DATA
      addr 0x0000000101a42ba0
      size 0x00000000001119e0
    offset 0
     align 2^5 (32)
    reloff 0
    nreloc 0
      type S_ZEROFILL
attributes (none)
 reserved1 0
 reserved2 0
Section
  sectname __bss4
   segname __DATA
      addr 0x0000000101b54580
      size 0x0000000000000e30
    offset 0
     align 2^4 (16)
    reloff 0
    nreloc 0
      type S_ZEROFILL
attributes (none)
 reserved1 0
 reserved2 0
Section
  sectname __pu_bss2
   segname __DATA
      addr 0x0000000101b553b0
      size 0x0000000000000214
    offset 0
     align 2^2 (4)
    reloff 0
    nreloc 0
      type S_ZEROFILL
attributes (none)
 reserved1 0
 reserved2 0
Section
  sectname __pu_bss4
   segname __DATA
      addr 0x0000000101b555d0
      size 0x0000000000000120
    offset 0
     align 2^4 (16)
    reloff 0
    nreloc 0
      type S_ZEROFILL
attributes (none)
 reserved1 0
 reserved2 0
Section
  sectname __pu_bss0
   segname __DATA
      addr 0x0000000101b556f0
      size 0x000000000000000f
    offset 0
     align 2^0 (1)
    reloff 0
    nreloc 0
      type S_ZEROFILL
attributes (none)
 reserved1 0
 reserved2 0
Section
  sectname __bss12
   segname __DATA
      addr 0x0000000101b56000
      size 0x0000000040000000 (past end of file)
    offset 0
     align 2^12 (4096)
    reloff 0
    nreloc 0
      type S_ZEROFILL
attributes (none)
 reserved1 0
 reserved2 0
Load command 3
      cmd LC_SEGMENT_64
  cmdsize 72
  segname __LINKEDIT
   vmaddr 0x0000000141b58000
   vmsize 0x0000000000864000
  fileoff 27262976
 filesize 8784880
  maxprot r--
 initprot r--
   nsects 0
    flags (none)
Load command 4
            cmd LC_DYLD_INFO_ONLY
        cmdsize 48
     rebase_off 0
    rebase_size 0
       bind_off 27262976
      bind_size 872
  weak_bind_off 27263848
 weak_bind_size 151024
  lazy_bind_off 27414872
 lazy_bind_size 7376
     export_off 27422248
    export_size 946224
Load command 5
     cmd LC_SYMTAB
 cmdsize 24
  symoff 28426992
   nsyms 208839
  stroff 31783304
 strsize 4264552
Load command 6
            cmd LC_DYSYMTAB
        cmdsize 80
      ilocalsym 0
      nlocalsym 186287
     iextdefsym 186287
     nextdefsym 22170
      iundefsym 208457
      nundefsym 382
         tocoff 0
           ntoc 0
      modtaboff 0
        nmodtab 0
   extrefsymoff 0
    nextrefsyms 0
 indirectsymoff 31768416
  nindirectsyms 3721
      extreloff 0
        nextrel 0
      locreloff 0
        nlocrel 0
Load command 7
          cmd LC_LOAD_DYLINKER
      cmdsize 32
         name /usr/lib/dyld (offset 12)
Load command 8
     cmd LC_UUID
 cmdsize 24
    uuid 38BEAD34-AA41-3548-AE28-6D460EE9EC0D
Load command 9
       cmd LC_BUILD_VERSION
   cmdsize 32
  platform macos
       sdk 10.16
     minos 10.16
    ntools 1
      tool ld
   version 609.7
Load command 10
      cmd LC_SOURCE_VERSION
  cmdsize 16
  version 0.0
Load command 11
       cmd LC_MAIN
   cmdsize 24
  entryoff 18158560
 stacksize 0
Load command 12
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name /usr/lib/libz.1.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 1.2.11
compatibility version 1.0.0
Load command 13
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/lib/libiconv.2.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 7.0.0
compatibility version 7.0.0
Load command 14
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/local/lib/libmpc.3.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 4.0.0
compatibility version 4.0.0
Load command 15
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/local/lib/libmpfr.4.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 6.2.0
compatibility version 6.0.0
Load command 16
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/local/lib/libgmp.10.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 13.0.0
compatibility version 13.0.0
Load command 17
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/lib/libSystem.B.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 1292.0.0
compatibility version 1.0.0
Load command 18
      cmd LC_FUNCTION_STARTS
  cmdsize 16
  dataoff 28368472
 datasize 58520
Load command 19
      cmd LC_DATA_IN_CODE
  cmdsize 16
  dataoff 28426992
 datasize 0

[samuel-emrys]

I've been digging into this for a little bit, and to summarise my findings, some of which I think are implied above but just to be explicit, this recipe has a number of problems:

• If the user isn't using Windows, both Linux and MacOS source archives will be downloaded.
• The mirror it's using to download the sources seems to keep a rolling window of snapshots, so the snapshot referenced no longer exists - recommend moving to a clone of the gcc git repo: https://gcc.gnu.org/git/gcc.git
• This recipe doesn't actually build gfortran, it just copies it from the downloaded sources
• When it copies gfortran, it fails to also copy other relevant binaries, including f951, the actual compiler.
• The recipe makes no attempt to resolve other shared lib dependencies, including libisl, libmpc, libmpfr, libgmp, libdl, libm, libc on linux. These deps appear to be slightly different on macos as above.
• The recipe test package does not adequately test valid performance of the compiler. It runs gfortran --version and passes as long as this doesn't yield a non-zero exit code (presumably? it might just pass regardless of this). This should at least compile and run a basic exemplar program.

Whilst not directly applicable to rectifying this, the Guide to Building gcc/gfortran should be illustrative of the requirements and process of building gfortran successfully. The pre-requisites can be summarised as follows:

• ISO C++11 compiler
• C standard library and headers
• GNAT (not clear if required for gfortran)
• bash
• awk
• GNU binutils (not clear if required for gfortran)
• gzip 1.2.4 or later, or bzip2 1.0.2 or later
• GNU make 3.80 or later
• GNU tar 1.14 or later
• Perl >= 5.6.1 <=5.6.24
• GNU Multiple Precision Library (GMP) 4.3.2 or later
• MFPR 3.1.0 or later
• MPC 1.0.1 or later
• isl 0.15 or later
• zstd

The methods suggested to obtain these are:

• If it provides sufficiently recent versions, use your OS package management system to install the support libraries in standard system locations. For Debian-based systems, including Ubuntu, you should install the packages libgmp-dev, libmpfr-dev and libmpc-dev. For RPM-based systems, including Fedora and SUSE, you should install gmp-devel, mpfr-devel and libmpc-devel (or mpc-devel on SUSE) packages. The packages will install the libraries and headers in standard system directories so they can be found automatically when building GCC.
• Alternatively, after extracting the GCC source archive, simply run the ./contrib/download_prerequisites script in the GCC source directory. That will download the support libraries and create symlinks, causing them to be built automatically as part of the GCC build process. Set GRAPHITE_LOOP_OPT=no in the script if you want to build GCC without ISL, which is only needed for the optional Graphite loop optimizations.
• The difficult way, which is not recommended, is to download the sources for GMP, MPFR and MPC, then configure and install each of them in non-standard locations, then configure GCC with --with-gmp=/some/silly/path/gmp --with-mpfr=/some/silly/path/mpfr --with-mpc=/some/silly/path/mpc, then be forced to set LD_LIBRARY_PATH=/some/silly/path/gmp:/some/silly/path/mpfr:/some/silly/path/mpc/lib in your environment forever. This is silly and causes major problems for anyone who doesn't understand how dynamic linkers find libraries at runtime. Do not do this. If building GCC fails when using any of the --with-gmp or --with-mpfr or --with-mpc options then you probably shouldn't be using them.

It strikes me that the "difficult way" is probably what we should aim to achieve here? Mostly because this gives us proper isolation of the build artefacts from system files and conan would be responsible for populating those flags/variables when building and linking. The question is though, how far through the above list of requirements do we go before considering that the easiest/best solution here may just be to download pre-built binaries or expectation that the compiler/pre-requisite libs are pre-installed on the system? I know that a working fortran compiler is a blocker on a number of existing recipes, including:

• Lapack supprt in OpenBLAS
• Armadillo
• ATLAS
• Himalaya
• lapack

[uilianries]

Conan Docker Tools provides new GCC images with fortran support.

[samuel-emrys]

Conan Docker Tools provides new GCC images with fortran support.

To clarify, it sounds like you mean that it's fine to specify gfortran as a system dependency because the CI now ships with it? If this is the case, perhaps it would be worth updating KB-H032 to reflect exceptions to this rule?

[samuel-emrys]

In a conversation with @uilianries on slack, this is the current state of a gfortran compiler on the CI machines:

gfortran package is broken, should use conanio/gcc10-ubuntu16.04 image or similar
No [there is no way to force conan index CI to use the conanio/gcc10-ubuntu16.04 image], because they are incompatible with all packages available due glibc and libstdc++ used
Yes [there are plans to make gfortran available on the CI runners], in some months. It will require a big effort, as we will need to rebuild many packages