[R-sig-hpc] Installing R-3.3 on a HPC cluster

[Ali Siavosh-Haghighi]

Hi All,
I recently started having problem installing R using the source code on our cluster. The cluster runs on CentOS 6.4 and
[***@phoenix2 siavoa01]# uname -a
Linux phoenix2 2.6.32-573.22.1.el6.x86_64 #1 SMP Wed Mar 23 03:35:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

It sucks it the following step inspire the fact that I installed the latest bzip2 available and the include/ and lib/ of it is already included in $CPATH and $LIB (I tried the R-3.3 and also previous R-devel):
checking if bzip2 version >= 1.0.6... no
checking whether bzip2 support suffices... configure: error: bzip2 library and headers are required
[***@phoenix2 R-3.3.0]# which bzip2
/local/apps/bzip2/1.0.6/bin/bzip2
[***@phoenix2 R-3.3.0]# bzip2 -V
bzip2, a block-sorting file compressor. Version 1.0.6, 6-Sept-2010.

Copyright (C) 1996-2010 by Julian Seward.

This program is free software; you can redistribute it and/or modify
it under the terms set out in the LICENSE file, which is included
in the bzip2-1.0.6 source distribution.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
LICENSE file for more details.

bzip2: I won't write compressed data to a terminal.
bzip2: For help, type: `bzip2 --help'.

==============================================================
Ali Siavosh-Haghighi, Ph.D.
HPC System Administrator
High Performance Computing Facility
Center for Health Informatics and Bioinformatics
NYU Langone Medical Center
Phone: [phone]
http://www.med.nyu.edu/chibi/services/hpcf
==============================================================



[[alternative HTML version deleted]]

[R P Herrold]

Post by Ali Siavosh-Haghighi
checking whether bzip2 support suffices... configure: error: bzip2 library and headers are required

install: bzip2-devel using yum

-- Russ herrold

[Ali Siavosh-Haghighi]

Thanks for your help. It needs newer glibc and I rather not to mess with it on our cluster. Is there a work around?
==============================================================
Ali Siavosh-Haghighi, Ph.D.
HPC System Administrator
High Performance Computing Facility
Center for Health Informatics and Bioinformatics
NYU Langone Medical Center
Phone: [phone]
http://www.med.nyu.edu/chibi/services/hpcf
==============================================================

Post by R P Herrold

Post by Ali Siavosh-Haghighi
checking whether bzip2 support suffices... configure: error: bzip2 library and headers are required

install: bzip2-devel using yum
-- Russ herrold

[[alternative HTML version deleted]]

[Simon Urbanek]

Post by Ali Siavosh-Haghighi
Thanks for your help. It needs newer glibc and I rather not to mess with it on our cluster. Is there a work around?

You need libbz2 devel files (and BTW that's just one hurdle, you also need xz, pcre etc.). If you have issues installing with yum, then it' likely a problem with your setup (i.e. you're trying an upgrade instead of installing the versions present in your system). Just fetch the devel versions that correspond to the libraries installed in the system.

Cheers,
Simo

Post by Ali Siavosh-Haghighi
==============================================================
Ali Siavosh-Haghighi, Ph.D.
HPC System Administrator
High Performance Computing Facility
Center for Health Informatics and Bioinformatics
NYU Langone Medical Center
Phone: [phone]
http://www.med.nyu.edu/chibi/services/hpcf
==============================================================

Post by R P Herrold

Post by Ali Siavosh-Haghighi
checking whether bzip2 support suffices... configure: error: bzip2 library and headers are required

install: bzip2-devel using yum
-- Russ herrold

[[alternative HTML version deleted]]
_______________________________________________
R-sig-hpc mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-hpc

[R P Herrold]

Post by Ali Siavosh-Haghighi
Thanks for your help. It needs newer glibc and I rather not
to mess with it on our cluster. Is there a work around?

actually ./configure then dies due to wanting a later zlib
version than is available

checking if zlib version >= 1.2.5... no
checking whether zlib support suffices... configure: error:
zlib library and headers are required

[***@centos-6 ~]# rpm -qa zlib\*
zlib-devel-1.2.3-29.el6.x86_64
zlib-1.2.3-29.el6.x86_64

There is not any easy workaround within the four corners of
RHEL / CentOS 6 series. a 'difficult' workaround is the 'scl'
extensions to permit carrying around later libraries on a
binary by binary basis, later package versions along side of
the 'normal' ones present, but this is out of scope here --
see:
https://wiki.centos.org/AdditionalResources/Repositories/SCL

As I recall there is a miling list, or failing that, the
centos-devel mailing list (referenced at the foot of that
page) would be monitored by people who 'live and breathe'
those picky details

-- Russ herrold

[Ali Siavosh-Haghighi]

I did install zlib as a module and bypassed that error but bzip2-devel-1.0.6 asks for some new version of glibc that is a bit more fundamental. That glibc and that bzip2 It is default for CentOS 7. So I was thinking if there is a way to install bzip2-devel as module as well but could not find the source.
==============================================================
Ali Siavosh-Haghighi, Ph.D.
HPC System Administrator
High Performance Computing Facility
Center for Health Informatics and Bioinformatics
NYU Langone Medical Center
Phone: [phone]
http://www.med.nyu.edu/chibi/services/hpcf
==============================================================

Post by R P Herrold

Post by Ali Siavosh-Haghighi
Thanks for your help. It needs newer glibc and I rather not
to mess with it on our cluster. Is there a work around?

actually ./configure then dies due to wanting a later zlib
version than is available
checking if zlib version >= 1.2.5... no
zlib library and headers are required
zlib-devel-1.2.3-29.el6.x86_64
zlib-1.2.3-29.el6.x86_64
There is not any easy workaround within the four corners of
RHEL / CentOS 6 series. a 'difficult' workaround is the 'scl'
extensions to permit carrying around later libraries on a
binary by binary basis, later package versions along side of
the 'normal' ones present, but this is out of scope here --
https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.centos.org_AdditionalResources_Repositories_SCL&d=CwIBAg&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=XaOCmLUnAz9o9ru8VyC-GVRS6cJNKKW6WtTDkj1afBA&m=A9KCS26yvhYAa8S1htHAnv_gd4Zk8-bpI507tPXMwXg&s=XT54HpXs7Gl_z0wp9qIoGloJP55Bodf9zM91QLVjyBA&e=
As I recall there is a miling list, or failing that, the
centos-devel mailing list (referenced at the foot of that
page) would be monitored by people who 'live and breathe'
those picky details
-- Russ herrold

[[alternative HTML version deleted]]

[Bjørn-Helge Mevik]

Post by Ali Siavosh-Haghighi
I did install zlib as a module and bypassed that error but bzip2-devel-1.0.6
asks for some new version of glibc that is a bit more fundamental. That glibc
and that bzip2 It is default for CentOS 7. So I was thinking if there is a way
to install bzip2-devel as module as well but could not find the
source.

I guess there should be no problem in downloading the latest bzip2
source code, build it and install it and create a module file for it.
We recently had to do this for libcurl because the OS provided version
was too old (didn't support https).

If you google "bzip source" you will find that the source code can be
found here: http://www.bzip.org/

[Ali Siavosh-Haghighi]

Hi Bjorn,
Installing the bzip2 itself is not the problem. R>3.2.5 requires one of the headers in bzip2-devel. There is no source for bzip2-devel-1.0.6 (could not find it and I contacted the developer and no response yet). The RPM installation complains about lack of newer version of glibc (a version that is the default for CentOS 7). So, unless R be wrapped with the necessary headers of bzip2-devel I dont see an easy installation for new versions of R on CentOS 6. And we have very heavy R users.
I appreciate any helpful comment
==============================================================
Ali Siavosh-Haghighi, Ph.D.
HPC System Administrator
High Performance Computing Facility
Center for Health Informatics and Bioinformatics
NYU Langone Medical Center
Phone: [phone]
http://www.med.nyu.edu/chibi/services/hpcf
==============================================================

Post by Bjørn-Helge Mevik

Post by Ali Siavosh-Haghighi
I did install zlib as a module and bypassed that error but bzip2-devel-1.0.6
asks for some new version of glibc that is a bit more fundamental. That glibc
and that bzip2 It is default for CentOS 7. So I was thinking if there is a way
to install bzip2-devel as module as well but could not find the
source.

I guess there should be no problem in downloading the latest bzip2
source code, build it and install it and create a module file for it.
We recently had to do this for libcurl because the OS provided version
was too old (didn't support https).
If you google "bzip source" you will find that the source code can be
found here: https://urldefense.proofpoint.com/v2/url?u=http-3A__www.bzip.org_&d=CwIGaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=XaOCmLUnAz9o9ru8VyC-GVRS6cJNKKW6WtTDkj1afBA&m=Im1-S_Gpzb4vbMNNYwsYReogsNXaH_YHZYF1e0OB0Gc&s=bWQk-f2kOA-fgOmk0fluNBVwcjM2j7pU7EUz2gk4mTk&e=
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
_______________________________________________
R-sig-hpc mailing list
https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_r-2Dsig-2Dhpc&d=CwIGaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=XaOCmLUnAz9o9ru8VyC-GVRS6cJNKKW6WtTDkj1afBA&m=Im1-S_Gpzb4vbMNNYwsYReogsNXaH_YHZYF1e0OB0Gc&s=3a19QMRAhuVX0jnFBslLjhWnQQH-oFDg0OgB43bcJJM&e=

[[alternative HTML version deleted]]

[Kasper Daniel Hansen]

Ali, I don't know how CentOS 6 relates to RHEL 6, but I have installed R
3.3 from source (by hand) on RHEL 6. It required installing a number of
packages by hand, including bzip2 (1.0.6), xz (5.2.1), pcre (8.36), curl
(7.43.0) and possibly other packages I have forgotten. But not glibc. My
sysadmins only install stuff from the official Red hat sources. We have

gcc --version
gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4)
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Best,
Kasper

Post by Ali Siavosh-Haghighi
Hi Bjorn,
Installing the bzip2 itself is not the problem. R>3.2.5 requires one of
the headers in bzip2-devel. There is no source for bzip2-devel-1.0.6 (could
not find it and I contacted the developer and no response yet). The RPM
installation complains about lack of newer version of glibc (a version that
is the default for CentOS 7). So, unless R be wrapped with the necessary
headers of bzip2-devel I dont see an easy installation for new versions of
R on CentOS 6. And we have very heavy R users.
I appreciate any helpful comment
==============================================================
Ali Siavosh-Haghighi, Ph.D.
HPC System Administrator
High Performance Computing Facility
Center for Health Informatics and Bioinformatics
NYU Langone Medical Center
Phone: [phone]
http://www.med.nyu.edu/chibi/services/hpcf
==============================================================

Post by Bjørn-Helge Mevik

Post by Ali Siavosh-Haghighi
I did install zlib as a module and bypassed that error but

bzip2-devel-1.0.6

Post by Bjørn-Helge Mevik

Post by Ali Siavosh-Haghighi
asks for some new version of glibc that is a bit more fundamental. That

glibc

Post by Bjørn-Helge Mevik

Post by Ali Siavosh-Haghighi
and that bzip2 It is default for CentOS 7. So I was thinking if there

is a way

Post by Bjørn-Helge Mevik

Post by Ali Siavosh-Haghighi
to install bzip2-devel as module as well but could not find the
source.

I guess there should be no problem in downloading the latest bzip2
source code, build it and install it and create a module file for it.
We recently had to do this for libcurl because the OS provided version
was too old (didn't support https).
If you google "bzip source" you will find that the source code can be

https://urldefense.proofpoint.com/v2/url?u=http-3A__www.bzip.org_&d=CwIGaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=XaOCmLUnAz9o9ru8VyC-GVRS6cJNKKW6WtTDkj1afBA&m=Im1-S_Gpzb4vbMNNYwsYReogsNXaH_YHZYF1e0OB0Gc&s=bWQk-f2kOA-fgOmk0fluNBVwcjM2j7pU7EUz2gk4mTk&e=

Post by Bjørn-Helge Mevik
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
_______________________________________________
R-sig-hpc mailing list

https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_r-2Dsig-2Dhpc&d=CwIGaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=XaOCmLUnAz9o9ru8VyC-GVRS6cJNKKW6WtTDkj1afBA&m=Im1-S_Gpzb4vbMNNYwsYReogsNXaH_YHZYF1e0OB0Gc&s=3a19QMRAhuVX0jnFBslLjhWnQQH-oFDg0OgB43bcJJM&e=
[[alternative HTML version deleted]]
_______________________________________________
R-sig-hpc mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-hpc

[[alternative HTML version deleted]]

[Bjørn-Helge Mevik]

Post by Ali Siavosh-Haghighi
Hi Bjorn,
Installing the bzip2 itself is not the problem. R>3.2.5 requires one of the
headers in bzip2-devel. There is no source for bzip2-devel-1.0.6 (could not
find it and I contacted the developer and no response yet).

The separation of software into "package" and "package-devel" RPMs is
purely a desicion of the RPM package maintainer (and similar for debs).
When you install something from source, in almost all cases, you will
get the header files needed for compiling against it installed as well,
so you only need to set the LIBRARY_PATH and CPATH (and possibly
FPATH/CXXPATH) to compile against it, and LD_LIBRARY_PATH to use it
later. (Typically, I'd do this in a module file.)

[Sebastian Luna Valero]

Hi All,

Adapting the installation instructions on the following blog post worked on
our RHEL6 environment:
http://pj.freefaculty.org/blog/?p=315

I hope it helps!

Best regards,
Sebastian
​

[[alternative HTML version deleted]]

[Ali Siavosh-Haghighi]

Hi All,
For cluster running on CentOS-6 compiled and installed the following using gcc-5.2.0:

1. bzip2-1.0.6 (compiling with shared library option, then make a link in lib from libbz2.so.1.0.6 to libbz2.so.1)
2. zlib-1.2.8
3. XZ-5.2.2 (needed to make a link from liblzma.so.5.2.2 to liblzma.so.5.2, this particular link was not there)
4. pcre-8.38
5. curl-7.48.0

All the LD_LIBRARY_PATH, CPATH, LIBRARY_PATH and PATH of course needed to be edited to prepend paths to the library and the headers of the packages above.
Then ./configure went through (I disabled openMP option).
In previous releases (3.2) there has been a directory /src/extra containing at least some of these packages. For 3.3 the necessary packages are not there (possibly for license issue?) and thus gone smooth installation.
Regards
==============================================================
Ali Siavosh-Haghighi, Ph.D.
HPC System Administrator
High Performance Computing Facility
Center for Health Informatics and Bioinformatics
NYU Langone Medical Center
Phone: [phone]
http://www.med.nyu.edu/chibi/services/hpcf
==============================================================

Post by Ali Siavosh-Haghighi
Hi All,
Adapting the installation instructions on the following blog post worked on
https://urldefense.proofpoint.com/v2/url?u=http-3A__pj.freefaculty.org_blog_-3Fp-3D315&d=CwIGaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=XaOCmLUnAz9o9ru8VyC-GVRS6cJNKKW6WtTDkj1afBA&m=WHdhsE7rCGbWj0FHhGkyjj4KZkJ9joPRcGklAmigMbs&s=QlSqIZ9VLFuqCZpJnlWYNpMRgYSOn9uLcpHXsQTNVv4&e=
I hope it helps!
Best regards,
Sebastian
​
[[alternative HTML version deleted]]
_______________________________________________
R-sig-hpc mailing list
https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_r-2Dsig-2Dhpc&d=CwIGaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=XaOCmLUnAz9o9ru8VyC-GVRS6cJNKKW6WtTDkj1afBA&m=WHdhsE7rCGbWj0FHhGkyjj4KZkJ9joPRcGklAmigMbs&s=ZGxC1PcbFyZVllAK6wkWLp2lV1pNorBofi83JbmGNOI&e=

[[alternative HTML version deleted]]