[R-sig-hpc] Loading rgdal as PBS script in R

[Hariom Singh]

Hey all,

I thought I'd share this since we spent quite some time with it and it
might be relevant to some of you. Or at least amusing, possibly.

It is stated that, I am getting difficulty in loading rgdal R library while
using PBS script for parallel computing environment....

First of all I would like to share my relevant files....Please check the
error..

*##### test.sh pbs script file for running R program*

#!/usr/bin/env bash

#PBS -N snow_1
#PBS -m abe
#PBS -q medium
#PBS -l nodes=2:ppn=12
#PBS -V

cd $PBS_O_WORKDIR


cp /UHOME/hariodce/R/x86_64-unknown-linux-gnu-library/3.2/Rmpi/Rprofile
./.Rprofile

cat $PBS_NODEFILE > pbsnodes

mpirun -machinefile $PBS_NODEFILE -np 24 R CMD BATCH *test.r *

*#### test.r ... test R file where rgdal is not loaded........*

# Load the R MPI package if it is not already loaded.

if (is.loaded("mpi_initialize"))

{

library("Rmpi")
*library("rgdal")......*

}
# In case R exits unexpectedly, have it automatically clean up
# resources taken up by Rmpi (slaves, memory, etc...)
.Last <- function(){
if (is.loaded("mpi_initialize")){
if (mpi.comm.size(1) > 0){
print("Please use mpi.close.Rslaves() to close slaves.")
mpi.close.Rslaves()
}
print("Please use mpi.quit() to quit R")
.Call("mpi_finalize")

}
}
# Tell all slaves to return a message identifying themselves
mpi.remote.exec(paste("I am",
mpi.comm.rank(),"of",mpi.comm.size(),system("hostname",intern=T)))

# Tell all slaves to close down, and exit the program

mpi.close.Rslaves()

*output file......*
..................
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

master (rank 0 , comm 1) of size 24 is running on: cn05
slave1 (rank 1 , comm 1) of size 24 is running on: cn05
slave2 (rank 2 , comm 1) of size 24 is running on: cn05
slave3 (rank 3 , comm 1) of size 24 is running on: cn05
... ... ...
slave22 (rank 22, comm 1) of size 24 is running on: cn07
slave23 (rank 23, comm 1) of size 24 is running on: cn07
[Previously saved workspace restored]

# Load the R MPI package if it is not already loaded.
if (is.loaded("mpi_initialize"))

+
+ {
+
+ library("Rmpi")
+ library("rgdal")
+
+ } ########error...*.it always loaded without pbs scipt like
library(rgdal) but here it is not loaded.....so what is the problem in
loading.......Plz help me out.*




*Loading required package: spError in dyn.load(file, DLLpath = DLLpath,
...) : unable to load shared object
'/SOFT/R320/lib64/R/library/rgdal/libs/rgdal.so': libgdal.so.20: cannot
open shared object file: No such file or directory*
Error: package or namespace load failed for ‘rgdal’

# In case R exits unexpectedly, have it automatically clean up
# resources taken up by Rmpi (slaves, memory, etc...)
.Last <- function(){

+ if (is.loaded("mpi_initialize")){
+ if (mpi.comm.size(1) > 0){
+ print("Please use mpi.close.Rslaves() to close slaves.")
+ mpi.close.Rslaves()
+ }
+ print("Please use mpi.quit() to quit R")
+ .Call("mpi_finalize")
+
+ }
+ }

# Tell all slaves to return a message identifying themselves
mpi.remote.exec(paste("I am",

mpi.comm.rank(),"of",mpi.comm.size(),system("hostname",intern=T)))
$slave1
[1] "I am 1 of 24 cn05"

$slave2
[1] "I am 2 of 24 cn05"

.......................
.............................


*# .bashrc *

# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi

export R_LIBS_USER=/SOFT/R320/lib64/R/library


*############## .bash_profile*

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH
source /SOFT/ics_2013.1.046/icsxe/2013.1.046/ictvars.sh intel64
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64
export PATH=/SOFT/R320/bin/:/SOFT/mpich-3.1.1/bin:$PATH
export
LD_LIBRARY_PATH=/SOFT/R320/lib64/:/SOFT/mpich-3.1.1/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

Please help me. I would highly appreciate for your time and concern.

Thanks in anticipation.

Hariom Singh

Ph.D research scholar , IIT Roorkee.

[[alternative HTML version deleted]]

[Ei-ji Nakama]

Hi, suddenly remember...

1) When you build in intel Compiler, did you properly configured AR,LD, etc.?

Post by Hariom Singh
source /SOFT/ics_2013.1.046/icsxe/2013.1.046/ictvars.sh intel64

cf. https://software.intel.com/en-us/articles/build-r-301-with-intel-c-compiler-and-intel-mkl-on-linux

intel obj link in the of gnu-ar and gnu-ld to fail(but build time no
error report).
not good mixed tool-chain. please all re-build with IntelCompiler
(many librarys) or
not use IntelCompiler.

2) maybe start process in rsh(or ssh)?...# not under PBS process.

Post by Hariom Singh
mpirun -machinefile $PBS_NODEFILE -np 24 R CMD BATCH *test.r *

cf. https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager#Resource_Managers_and_Launchers

CAUTION : I do not know all the combinations on cluster...

Post by Hariom Singh
Hey all,
I thought I'd share this since we spent quite some time with it and it
might be relevant to some of you. Or at least amusing, possibly.
It is stated that, I am getting difficulty in loading rgdal R library while
using PBS script for parallel computing environment....
First of all I would like to share my relevant files....Please check the
error..
*##### test.sh pbs script file for running R program*
#!/usr/bin/env bash
#PBS -N snow_1
#PBS -m abe
#PBS -q medium
#PBS -l nodes=2:ppn=12
#PBS -V
cd $PBS_O_WORKDIR
cp /UHOME/hariodce/R/x86_64-unknown-linux-gnu-library/3.2/Rmpi/Rprofile
./.Rprofile
cat $PBS_NODEFILE > pbsnodes
mpirun -machinefile $PBS_NODEFILE -np 24 R CMD BATCH *test.r *
*#### test.r ... test R file where rgdal is not loaded........*
# Load the R MPI package if it is not already loaded.
if (is.loaded("mpi_initialize"))
{
library("Rmpi")
*library("rgdal")......*
}
# In case R exits unexpectedly, have it automatically clean up
# resources taken up by Rmpi (slaves, memory, etc...)
.Last <- function(){
if (is.loaded("mpi_initialize")){
if (mpi.comm.size(1) > 0){
print("Please use mpi.close.Rslaves() to close slaves.")
mpi.close.Rslaves()
}
print("Please use mpi.quit() to quit R")
.Call("mpi_finalize")
}
}
# Tell all slaves to return a message identifying themselves
mpi.remote.exec(paste("I am",
mpi.comm.rank(),"of",mpi.comm.size(),system("hostname",intern=T)))
# Tell all slaves to close down, and exit the program
mpi.close.Rslaves()
*output file......*
..................
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
master (rank 0 , comm 1) of size 24 is running on: cn05
slave1 (rank 1 , comm 1) of size 24 is running on: cn05
slave2 (rank 2 , comm 1) of size 24 is running on: cn05
slave3 (rank 3 , comm 1) of size 24 is running on: cn05
... ... ...
slave22 (rank 22, comm 1) of size 24 is running on: cn07
slave23 (rank 23, comm 1) of size 24 is running on: cn07
[Previously saved workspace restored]

Post by Hariom Singh
# Load the R MPI package if it is not already loaded.
if (is.loaded("mpi_initialize"))

+
+ {
+
+ library("Rmpi")
+ library("rgdal")
+
+ } ########error...*.it always loaded without pbs scipt like
library(rgdal) but here it is not loaded.....so what is the problem in
loading.......Plz help me out.*
*Loading required package: spError in dyn.load(file, DLLpath = DLLpath,
...) : unable to load shared object
'/SOFT/R320/lib64/R/library/rgdal/libs/rgdal.so': libgdal.so.20: cannot
open shared object file: No such file or directory*
Error: package or namespace load failed for ‘rgdal’

Post by Hariom Singh
# In case R exits unexpectedly, have it automatically clean up
# resources taken up by Rmpi (slaves, memory, etc...)
.Last <- function(){

+ if (is.loaded("mpi_initialize")){
+ if (mpi.comm.size(1) > 0){
+ print("Please use mpi.close.Rslaves() to close slaves.")
+ mpi.close.Rslaves()
+ }
+ print("Please use mpi.quit() to quit R")
+ .Call("mpi_finalize")
+
+ }
+ }

Post by Hariom Singh
# Tell all slaves to return a message identifying themselves
mpi.remote.exec(paste("I am",

mpi.comm.rank(),"of",mpi.comm.size(),system("hostname",intern=T)))
$slave1
[1] "I am 1 of 24 cn05"
$slave2
[1] "I am 2 of 24 cn05"
.......................
.............................
*# .bashrc *
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
export R_LIBS_USER=/SOFT/R320/lib64/R/library
*############## .bash_profile*
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
source /SOFT/ics_2013.1.046/icsxe/2013.1.046/ictvars.sh intel64
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64
export PATH=/SOFT/R320/bin/:/SOFT/mpich-3.1.1/bin:$PATH
export
LD_LIBRARY_PATH=/SOFT/R320/lib64/:/SOFT/mpich-3.1.1/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
Please help me. I would highly appreciate for your time and concern.
Thanks in anticipation.
Hariom Singh
Ph.D research scholar , IIT Roorkee.
[[alternative HTML version deleted]]
_______________________________________________
R-sig-hpc mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-hpc

[Roger Bivand]

Post by Hariom Singh
Hey all,
I thought I'd share this since we spent quite some time with it and it
might be relevant to some of you. Or at least amusing, possibly.
It is stated that, I am getting difficulty in loading rgdal R library while
using PBS script for parallel computing environment....
+ library("rgdal")
+
+ } ########error...*.it always loaded without pbs scipt like
library(rgdal) but here it is not loaded.....so what is the problem in
loading.......Plz help me out.*
*Loading required package: spError in dyn.load(file, DLLpath = DLLpath,
...) : unable to load shared object
'/SOFT/R320/lib64/R/library/rgdal/libs/rgdal.so': libgdal.so.20: cannot
open shared object file: No such file or directory*
Error: package or namespace load failed for ‘rgdal’

The cause seems to be very obvious:

unable to load shared object
'/SOFT/R320/lib64/R/library/rgdal/libs/rgdal.so': libgdal.so.20: cannot
open shared object file: No such file or directory

On a single core/node, the answer would be to (re)-run ldconfig and/or make
sure the underlying shared object to which rgdal links dynamically is
visible to your R process. If rgdal runs correctly on one node and not
others, then your setup is the problem, if nowhere, rgdal may have been
installed against a different GDAL installation than the one seen in the
search path. libgdal.so.20 (if that was the one rgdal.so was built against)
needs to be visible everywhere.

Roger

Post by Hariom Singh
Thanks in anticipation.
Hariom Singh
Ph.D research scholar , IIT Roorkee.