Adding libraries

Home Adding libraries

The UCT HPC cluster has a wide range of codes installed. However occasionally these need to be extended to add additional functionality by adding packages or libraries. In order to keep the base install clean we require researchers to add these packages or libraries locally as they do not have write access to /opt/exp_soft, the central software volume.

Important
The methods outlined below induce high load on the server they are run on due to the fact that code compilation occurs. Please do not do this on the head node, rather start an interactive job and then exit the job once the packages are installed. To start an interactive job:

sint --account=<your_account> --time=10:00:00 --partition=ada --ntasks=1 --nodes=1

 

R

To install packages locally the following method can be used:

Start an interactive job and activate the version of R you require with module

module load software/R-3.6.0

Start R and install the package

install.packages("XYZ")

You will immediately be challenged by the installer as you do not have write rights to the software location

Warning in install.packages("XYZ") :
'lib = "/opt/exp_soft/R-3.6.0/lib64"' is not writable
Would you like to use a personal library instead? (yes/No/cancel)

Enter yes, the package will then download, compile and install. You can then load the library.

library(XYZ)

Once this is confirmed as working you no longer need to install the package each time you start R, just load the library.

 

Python\Anaconda pip

The pip package management tool can be used to install packages locally.

module load python/anaconda-python-3.7
pip install --user PackageName

If a version of the package you’re trying to install exists in /opt/exp_soft already then pip will not install locally as the package already exists from its point of view. To get around this add the –ignore-installed argument.

pip install --user PackageName --ignore-installed

 

Python\Anaconda virtual environment

Python has a specific tool used to install packages, conda. Python also comes with a mechanism to set up virtual environments. Below is an example of setting up the short read aligner bowtie:

Start an interactive job, then:

module load python/anaconda-python-3.7
conda create -y -n RelevantName
source activate RelevantName
conda config --add channels defaults
conda config --add channels bioconda
conda config --add channels conda-forge
conda install bowtie
conda deactivate
exit

The above only needs to be done once. The term RelevantName should be a short text word describing your project\environment. Additionally the channel names are for bowtie specifically. Your package may require different channels. Now when you launch jobs your script should contain:

module load python/anaconda-python-3.7
source activate RelevantName

 

Python\Anaconda Singularity Containers

I’m using a Singularity container and can’t add python libraries as conda\pip was not added to the Singularity container by the person who created it. (#21stCenturyProblems)

You’re going to need to make pip available. Start an interactive job and then run:

module load [NameOfModuleForSingularityContainer]
wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py --user

You have now installed your own version of pip which you can use with the singularity version of python. Now you can add your own libraries, for example to install Pandas run:

python -m pip install Pandas --user

Now you can run python and import the library:

python
Python [VER]
>>> import pandas as pd
>>>

The libraries will be installed into $HOME/.local/lib/python[VER]/site-packages

where [VER] is the version of python in the Singularity container.

Menu