The UCT HPC cluster has a wide range of codes installed. However occasionally these need to be extended to add additional functionality by adding packages or libraries. In order to keep the base install clean we require researchers to add these packages or libraries locally as they do not have write access to /opt/exp_soft, the central software volume.
The methods outlined below induce high load on the server they are run on due to the fact that code compilation occurs. Please do not do this on the head node, rather start an interactive job and then exit the job once the packages are installed. To start an interactive job:
sint --account=<your_account> --time=10:00:00 --partition=ada --ntasks=1 --nodes=1
To install packages locally the following method can be used:
Start an interactive job and activate the version of R you require with module
module load software/R-3.6.0
Start R and install the package
You will immediately be challenged by the installer as you do not have write rights to the software location
Warning in install.packages("XYZ") : 'lib = "/opt/exp_soft/R-3.6.0/lib64"' is not writable Would you like to use a personal library instead? (yes/No/cancel)
Enter yes, the package will then download, compile and install. You can then load the library.
Once this is confirmed as working you no longer need to install the package each time you start R, just load the library.
The pip package management tool can be used to install packages locally.
module load python/anaconda-python-3.7 pip install --user PackageName
If a version of the package you’re trying to install exists in /opt/exp_soft already then pip will not install locally as the package already exists from its point of view. To get around this add the –ignore-installed argument.
pip install --user PackageName --ignore-installed
Python\Anaconda virtual environment
Python has a specific tool used to install packages, conda. Python also comes with a mechanism to set up virtual environments. Below is an example of setting up the short read aligner bowtie:
Start an interactive job, then:
module load python/anaconda-python-3.7 conda create -y -n RelevantName source activate RelevantName conda config --add channels defaults conda config --add channels bioconda conda config --add channels conda-forge conda install bowtie conda deactivate exit
The above only needs to be done once. The term RelevantName should be a short text word describing your project\environment. Additionally the channel names are for bowtie specifically. Your package may require different channels. Now when you launch jobs your script should contain:
module load python/anaconda-python-3.7 source activate RelevantName
Python\Anaconda Singularity Containers
I’m using a Singularity container and can’t add python libraries as conda\pip was not added to the Singularity container by the person who created it. (#21stCenturyProblems)
You’re going to need to make pip available. Start an interactive job and then run:
module load [NameOfModuleForSingularityContainer] wget https://bootstrap.pypa.io/get-pip.py python get-pip.py --user
You have now installed your own version of pip which you can use with the singularity version of python. Now you can add your own libraries, for example to install Pandas run:
python -m pip install Pandas --user
Now you can run python and import the library:
python Python [VER] >>> import pandas as pd >>>
The libraries will be installed into $HOME/.local/lib/python[VER]/site-packages
where [VER] is the version of python in the Singularity container.