(2022-12-20) Lab Notebook: Conda --- Cleaning out your cache on HPCC

Warning

This is as a Lab Notebook which describes how to solve a specific problem at a specific time. Please keep this in mind as you read and use the content. Please pay close attention to the date, version information and other details.

Lab Notebook --- Cleaning out your Conda Cache on HPCC (2022-12-20)

Warning

EXTRA WARNING: This process is MOSTLY safe in that the default behavior of conda is to hardlink from environments to the packages, so if packages are removed from the conda cache, it will not affect the environment. However, if you have enable softlinking with conda (by setting allow_softlinks or always_softlinks in your config) or manually soflinked to files/folders in the packages directory, this WILL break those links. For a discussion of why this process is MOSTLY safe, see the following links:

• "Is it safe to manually delete all files in pkgs folder in anaconda python?" (Stackoverflow)
• "Can I delete files in the pkgs directory after installation." (Anaconda Google Group)

Conda often creates many small files contributing to user's file quota on HPCC. A more permanent solution for this problem can be found using Singularity overlays, but as of the time of writing, Singularity overlays remains something of a work in progress. Hence, this guide is meant to provide an interim solution by outlining ways of cleaning out some of the extraneous files produced by Conda.

Conda clean

The command 'conda clean' can be used to clean out unneeded packages in the 'pkgs' cache folder. Full options for this command can be found here, but the simplest options are to either remove all unused packages with:

conda clean -p

Or remove ALL packages with:

conda clean -a

Again, this should have no affect on hard-link environments installed under normal circumstances, however if there is any doubts consider making a backup of existing, important python environments as .yaml files which can be used to reinstalling environments or other options to reduce file count such as archiving old/unused files and directories.