(2022-12-20) Lab Notebook: Conda --- Cleaning out your cache on HPCC
Warning
This is as a Lab Notebook which describes how to solve a specific problem at a specific time. Please keep this in mind as you read and use the content. Please pay close attention to the date, version information and other details.
Lab Notebook --- Cleaning out your Conda Cache on HPCC (2022-12-20)
Warning
EXTRA WARNING: This process is MOSTLY safe in that the default behavior of conda is to hardlink from environments to the packages, so if packages are removed from the conda cache, it will not affect the environment. However, if you have enable softlinking with conda (by setting allow_softlinks or always_softlinks in your config) or manually soflinked to files/folders in the packages directory, this WILL break those links. For a discussion of why this process is MOSTLY safe, see the following links:
Conda often creates many small files contributing to user's file quota on HPCC. A more permanent solution for this problem can be found using Singularity overlays, but as of the time of writing, Singularity overlays remains something of a work in progress. Hence, this guide is meant to provide an interim solution by outlining ways of cleaning out some of the extraneous files produced by Conda.
Conda clean
The command 'conda clean' can be used to clean out unneeded packages in the 'pkgs' cache folder. Full options for this command can be found here, but the simplest options are to either remove all unused packages with:
conda clean -p
Or remove ALL packages with:
conda clean -a
Again, this should have no affect on hard-link environments installed under normal circumstances, however if there is any doubts consider making a backup of existing, important python environments as .yaml files which can be used to reinstalling environments or other options to reduce file count such as archiving old/unused files and directories.