(2022-12-20) Lab Notebook: Anaconda --- Cleanning out your cache on HPCC

Warning

This is as a Lab Notebook which describes how to solve a specific problem at a specific time. Please keep this in mind as you read and use the content. Please pay close attention to the date, version information and other details.

Lab Notebook --- Cleanning out your Anaconda Cache on HPCC (2022-12-20)

Warning

EXTRA WARNING: This process is MOSTLY safe in that the deafult behavior of conda is to hardlink from enviroments to the packages, so if packages are removed from the conda cache, it will not affect the environment. However, if you have enable softlinking with conda (by setting allow_softlinks or always_softlinks in your config) or manually soflinked to files/folders in the packages directory, this WILL break those links. For a discussion of why this process is MOSTLY safe, see the following links:

• "Is it safe to manually delete all files in pkgs folder in anaconda python?" (Stackoverflow)
• "Can I delete files in the pkgs directory after installation." (Anaconda Google Group)

Anaconda often creates many small files contributing to user's file quota on HPCC. A more permenant solution for this problem can be found using Singularity overlays, but as of the time of writing, Sigularity overlays remains something of a work in progress. Hence, this guide is meant to provide an interm solution by outling ways of cleaning out some of the extraneous files produced by Anaconda.

Conda clean

The command 'conda clean' can be used to clean out uneeded packages in the 'pkgs' cachce folder. Full options for this command can be found here, but the simplest options are to either remove all unused packages with:

conda clean -p

Or remove ALL packages with:

conda clean -a

Again, this should have no affect on hard-link enviroments installed under normal circumstances, however if there is any doubts consider making a backup of existing, important python environments as .yaml files which can be used to reinstalling enviroments or other options to reduce file count such as archiving old/unsued files and directories.