tokenizers

Fast State-of-the-Art Tokenizers optimized for Research and Production

https://github.com/huggingface/tokenizers

Available modules

The overview below shows which tokenizers installations are available per target architecture in the HPCC module system, ordered based on software version (new to old).

To start using tokenizers, load one of these modules using a module load command like:

module load tokenizers/0.15.2-GCCcore-12.3.0

(This data was automatically generated on Thu, 22 Jan 2026 at 12:03:27 EST)

	gateway	generic	zen2	zen3	zen4	skylake_avx512
	Gateway nodes	everywhere (except Grace nodes)	amd20	amd22	amd24	intel18,amd20-v100,amd21,intel21
tokenizers/0.15.2-GCCcore-12.3.0	-	x	-	-	-	-

tokenizers/0.15.2-GCCcore-12.3.0

This is a list of extensions included in the module:

huggingface-hub-0.21.4, tokenizers-0.15.2