Skip to content

(2025-02-07) Lab Notebook: Load Custom Models in Ollama

Warning

This is as a Lab Notebook which describes how to solve a specific problem at a specific time. Please keep this in mind as you read and use the content. Please pay close attention to the date, version information and other details.

This lab notebook provides a quick guide to running custom models on the HPCC using the .gguf file format with Ollama. A Modelfile serves as a blueprint for building and configuring your model; it tells Ollama where to find the model, how to configure it, and how to load it.

A .gguf file is a compressed format that is the industry standard for storing and transferring models.

Example 1: Running a Custom Model from a .gguf File

This example shows how to use a custom .gguf file directly. (The file must be on the HPCC; if it isn’t already, please view our File Transfer documentation.)

  1. Create a Modelfile:

    Save the following content as Modelfile (update the path as necessary; no file extension is required). This file specifies the location of your .gguf model.

    1
    FROM /path/to/your/model.gguf

  2. Load the ollama module:

    1
    module load Ollama

  3. Start the ollama server:

    1
    ollama serve &

  4. Create the custom model:

    1
    ollama create new-model-name -f Modelfile

    The name given here will be the name displayed in ollama list.

  5. Run the model:

    1
    ollama run new-model-name

Example 2: Customizing a Pre-built Base Model

This example shows how to build upon a pre-existing model (e.g., llama3.2), adjust parameters, and set a custom system message.

  1. Create a Modelfile:

    Save this content as Modelfile:

    1
2
3
4
    FROM llama3.2
PARAMETER temperature 0.7
PARAMETER num_ctx 4096
SYSTEM You are a knowledgeable assistant ready to help with detailed explanations.

  2. Load the ollama module:

    1
    module load Ollama

  3. Start the ollama server:

    1
    ollama serve &

  4. Create your custom model:

    1
    ollama create custom-llama -f Modelfile

  5. Run the model:

    1
    ollama run custom-llama

Modelfile Syntax

The syntax for the Modelfile is still in development. Below are some common instructions you can use:

  • FROM: Specifies the base model to use. For example, a local .gguf file:

    1
    FROM /path/to/your/model.gguf

    or a pre-built model from the ollama library:

    1
    FROM llama3.2

    The Ollama Library has a wide range of pre-built models for you to choose from.

  • PARAMETER: Sets configuration values for the model such as temperature or context window size. For example:

    1
2
    PARAMETER temperature 0.7
PARAMETER num_ctx 4096

  • SYSTEM: Defines a custom system message to control the model's behavior. For example:

    1
    SYSTEM You are a knowledgeable assistant ready to help.

Other instructions like TEMPLATE, ADAPTER, and MESSAGE are available for advanced configurations. View the Ollama Modelfile Docs or Ollama import docs for more information.