From c0e7a3b90ec64101ee2999f200f71f019d447f77 Mon Sep 17 00:00:00 2001 From: Arturas Smorgun Date: Fri, 11 Aug 2023 00:58:09 +0100 Subject: [PATCH] Document num_gqa parameter It is required to be adjusted for some models, see https://github.com/jmorganca/ollama/issues/320 for more context --- docs/modelfile.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/modelfile.md b/docs/modelfile.md index f49956cb..294ebddc 100644 --- a/docs/modelfile.md +++ b/docs/modelfile.md @@ -104,6 +104,7 @@ PARAMETER | mirostat_eta | Influences how quickly the algorithm responds to feedback from the generated text. A lower learning rate will result in slower adjustments, while a higher learning rate will make the algorithm more responsive. (Default: 0.1) | float | mirostat_eta 0.1 | | mirostat_tau | Controls the balance between coherence and diversity of the output. A lower value will result in more focused and coherent text. (Default: 5.0) | float | mirostat_tau 5.0 | | num_ctx | Sets the size of the context window used to generate the next token. (Default: 2048) | int | num_ctx 4096 | +| num_gqa | The number of GQA groups in the transformer layer. Required for some models, for example it is 8 for llama2:70b | int | num_gqa 8 | | num_gpu | The number of GPUs to use. On macOS it defaults to 1 to enable metal support, 0 to disable. | int | num_gpu 1 | | num_thread | Sets the number of threads to use during computation. By default, Ollama will detect this for optimal performance. It is recommended to set this value to the number of physical CPU cores your system has (as opposed to the logical number of cores). | int | num_thread 8 | | repeat_last_n | Sets how far back for the model to look back to prevent repetition. (Default: 64, 0 = disabled, -1 = num_ctx) | int | repeat_last_n 64 | @@ -163,4 +164,4 @@ LICENSE """ ## Notes - the **modelfile is not case sensitive**. In the examples, we use uppercase for instructions to make it easier to distinguish it from arguments. -- Instructions can be in any order. In the examples, we start with FROM instruction to keep it easily readable. \ No newline at end of file +- Instructions can be in any order. In the examples, we start with FROM instruction to keep it easily readable.