gered/ollama
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity
3070 commits 3 branches 2 tags 9.8 MiB
Commit graph

559 commits

Author SHA1 Message Date
Jeffrey Morgan ead259d877
llm: fix seed value not being applied to requests (#4986) 2024-06-11 14:24:41 -07:00
Michael Yang f5f245cc15
Merge pull request #4938 from ollama/mxyng/fix-byte-order 
fix parsing big endian gguf
 2024-06-10 09:38:12 -07:00
Craig Hughes b84aea1685
Critical fix from llama.cpp JSON grammar to forbid un-escaped escape characters inside strings, which breaks parsing. (#3782) 2024-06-09 10:57:09 -07:00
Jeffrey Morgan 34f142797a
llm: always add bos token to prompt (#4941) 
* fix embedding by adding fixes from llama.cpp upstream

* remove assert

---------

Co-authored-by: Jesper Ek <deadbeef84@gmail.com>
 2024-06-08 18:47:10 -07:00
Michael Yang 620d5c569e fix parsing big endian gguf 2024-06-08 12:35:26 -07:00
Daniel Hiltgen cddc63381c
Merge pull request #4909 from dhiltgen/oneapi_disable 
Add ability to skip oneapi generate
 2024-06-07 14:07:15 -07:00
Michael Yang 030e765e76 fix create model when template detection errors 2024-06-07 10:51:35 -07:00
Daniel Hiltgen ab8c929e20 Add ability to skip oneapi generate 
This follows the same pattern for cuda and rocm to allow
disabling the build even when we detect the dependent libraries
 2024-06-07 08:32:49 -07:00
Jeffrey Morgan ce0dc33cb8
llm: patch to fix qwen 2 temporarily on nvidia (#4897) 2024-06-06 23:14:33 -07:00
Michael Yang 9b6c2e6eb6 detect chat template from KV 2024-06-06 16:03:47 -07:00
Michael Yang 6297f85606 gofmt, goimports 2024-06-04 13:20:24 -07:00
Michael Yang e40145a39d lint 2024-06-04 11:13:30 -07:00
Michael Yang c895a7d13f some gocritic 2024-06-04 11:13:30 -07:00
Michael Yang 04f3c12bb7 replace x/exp/slices with slices 2024-06-04 11:13:30 -07:00
Michael Yang 829ff87bd1
revert tokenize ffi (#4761) 
* Revert "use `int32_t` for call to tokenize (#4738)"

This reverts commit 763bb65dbb.

* Revert "vocab only"

This reverts commit bf54c845e9.

* Revert "use ffi for tokenizing/detokenizing"

This reverts commit 26a00a0410.
 2024-05-31 18:54:21 -07:00
Jeffrey Morgan 763bb65dbb
use int32_t for call to tokenize (#4738) 
* use `int32_t` for call to tokenize

* variable naming

* cleanup

* fix crash
 2024-05-30 21:43:30 -07:00
Jeffrey Morgan 7ca9605f54
speed up tests by only building static lib (#4740) 2024-05-30 21:43:15 -07:00
Michael Yang eb2c443a79
Merge pull request #4736 from ollama/mxyng/vocab-only 
vocab only for tokenize
 2024-05-30 17:21:00 -07:00
Jeffrey Morgan a50a87a7b8
partial offloading: allow flash attention and disable mmap (#4734) 
* partial offloading: allow flash attention and disable mmap

* allow mmap with num_gpu=0
 2024-05-30 16:58:01 -07:00
Michael Yang bf54c845e9 vocab only 2024-05-30 16:49:28 -07:00
Jeffrey Morgan 22f5c12ced
Update llama.cpp submodule to 5921b8f0 (#4731) 
* update llama.cpp submodule to `5921b8f089d3b7bda86aac5a66825df6a6c10603`

* add patch
 2024-05-30 16:20:22 -07:00
Michael Yang de781b37c8 rm unused infill 2024-05-29 11:26:47 -07:00
Michael Yang 3e21799377 rm unused system prompt 2024-05-29 11:26:47 -07:00
Michael Yang 26a00a0410 use ffi for tokenizing/detokenizing 2024-05-29 11:26:47 -07:00
Daniel Hiltgen 646371f56d
Merge pull request #3278 from zhewang1-intc/rebase_ollama_main 
Enabling ollama to run on Intel GPUs with SYCL backend
 2024-05-28 16:30:50 -07:00
Daniel Hiltgen 92c81e8117 Give the final model loading more time 
On some systems, 1 minute isn't sufficient to finish the load after it
hits 100% This creates 2 distinct timers, although they're both set to
the same value for now so we can refine the timeouts further.
 2024-05-28 09:08:10 -07:00
Lei Jitang 7487229c34
llm/server.go: Fix 2 minor typos (#4661) 
Signed-off-by: Lei Jitang <leijitang@outlook.com>
 2024-05-27 17:21:10 -07:00
Daniel Hiltgen 0165ba1651
Merge pull request #4638 from dhiltgen/better_error 
Report better warning on client closed abort of load
 2024-05-25 14:32:28 -07:00
Daniel Hiltgen c4209d6d21 Report better warning on client closed abort of load 
If the client closes the connection before we finish loading the model
we abort, so lets make the log message clearer why to help users
understand this failure mode
 2024-05-25 09:23:28 -07:00
Michael Yang d51f15257c
Update llm/ggml.go 
Co-authored-by: Bruce MacDonald <brucewmacdonald@gmail.com>
 2024-05-24 16:10:43 -07:00
Michael Yang 8f440d579a fix q5_0, q5_1 2024-05-24 16:01:46 -07:00
Patrick Devine 4cc3be3035
Move envconfig and consolidate env vars (#4608) 2024-05-24 14:57:15 -07:00
Wang,Zhe fd5971be0b support ollama run on Intel GPUs 2024-05-24 11:18:27 +08:00
Michael Yang 714adb8bd1
bump (#4597) 2024-05-23 14:16:26 -07:00
Daniel Hiltgen 95b1133d0c
Merge pull request #4547 from dhiltgen/load_progress 
Wire up load progress
 2024-05-23 14:06:02 -07:00
Daniel Hiltgen b37b496a12 Wire up load progress 
This doesn't expose a UX yet, but wires the initial server portion
of progress reporting during load
 2024-05-23 13:36:48 -07:00
Bruce MacDonald d6f692ad1a
Add support for IQ1_S, IQ3_S, IQ2_S, IQ4_XS. IQ4_NL (#4322) 
Co-authored-by: ManniX-ITA <20623405+mann1x@users.noreply.github.com>
 2024-05-23 13:21:49 -07:00
Jeffrey Morgan 38255d2af1
Use flash attention flag for now (#4580) 
* put flash attention behind flag for now

* add test

* remove print

* up timeout for sheduler tests
 2024-05-22 21:52:09 -07:00
Michael Yang 171eb040fc simplify safetensors reading 2024-05-21 11:28:22 -07:00
Michael Yang bbbd9f20f3 cleanup 2024-05-20 16:13:57 -07:00
Michael Yang 547132e820 bpe pretokenizer 2024-05-20 16:13:57 -07:00
Patrick Devine c8cf0d94ed llama3 conversion 2024-05-20 16:13:57 -07:00
jmorganca 5cab13739e set llama.cpp submodule commit to 614d3b9 2024-05-20 15:28:17 -07:00
Josh Yan 8aadad9c72 updated updateURL 2024-05-20 15:24:32 -07:00
Sam e15307fdf4
feat: add support for flash_attn (#4120) 
* feat: enable flash attention if supported

* feat: enable flash attention if supported

* feat: enable flash attention if supported

* feat: add flash_attn support
 2024-05-20 13:36:03 -07:00
Jeffrey Morgan 583c1f472c
update llama.cpp submodule to 614d3b9 (#4414) 2024-05-16 13:53:09 -07:00
Daniel Hiltgen c48c1d7c46 Port cuda/rocm skip build vars to linux 
Windows already implements these, carry over to linux.
 2024-05-15 15:56:43 -07:00
Patrick Devine d1692fd3e0
fix the cpu estimatedTotal memory + get the expiry time for loading models (#4461) 2024-05-15 15:43:16 -07:00
Daniel Hiltgen 853ae490e1 Sanitize the env var debug log 
Only dump env vars we care about in the logs
 2024-05-15 14:42:57 -07:00
Michael Yang 0e331c7168
Merge pull request #4328 from ollama/mxyng/mem 
count memory up to NumGPU if set by user
 2024-05-14 13:47:44 -07:00
Patrick Devine 6845988807
Ollama ps command for showing currently loaded models (#4327) 2024-05-13 17:17:36 -07:00
Michael Yang 1d359e737e typo 2024-05-13 14:18:34 -07:00
Michael Yang 50b9056e09 count memory up to NumGPU 2024-05-13 14:13:10 -07:00
jmorganca 92ca2cca95 Revert "only forward some env vars" 
This reverts commit ce3b212d12.
 2024-05-10 22:53:21 -07:00
Daniel Hiltgen c4014e73a2 Fall back to CPU runner with zero layers 2024-05-10 15:09:48 -07:00
Michael Yang 1eb382da5a add phi2 mem 2024-05-10 12:13:28 -07:00
Jeffrey Morgan bb6fd02298
Don't clamp ctx size in PredictServerFit (#4317) 
* dont clamp ctx size in `PredictServerFit`

* minimum 4 context

* remove context warning
 2024-05-10 10:17:12 -07:00
Michael Yang cf442cd57e fix typo 2024-05-09 16:23:37 -07:00
Michael Yang ce3b212d12 only forward some env vars 2024-05-09 15:16:09 -07:00
Michael Yang 58876091f7 log clean up 2024-05-09 14:55:36 -07:00
Daniel Hiltgen d0425f26cf
Merge pull request #4294 from dhiltgen/harden_subprocess_reaping 
Harden subprocess reaping
 2024-05-09 14:02:16 -07:00
Bruce MacDonald cfa84b8470
add done_reason to the api (#4235) 2024-05-09 13:30:14 -07:00
Daniel Hiltgen 84ac7ce139 Refine subprocess reaping 2024-05-09 11:21:31 -07:00
Daniel Hiltgen 920a4b0794 Merge remote-tracking branch 'upstream/main' into pr3702 2024-05-08 16:44:35 -07:00
Daniel Hiltgen ee49844d09
Merge pull request #4153 from dhiltgen/gpu_verbose_response 
Add GPU usage
 2024-05-08 16:39:11 -07:00
Daniel Hiltgen 8a516ac862
Merge pull request #4241 from dhiltgen/fix_tmp_override 
Detect noexec and report a better error
 2024-05-08 15:34:22 -07:00
Daniel Hiltgen bee2f4a3b0 Record GPU usage information 
This records more GPU usage information for eventual UX inclusion.
 2024-05-08 14:45:39 -07:00
Michael Yang eeb695261f skip if same quantization 2024-05-07 17:44:19 -07:00
Daniel Hiltgen 72700279e2 Detect noexec and report a better error 
This will bubble up a much more informative error message if noexec
is preventing us from running the subprocess
 2024-05-07 16:46:15 -07:00
Michael Yang 1e0a669f75
Merge pull request #3682 from ollama/mxyng/quantize-all-the-things 
quantize any fp16/fp32 model
 2024-05-07 15:20:49 -07:00
Michael Yang 4736391bfb llm: add minimum based on layer size 2024-05-06 17:04:19 -07:00
Michael Yang 01811c176a comments 2024-05-06 15:24:01 -07:00
Michael Yang 9685c34509 quantize any fp16/fp32 model 
- FROM /path/to/{safetensors,pytorch}
- FROM /path/to/fp{16,32}.bin
- FROM model:fp{16,32}
 2024-05-06 15:24:01 -07:00
Daniel Hiltgen 380378cc80 Use our libraries first 
Trying to live off the land for cuda libraries was not the right strategy.  We need to use the version we compiled against to ensure things work properly
 2024-05-06 14:23:29 -07:00
Jeffrey Morgan ed740a2504
Fix no slots available error with concurrent requests (#4160) 2024-05-06 14:22:53 -07:00
Jeffrey Morgan 1b0e6c9c0e
Fix llava models not working after first request (#4164) 
* fix llava models not working after first request

* individual requests only for llava models
 2024-05-05 20:50:31 -07:00
Daniel Hiltgen f56aa20014 Centralize server config handling 
This moves all the env var reading into one central module
and logs the loaded config once at startup which should
help in troubleshooting user server logs
 2024-05-05 16:49:50 -07:00
Michael Yang 44869c59d6 omit prompt and generate settings from final response 2024-05-03 17:00:02 -07:00
Mark Ward 321d57e1a0 Removing go routine calling .wait from load. 2024-05-01 18:51:10 +00:00
Mark Ward ba26c7aa00 it will always return an error due to Kill() discarding Wait() errors 2024-05-01 18:51:10 +00:00
Mark Ward 63c763685f log when the waiting for the process to stop to help debug when other tasks execute during this wait. 
expire timer clear the timer reference because it will not be reused.
close will clean up expireTimer if calling code has not already done this.
 2024-05-01 18:51:10 +00:00
Mark Ward 948114e3e3 fix sched to wait for the runner to terminate to ensure following vram check will be more accurate 2024-05-01 18:51:10 +00:00
Jeffrey Morgan f0c454ab57
gpu: add 512MiB to darwin minimum, metal doesn't have partial offloading overhead (#4068) 2024-05-01 11:46:03 -04:00
jmorganca fcf4d60eee llm: add back check for empty token cache 2024-04-30 17:38:44 -04:00
jmorganca e33d5c2dbc update llama.cpp commit to 952d03d 2024-04-30 17:31:20 -04:00
Jeffrey Morgan 18d9a7e1f1
update llama.cpp submodule to f364eb6 (#4060) 2024-04-30 17:25:39 -04:00
Daniel Hiltgen 23d23409a0
Update llama.cpp (#4036) 
* Bump llama.cpp to b2761

* Adjust types for bump
 2024-04-29 23:18:48 -04:00
Jeffrey Morgan 7aa08a77ca
llm: dont cap context window limit to training context window (#3988) 2024-04-29 10:07:30 -04:00
Hernan Martinez 8a65717f55 Do not build AVX runners on ARM64 2024-04-26 23:55:32 -06:00
Hernan Martinez b438d485f1 Use architecture specific folders in the generate script 2024-04-26 23:34:12 -06:00
Hernan Martinez 86e67fc4a9 Add import declaration for windows,arm64 to llm.go 2024-04-26 23:23:53 -06:00
Daniel Hiltgen e4859c4563 Fine grain control over windows generate steps 
This will speed up CI which already tries to only build static for unit tests
 2024-04-26 15:49:46 -07:00
Daniel Hiltgen 0b5c589ca2
Merge pull request #3966 from dhiltgen/bump 
Fix target in gen_windows.ps1
 2024-04-26 15:36:53 -07:00
Michael Yang 65fadddc85
Merge pull request #3964 from ollama/mxyng/weights 
fix gemma, command-r layer weights
 2024-04-26 15:23:33 -07:00
Daniel Hiltgen ed5fb088c4 Fix target in gen_windows.ps1 2024-04-26 15:10:42 -07:00
Michael Yang f81f308118 fix gemma, command-r layer weights 2024-04-26 15:00:55 -07:00
Jeffrey Morgan bb31def011
return code 499 when user cancels request while a model is loading (#3955) 2024-04-26 17:38:29 -04:00
Daniel Hiltgen 5c0c2d1d09
Merge pull request #3954 from dhiltgen/ci_fixes 
Put back non-avx CPU build for windows
 2024-04-26 13:09:03 -07:00
Daniel Hiltgen 421c878a2d Put back non-avx CPU build for windows 2024-04-26 12:44:07 -07:00
Daniel Hiltgen 85801317d1 Fix clip log import 2024-04-26 09:43:46 -07:00