Daniel Hiltgen
fc85f50a2b
Ensure sparse files on windows during download
...
The file.Truncate call on windows will write the whole file
unless you set the sparse flag, leading to heavy I/O at the
beginning of download. This should improve our
I/O behavior on windows and put less stress on the users disk.
2024-08-06 10:58:08 -07:00
Michael Yang
b732beba6a
lint
2024-08-01 17:06:06 -07:00
Blake Mizerany
dc77bbcfa4
server: fix json marshalling of downloadBlobPart ( #6108 )
2024-07-31 16:01:24 -07:00
Blake Mizerany
750c1c55f7
server: fix race conditions during download ( #5994 )
...
This fixes various data races scattered throughout the download/pull
client where the client was accessing the download state concurrently.
This commit is mostly a hot-fix and will be replaced by a new client one
day soon.
Also, remove the unnecessary opts argument from downloadChunk.
2024-07-26 14:24:24 -07:00
Blake Mizerany
c8af3c2d96
server: reuse original download URL for images ( #5962 )
...
This changes the registry client to reuse the original download URL
it gets on the first redirect response for all subsequent requests,
preventing thundering herd issues when hot new LLMs are released.
2024-07-25 15:58:30 -07:00
Blake Mizerany
de5beb06b3
server: skip blob verification for already verified blobs
2024-06-05 16:39:11 -07:00
Michael Yang
8ffb51749f
nolintlint
2024-06-04 11:13:30 -07:00
Tim Scheuermann
db2ffa79f1
Fix download retry issue
2024-05-24 20:30:42 +02:00
Michael Yang
ac145f75ca
return on part done
2024-05-14 13:04:30 -07:00
Blake Mizerany
a7b431e743
server: provide helpful workaround hint when stalling on pull ( #3584 )
...
This is a quick fix to help users who are stuck on the "pull" step at
99%.
In the near future we're introducing a new registry client that
should/will hopefully be smarter. In the meantime, this should unblock
the users hitting issue #1736 .
2024-04-10 16:24:37 -07:00
Patrick Devine
1b272d5bcd
change github.com/jmorganca/ollama to github.com/ollama/ollama ( #3347 )
2024-03-26 13:04:17 -07:00
Jeffrey Morgan
fc06205971
Revert "adjust download and upload concurrency based on available bandwidth" ( #2995 )
2024-03-07 18:10:16 -08:00
Michael Yang
084d846621
refactor
2024-02-21 13:42:48 -08:00
Michael Yang
6a4b994433
lint
2024-02-21 13:42:48 -08:00
Michael Yang
074934be03
adjust group limit based on download speed
2024-02-21 13:42:48 -08:00
Michael Yang
0de12368a0
add new LimitGroup for dynamic concurrency
2024-02-21 13:42:48 -08:00
Michael Yang
917bd61084
refactor download run
2024-02-21 13:42:46 -08:00
Michael Yang
e43648afe5
rerefactor
2024-02-15 05:56:45 +00:00
Daniel Hiltgen
f397e0e988
Move hub auth out to new package
2024-02-15 05:56:45 +00:00
Michael Yang
9d3dcfd0ec
fix logging
2024-01-26 11:04:27 -08:00
Michael Yang
6e0ea5ecc8
Merge pull request #1916 from ollama/mxyng/inactivity-monitor
...
download: add inactivity monitor
2024-01-26 10:56:00 -08:00
Daniel Hiltgen
fedd705aea
Mechanical switch from log to slog
...
A few obvious levels were adjusted, but generally everything mapped to "info" level.
2024-01-18 14:12:57 -08:00
Michael Yang
27331ae3a8
download: add inactivity monitor
...
if a download part is inactive for some time, restart it
2024-01-12 15:23:15 -08:00
Michael Yang
2bb2bdd5d4
fix lint
2024-01-09 09:36:58 -08:00
Michael Yang
acfc376efd
add .golangci.yaml
2024-01-09 09:36:58 -08:00
Jeffrey Morgan
ac5076ce1e
exponential backoff up to 30s
2023-11-19 14:32:19 -05:00
Michael Yang
cb42589792
adjust download/upload parts
2023-11-19 14:32:19 -05:00
Jeffrey Morgan
e1d7056496
update progress statuses
2023-11-19 09:21:13 -05:00
Michael Yang
d6ecaa2cbf
update progress responses
2023-11-17 10:06:19 -08:00
Daniel Hiltgen
cc54a416c6
Resume chunk download on UnexpectedEOF errors
...
If the chunk download is interrupted, resume from where we left off
2023-11-10 08:29:42 -08:00
Michael Yang
434a6f9d46
return last error
2023-11-03 16:49:51 -07:00
Michael Yang
d39709260f
download with retry
2023-11-02 13:16:11 -07:00
Michael Yang
60bb3c03a1
use http.Method
2023-11-02 13:12:45 -07:00
Michael Yang
4e09aab8b9
concurrent uploads
2023-10-27 17:07:33 -07:00
Michael Yang
910816a532
fix(download): no retry when out of space
2023-10-26 11:34:07 -07:00
Michael Yang
257ffeb997
fix download
2023-10-12 12:52:43 -07:00
Michael Yang
c413a55093
download: handle inner errors
2023-10-11 14:15:30 -07:00
Michael Yang
630bb75d2a
dynamically size download parts based on file size
2023-10-11 14:10:25 -07:00
Michael Yang
a2055a1e93
update download
2023-10-11 14:10:25 -07:00
Jeffrey Morgan
65dcd0ce35
always cleanup blob download ( #747 )
2023-10-10 13:12:29 -04:00
Michael Yang
0560b28a8d
names
2023-10-06 12:56:56 -07:00
Michael Yang
10199c5987
replace done channel with file check
2023-10-06 12:56:56 -07:00
Michael Yang
288814d3e4
fix ref counts
2023-10-06 12:56:43 -07:00
Michael Yang
04733438da
check head request response
2023-10-06 12:56:43 -07:00
Michael Yang
711e891f0f
fix resumable downloads
...
glob returns files in lexical order which is not appropriate when
rebuilding the parts list
2023-10-06 12:56:43 -07:00
Michael Yang
090d08422b
handle unexpected eofs
2023-10-06 12:56:43 -07:00
Michael Yang
5b84404c64
handle concurrent requests for the same blobs
2023-10-06 12:56:43 -07:00
Michael Yang
8544edca21
parallel chunked downloads
2023-10-06 12:56:43 -07:00
Michael Yang
f40b3de758
use int64 consistently
2023-09-28 11:07:24 -07:00
Michael Yang
b58d5d16b0
fix mkdir on windows
2023-09-19 09:41:13 -07:00
