Git 服务器上的 LFS 下载

avatar
作者
筋斗云
阅读量:1

以llama为例:
https://huggingface.co/meta-llama/Llama-2-7b-hf

Github

# 1. 安装完成后,首先先初始化;如果有反馈,一般表示初始化成功 git lfs install# 2. 如果刚刚下载的那个项目没啥更改,重新下一遍,不算麻烦事(因为下载大文件,一般会比较慢) git lfs clone https://huggingface.co/meta-llama/Llama-2-7b-hf # 在下载的过程中,你也可以查看一下,你刚刚无法解析的那个pkl大文件,是不是在这个项目中,(进入项目目录)使用如下指令: cd Llama-2-7b-hf/  git lfs track ​ # 3. 如果不想重新下载整个项目,可以使用如下命令,单独下载需要使用lfs下载的大文件。 git lfs fetch git lfs checkout #(备选:git lfs pull),不建议 
GIT_LFS_SKIP_SMUDGE=0 git clone https://huggingface.co/meta-llama/Llama-2-7b-hf 

报错

$ git clone https://huggingface.co/meta-llama/Llama-2-7b-hf Cloning into 'Llama-2-7b-hf'... fatal: unable to access 'https://huggingface.co/meta-llama/Llama-2-7b-hf/': Failed to connect to huggingface.co port 443 after 21050 ms: Timed out 

找国内镜像

GIT_LFS_SKIP_SMUDGE=0

rm -rf * 
git lfs clone clone https://gitee.com/hf-models/Llama-2-7b-hf.git GIT_LFS_SKIP_SMUDGE=1 git clone https://gitee.com/hf-models/Llama-2-7b-hf.git GIT_LFS_SKIP_SMUDGE=0 git clone https://gitee.com/hf-models/Llama-2-7b-hf.git 

1

报错

$ git lfs clone https://gitee.com/hf-models/Llama-2-7b-hf.git WARNING: `git lfs clone` is deprecated and will not be updated           with new flags from `git clone`  `git clone` has been updated in upstream Git to have comparable speeds to `git lfs clone`. Cloning into 'Llama-2-7b-hf'... remote: Enumerating objects: 33, done. remote: Counting objects: 100% (33/33), done. remote: Compressing objects: 100% (32/32), done. remote: Total 33 (delta 9), reused 0 (delta 0), pack-reused 0 Receiving objects: 100% (33/33), 491.84 KiB | 631.00 KiB/s, done. Resolving deltas: 100% (9/9), done. Not in a Git repository. 

注意 Not in a Git repository.报错,查看:

$ git rev-parse --is-inside-work-tree fatal: unsafe repository ('E:/tst/t/Llama-2-7b-hf' is owned by someone else) To add an exception for this directory, call:         git config --global --add safe.directory E:/tst/t/Llama-2-7b-hf 

因此:

git config --global --add safe.directory E:/tst/t/Llama-2-7b-hf  git lfs track # 3. 如果不想重新下载整个项目,可以使用如下命令,单独下载需要使用lfs下载的大文件。 git lfs fetch git lfs checkout #(备选:git lfs pull),不建议 

使用git lfs fetch,报错:

$ git lfs fetch fetch: Fetching reference refs/heads/main [4ec71fd53e99766de38f24753b30c9e8942630e9e576a1ba27b0ec531e87be41] Object not found: [404] Object not found [60632e7f2530e713d57cfe2152019a01b1018070ce497e1f0c7fae4bf1340314] Object not found: [404] Object not found [fa574ee4fa536cc1dfc1c1baf622144fa037a5259967112eeb1a40c7cc00484a] Object not found: [404] Object not found [fd491dd3738e74dd4e18b964b9f8a15f785d487ab2d0c03c7a7b22fdac7bce1a] Object not found: [404] Object not found [41780b5dac322ac35598737e99208d90bdc632a1ba3389ebedbb46a1d8385a7f] Object not found: [404] Object not found error: failed to fetch some objects from 'https://gitee.com/hf-models/Llama-2-7b-hf.git/info/lfs' 

原因:

  • 对象不存在:指定的对象在远程仓库中不存在。这可能是因为对象已被删除、重命名或移动。请确保你正在尝试获取正确的对象。
  • 访问权限限制:你可能没有足够的权限来访问远程仓库中的对象。请确保你具有正确的访问权限,并且已经通过身份验证进行访问。

2

报错:

$ GIT_LFS_SKIP_SMUDGE=0 git clone https://gitee.com/hf-models/Llama-2-7b-hf.git Cloning into 'Llama-2-7b-hf'... remote: Enumerating objects: 33, done. remote: Counting objects: 100% (33/33), done. remote: Compressing objects: 100% (32/32), done. remote: Total 33 (delta 9), reused 0 (delta 0), pack-reused 0 Receiving objects: 100% (33/33), 491.84 KiB | 729.00 KiB/s, done. Resolving deltas: 100% (9/9), done. Updating files: 100% (18/18), done. Downloading model-00001-of-00002.safetensors (10 GB)/s Error downloading object: model-00001-of-00002.safetensors (4ec71fd): Smudge error: Error downloading model-00001-of-00002.safetensors (4ec71fd53e99766de38f24753b30c9e8942630e9e576a1ba27b0ec531e87be41): [4ec71fd53e99766de38f24753b30c9e8942630e9e576a1ba27b0ec531e87be41] Object not found: [404] Object not found  Errors logged to 'E:\Llama-2-7b-hf\.git\lfs\logs\20231112T152236.9629598.log'. Use `git lfs logs last` to view the log. error: external filter 'git-lfs filter-process' failed fatal: model-00001-of-00002.safetensors: smudge filter lfs failed warning: Clone succeeded, but checkout failed. You can inspect what was checked out with 'git status' and retry with 'git restore --source=HEAD :/' 

https://gitee.com/hf-models/Llama-2-7b-hf/tree/main

ref

https://blog.shipengx.com/archives/af0e9080.html

https://zhuanlan.zhihu.com/p/350011379

广告一刻

为您即时展示最新活动产品广告消息,让您随时掌握产品活动新动态!