GPU方式运行模型服务出错

[a652]

按照README运行命令：
./server -m ./models/codeshell-chat-q4_0.gguf --host 127.0.0.1 --port 8080

报错信息如下：
ggml_metal_init: GPU name: Apple M1
ggml_metal_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_init: hasUnifiedMemory = true
ggml_metal_init: recommendedMaxWorkingSetSize = 5461.34 MB
ggml_metal_init: maxTransferRate = built-in GPU
llama_new_context_with_model: compute buffer total size = 558.13 MB
llama_new_context_with_model: max tensor size = 224.77 MB
ggml_metal_add_buffer: allocated 'data ' buffer, size = 4096.00 MB, offs = 0
ggml_metal_add_buffer: allocated 'data ' buffer, size = 486.91 MB, offs = 4059267072, ( 4583.53 / 5461.34)
ggml_metal_add_buffer: allocated 'kv ' buffer, size = 1346.00 MB, ( 5929.53 / 5461.34), warning: current allocated size is greater than the recommended max working set size
ggml_metal_add_buffer: allocated 'alloc ' buffer, size = 552.02 MB, ( 6481.55 / 5461.34), warning: current allocated size is greater than the recommended max working set size
ggml_metal_graph_compute: command buffer 0 failed with status 5
GGML_ASSERT: ggml-metal.m:1459: false
Abort trap: 6

电脑信息：
M1 MacBook Pro
MacOS Sonoma 14.1

[a652]

相关issue：ggml-org/llama.cpp#2048

建议列一个对应的硬件指标。

[qqggwm]

llm_load_print_meta: format = GGUF V2 (latest)
llm_load_print_meta: arch = codeshell
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 70144
llm_load_print_meta: n_merges = 72075
llm_load_print_meta: n_ctx_train = 8192
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_layer = 42
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: f_norm_eps = 1.0e-05
llm_load_print_meta: f_norm_rms_eps = 0.0e+00
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff = 16384
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: model type = 7B
llm_load_print_meta: model ftype = mostly Q4_0
llm_load_print_meta: model params = 7.98 B
llm_load_print_meta: model size = 4.25 GiB (4.58 BPW)
llm_load_print_meta: general.name = CodeShell
llm_load_print_meta: BOS token = 70000 '<|endoftext|>'
llm_load_print_meta: EOS token = 70000 '<|endoftext|>'
llm_load_print_meta: UNK token = 70000 '<|endoftext|>'
llm_load_print_meta: PAD token = 70000 '<|endoftext|>'
llm_load_print_meta: LF token = 28544 'ÄĬ'
llm_load_tensors: ggml ctx size = 0.17 MB
llm_load_tensors: mem required = 4355.64 MB
..............................................................................................
llama_new_context_with_model: n_ctx = 8192
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_new_context_with_model: kv self size = 1344.00 MB
llama_new_context_with_model: compute buffer total size = 558.13 MB
Segmentation fault (core dumped)

楼主解决了吗遇到类似的问题

[a652]

llm_load_print_meta: format = GGUF V2 (latest) llm_load_print_meta: arch = codeshell llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 70144 llm_load_print_meta: n_merges = 72075 llm_load_print_meta: n_ctx_train = 8192 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_layer = 42 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: f_norm_eps = 1.0e-05 llm_load_print_meta: f_norm_rms_eps = 0.0e+00 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: n_ff = 16384 llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: model type = 7B llm_load_print_meta: model ftype = mostly Q4_0 llm_load_print_meta: model params = 7.98 B llm_load_print_meta: model size = 4.25 GiB (4.58 BPW) llm_load_print_meta: general.name = CodeShell llm_load_print_meta: BOS token = 70000 '<|endoftext|>' llm_load_print_meta: EOS token = 70000 '<|endoftext|>' llm_load_print_meta: UNK token = 70000 '<|endoftext|>' llm_load_print_meta: PAD token = 70000 '<|endoftext|>' llm_load_print_meta: LF token = 28544 'ÄĬ' llm_load_tensors: ggml ctx size = 0.17 MB llm_load_tensors: mem required = 4355.64 MB .............................................................................................. llama_new_context_with_model: n_ctx = 8192 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 llama_new_context_with_model: kv self size = 1344.00 MB llama_new_context_with_model: compute buffer total size = 558.13 MB Segmentation fault (core dumped)

楼主解决了吗遇到类似的问题

没有，我的电脑只有8G内存，推测是不够用