allama

这里搞只用CPU运行的

docker pull soulteary/llama2:converter

下载docker镜像用来转换模型为GGML，接着去找一个中文模型

docker run --ulimit memlock=-1 --ulimit stack=67108864 --rm -it -v /data/llama/models/:/app/ soulteary/llama2:converter bash
python3 convert.py /app/LinkSoul/Chinese-Llama-2-7b/ --outfile /app/Chinese-Llama-2-7b-ggml.bin

这里很简单，看代码能看出是做什么

./quantize /app/soulteary/Chinese-Llama-2-7b-ggml.bin /app/soulteary/Chinese-Llama-2-7b-ggml-q4.bin q4_0

用别人做好的镜像

docker pull soulteary/llama2:runtime
docker run --ulimit memlock=-1 --ulimit stack=67108864 --rm -it -v `pwd`/soulteary:/app/soulteary soulteary/llama2:runtime bash

对话

./main -m /app/soulteary/Chinese-Llama-2-7b-ggml-q4.bin -n 256 --repeat_penalty 1.0 --color -i -r "User:" -f prompts/chat-with-bob.txt

更详细的请看github，链接在最上面

接下来我看下怎么接入到网页

上面的方案貌似没有支持server

回到官方的github，看到有server，还是考虑用官方的github去制作镜像

拉去git

git clone https://github.com/ggerganov/llama.cpp.git
进入目录
docker build . -t llamaapp/server -f .devops/server.Dockerfile
docker run --name test --ulimit memlock=-1 --ulimit stack=67108864 --rm -it -v `pwd`/soulteary:/app/soulteary  -p 33000:33000 llamacpp/server -m /app/soulteary/Trans-Chinese-Llama-2-7b-ggml-q4.bin --port 33000 --host 0.0.0.0 -n 512

这里改成自己的目录和端口就好了

注意，会碰到一个启动失败的问题

gguf_init_from_file: invalid magic characters tjgg

llama.cpp项目目录下有一个python文件, convert-llama-ggml-to-gguf.py用这个转换一下

还是用上面转化好的模型

python3 convert-llama-ggml-to-gguf.py -i 原模型 -o 新模型

这里也按照自己的目录改一下就好了

接下来用curl测试一下

curl --request POST \
    --url http://localhost:33000/completion \
    --header "Content-Type: application/json" \
    --data '{"prompt": "Building a website can be done in 10 simple steps:","n_predict": 128}'

{"content":"\n everybody can do it! Let's learn how:\nStep 1. Choose a domain name: The first step to building your website is selecting the perfect domain name for it, something that summarizes its content and easy-to-remember. If you don’t have any idea, search for some inspirations on Google or in domain registrar websites.\nStep 2. Get hosting services: Once you have selected a domain name, you will need to choose a reliable web host provider for your website. This is where you can find all the hosting companies online and compare their features and prices. It’s","generation_settings":{"dynatemp_exponent":1.0,"dynatemp_range":0.0,"frequency_penalty":0.0,"grammar":"","ignore_eos":false,"logit_bias":[],"min_keep":0,"min_p":0.05000000074505806,"mirostat":0,"mirostat_eta":0.10000000149011612,"mirostat_tau":5.0,"model":"/app/soulteary/t2.bin","n_ctx":2048,"n_keep":0,"n_predict":512,"n_probs":0,"penalize_nl":true,"penalty_prompt_tokens":[],"presence_penalty":0.0,"repeat_last_n":64,"repeat_penalty":1.100000023841858,"samplers":["top_k","tfs_z","typical_p","top_p","min_p","temperature"],"seed":4294967295,"stop":[],"stream":false,"temperature":0.800000011920929,"tfs_z":1.0,"top_k":40,"top_p":0.949999988079071,"typical_p":1.0,"use_penalty_prompt_tokens":false},"id_slot":0,"model":"/app/soulteary/t2.bin","prompt":"Building a website can be done in 10 simple steps:","stop":true,"stopped_eos":false,"stopped_limit":true,"stopped_word":false,"stopping_word":"","timings":{"predicted_ms":15325.428,"predicted_n":128,"predicted_per_second":8.352132155787102,"predicted_per_token_ms":119.72990625,"prompt_ms":723.089,"prompt_n":14,"prompt_per_second":19.36137875144,"prompt_per_token_ms":51.64921428571429},"tokens_cached":141,"tokens_evaluated":14,"tokens_predicted":128,"truncated":false}

这里就大功告成了,