Signin Signup

Setup AI locally by Ollama

This post was last updated for 287 days ago, and the information may already be changed

EDIT: With SiYuan >=3.0.0 setup is much easier. Please see my latest comment.

It is possible to serve a LLM locally using an OpenAI compatible API. It seems to be possible to use local inference with SiYuan:

Once

Install ollama
Install LiteLLM
- pipx install 'litellm[proxy]'
ollama run orca2

Serve

ollama serve & litellm --model ollama/orca2

The settings in Siyuan are:

Model: gpt-4
API Base URL http://0.0.0.0:8000

I entered a dummy OpenAI API key (any value works)

You can use any model provided by ollama (or see liteLLM for even more models)

3 Operate

dwinkl updated this article at 2024-02-21 02:09:46

dwinkl updated this article at 2024-01-06 06:56:14

dwinkl updated this article at 2024-01-06 06:54:34

dwinkl 10 months

2 Thank

3 Operate

2 Reply

2

1.3k 93 80 71 1 1.0k

Related articles

2 Reply

Setup AI locally by Ollama

Welcome to here!

Here we can learn from each other how to use SiYuan, give feedback and suggestions, and build SiYuan together.

Signup About

Please input reply content ...

dwinkl • 9 months • 1
PRO Author
With the release of SiYuan 3.0.0 we can now set the LLM to use. This makes the setup trivial. For example I downloaded dolphin-mixtral using ollama by running:
```
ollama run dolphin-mixtral
```
after downloading the weights simply run
```
ollama serve
```
or open the ollama app.

In SiYuan you set the model you just downloaded as OpenAI and your localhost as the server

dolphin-mixtral is quite large so you will need a powerfull computer. A model I can recommend for faster inference/less powerful machines is Microsoft's Phi-2:

phi Phi-2: a 2.7B language model by Microsoft Research that demonstrates outstanding reasoning and language understanding capabilities. ollama.com
MiscReply
dwinkl • 9 months • 2
PRO Author

Ollama is now OpenAI compatible by itself so the setup should be easier

OpenAI compatibility · Ollama Blog Ollama now has initial compatibility with the OpenAI Chat Completions API, making it possible to use existing tooling built for OpenAI with local models via Ollama. ollama.ai

dwinkl

Recent hot