Setup AI locally by Ollama

EDIT: With SiYuan >=3.0.0 setup is much easier. Please see my latest comment.

It is possible to serve a LLM locally using an OpenAI compatible API. It seems to be possible to use local inference with SiYuan:

Once

1. Install ollama
2. Install LiteLLM
   • pipx install 'litellm[proxy]'
3. ollama run orca2

Serve

ollama serve & litellm --model ollama/orca2

The settings in Siyuan are:

• Model: gpt-4
• API Base URL http://0.0.0.0:8000

I entered a dummy OpenAI API key (any value works)

You can use any model provided by ollama (or see liteLLM for even more models)