Connect langchainjs to a local llm installed in Jan AI (llama) - close it and answer in ISSUE #7539

getzelus · 2025-01-17T10:41:57Z

getzelus
Jan 17, 2025

Hello guys,
Im trying to use langchainjs with a local llm (llama3.2-1b-instruct) that i installed in Jan AI and I start its server.

I can do a simple call without langchainjs and its working.
But when I try with langchainjs, i have an error.
Its difficult to find the solution in the documentation, forum or chatgpt.

Here is the code working without langchainjs :

import axios from 'axios';

const url = 'http://127.0.0.1:1337/v1/chat/completions';
const model = 'llama3.2-1b-instruct';

async function query(system, prompt) {
    try {
        const response = await axios.post(
            url,
            {
                model: model,
                messages: [
                    { role: 'system', content: system },
                    { role: 'user', content: prompt }
                ],
            },
            {
                headers: {
                    'Content-Type': 'application/json',
                },
            }
        );
        const res = response.data.choices[0].message.content;
        return res;
    } catch (error) {
        console.error('Error querying Cortex:', error.response ? error.response.data : error.message);
    }
}

async function start() {
    let res = await query('your an assistant', 'whats the capital of Japan ?')
    console.log(res)
}

start()

Here is the code for langchainjs that doesnt work :

import { ChatOllama } from "@langchain/ollama";

const llm = new ChatOllama({
    model: "llama3.2-1b-instruct",
    baseUrl: "http://127.0.0.1:1337/v1/chat/completions",
    temperature: 0,
});

try {
    const res = await llm.invoke("Whats the capital of Japan");
    console.log(res);
} catch (error) {
    console.error("Error invoking LLM:", error);
}

I tried many options, like changing the url, the name of the model, using chatopenAI package etc...

the error i got :

Getting text from response
Error invoking LLM: ResponseError: <html>
<head><title>404 Not Found</title></head>
<body bgcolor="white" text="black">
<center><h1>404 Not Found</h1></center>
<hr><center>drogon/1.9.7</center>
</body>
</html>

Thank you for your attention

System Info

Windows 11
Node 20.14
"@langchain/core": "^0.3.30",
"@langchain/ollama": "^0.1.4",
"@langchain/openai": "^0.3.17",
"axios": "^1.7.9",
"dotenv": "^16.4.7",
"langchain": "^0.3.11"

@getzelus · 2025-01-17T10:44:20Z

dosubot[bot]
bot Jan 17, 2025

Hey there, @getzelus! I'm here to help you with your technical questions and bugs. Let's squash those bugs together! 🐞

To resolve the 404 Not Found error when using the ChatOllama class with your local LLM (llama3.2-1b-instruct) installed via Jan AI, ensure that the baseUrl is correctly set to the local server's address where the LLM is hosted. The default baseUrl in the ChatOllama class might not match your setup.

Here's how you can set the baseUrl correctly:

import { ChatOllama } from "@langchain/ollama";

const llm = new ChatOllama({
  model: "llama3.2-1b-instruct",
  baseUrl: "http://127.0.0.1:1337", // Ensure this matches your server's address and port
  temperature: 0,
});

try {
  const res = await llm.invoke("What's the capital of Japan?");
  console.log(res);
} catch (error) {
  console.error("Error invoking LLM:", error);
}

Make sure the baseUrl is set to the correct address and port where your local LLM server is running. If you continue to encounter a 404 error, it might be due to using a legacy version of Ollama. Consider upgrading to a newer version for better chat support. If upgrading is not an option, you can try using the legacy streaming method _streamResponseChunksLegacy as a fallback ^[1].

To continue talking to Dosu, mention @dosu.

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

2 replies

getzelus Jan 17, 2025
Author

@dosu

As mentionned, i tried other urls and it doesnt work :

Error invoking LLM: ResponseError: Not Found
    at checkOk (file:///C:/dev/ai/myagent/node_modules/.pnpm/ollama@0.5.12/node_modules/ollama/dist/browser.mjs:70:9)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async post (file:///C:/dev/ai/myagent/node_modules/.pnpm/ollama@0.5.12/node_modules/ollama/dist/browser.mjs:116:3)
    at async Ollama.processStreamableRequest 
}

dosubot[bot] bot Jan 17, 2025

The error you're encountering suggests that the URL or endpoint you're trying to access might not be correct or supported by the ChatOllama class. Here are a few things you can check or try:

Verify the Endpoint: Ensure that the endpoint http://127.0.0.1:1337 is correct and that the server is running and accessible at this address. The path /v1/chat/completions might not be necessary or correct for the ChatOllama class, as it might expect a different endpoint structure.
Check Server Logs: Look at the logs of your local LLM server to see if there are any errors or messages that could provide more context about why the request is not being found.
Legacy Version Fallback: If your server is using a legacy version of Ollama, the ChatOllama class has a fallback mechanism to handle legacy versions. Ensure that your server is compatible with the current version of the ChatOllama class, or consider updating your server if possible.
Headers and Authentication: Double-check if there are any specific headers or authentication requirements needed by your server that might not be included in the request made by ChatOllama.
Test with Direct HTTP Request: As a troubleshooting step, try making a direct HTTP request using a tool like curl or Postman to ensure that the server is responding correctly to requests at the specified URL.

If these steps don't resolve the issue, you might need to look into the specific configuration or setup of your local LLM server to ensure compatibility with the ChatOllama class.

To continue talking to Dosu, mention @dosu.

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connect langchainjs to a local llm installed in Jan AI (llama) - close it and answer in ISSUE #7539

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 2 replies

{{title}}

{{title}}

{{title}}

Select a reply

Connect langchainjs to a local llm installed in Jan AI (llama) - close it and answer in ISSUE #7539

getzelus Jan 17, 2025

System Info

Replies: 1 comment · 2 replies

dosubot[bot] bot Jan 17, 2025

getzelus Jan 17, 2025 Author

dosubot[bot] bot Jan 17, 2025

getzelus
Jan 17, 2025

Replies: 1 comment 2 replies

dosubot[bot]
bot Jan 17, 2025

getzelus Jan 17, 2025
Author