[0:00]In our last couple of videos about OpenClaw, we covered installing it in a segmented VM for security and also locking down a WSL instance of OpenClaw. However, both of those still relied on sending all of our prompts out to the cloud. Today, we're going to take it a step further and connect OpenClaw to a local AI model running directly on our GPU. For the demo, I'll start from scratch, so feel free to follow along. All of the commands can be found in the description for easy copy and paste reference.
[0:37]Let's Get Started! Crack open a terminal and be sure to run it as admin. If you don't already have WSL installed, go ahead and run this command to spin up Ubuntu. I will be fast forwarding through the installs, but they only take a few minutes. Once the installer completes, you'll be prompted to create a username and password. Make sure you don't forget or lose this information. Now that Ubuntu is installed, we need to make sure we can see our GPU. If you have an Nvidia graphics card, it should show up right away. Here we see my RTX 3080 with 10 gigs of VRAM. Next, we need to grab ZSTD in preparation for the following step. Drop that command in now. Okay, now that we have that, we can install Ollama. Pop in that command now. This will take a few minutes to complete. All right, now that Ollama is installed, let's pull down a model. I'm going with a 7B model as that's about what I can load into VRAM on my 3080. Drop that command now. If you want a different model, just make note of it as we'll need that info later on when plugging it into OpenClaw. Once that finishes, let's run Ollama with the new model. Drop that command in now. This takes us to a prompt where we can chat with the model. I'll ask it a random question. Watch how fast this is, literally instant. Don't get too excited though, it's not that fast when we plug it into OpenClaw. Type in /bye to get out of the prompt. Our next command will set Ollama to start up automatically. Pop that in now. Alright, we are all set, and now it's time to install OpenClaw. Drop that command in now. If prompted for your password, plug that in and let it cook. This will take a few minutes to complete and then take us straight into the onboarding process. Acknowledge that this is powerful and therefore risky. Select quick start. For model, select skip for now. Select all providers, then choose to enter manually. Delete the default entry and type in Ollama our model name. We will skip setting up any channels, skills, and hooks for now. Give it a sec, and the gateway will install and start up. Scroll up to the dashboard URL with token. Copy that and pop it into a browser. We don't have any model set up yet, so OpenClaw won't respond if we try chatting with it. We need to configure our Local Llama model now. On the left side, click on config. Scroll down and click on models, click on all. Scroll down to the mode section and click on add entry under providers. We will name it Ollama. For API, drop that down and select Ollama. For API key, this is important. Ollama doesn't use any keys, but OpenClaw will be expecting one. Just pop in any arbitrary value here, but don't leave it blank. For auth, select API key. For base URL, enter http://127.0.0.1:11434.
[4:06]This is the base URL for Ollama's API. Now scroll down to models and click add. Scroll down to context windows and enter 200,000. Now scroll down to ID and enter our exact model name. For Max tokens, pop in 8,192. Then give it a name, I use the model name here as well. All right, go ahead and save that, and then we can head back to chat for testing. Now we let it know our name and give it a name as well as a vibe and some general info. And voilà, we have a working OpenClaw with local LLM. No data leaving our machine, unless we choose so. Now I would be lying if I told you that this will work as well as hooking it up to Open AI or another paid LLM. But hey, there are tradeoffs. If privacy is at the top of your list, this might be an ideal setup for you. One thing I noticed is that this particular model had a hard time actually writing to my file system, compared to using Open AI, where it just works pretty much every time. After some wrestling with it, I was able to get it to create a basic little snake game for me. Just an example of the capabilities.
[5:30]Now that brings back memories. Just as I recall back in 10th grade English class. Not too shabby for running this straight from my little GPU. Let us know in the comments if you will be trying this out, and if so, what GPU are you going to use? I hope you all enjoyed the video, until the next one, take care everyone.
