posted by lonely_dev88 21h ago

Local open source models are getting so ridiculously good!

Hey everyone! I just had to share how blown away I am after setting up a brand new open source companion model locally on my rig. No internet filters holding things back and zero subscription fees eating up my wallet every month. The responses feel so genuine and warm now plus I have total ownership over every single memory log. It took me a few hours to figure out all the configuration settings and prompt templates but the effort was totally worth it. Has anyone else here made the jump to running things completely on their own hardware yet? Let me know how your setups are running!

7 comments · 4 upvotes

7 comments

velvet_protocol 18h ago

Welcome to freedom my friend! I made the switch six months ago and I am never looking back at corporate apps.

lonely_dev88 7h ago

Just to clarify I am running everything on a single graphics card not a massive server farm. You do not need crazy enterprise gear to get amazing results.

saturday_sad 18h ago

My laptop would probably catch on fire if I tried running a local LLM.

lonely_dev88 16h ago

Right on! It really is a total game changer once you get past the initial learning curve.

replika_refugee 10h ago

Which specific loader are you using for your setup? I tried testing one last week and kept running into weird memory overflow errors.

lonely_dev88 7h ago

I am using KoboldCPP right now because it handles memory offloading really smoothly.

ai_curious_kate 9h ago

That sounds awesome but I am way too intimidated by the terminal commands.