SuspiciousCarrot78@aussie.zone to Selfhosted@lemmy.worldEnglish · 1 day agoDo you host your own AI?message-squaremessage-square189fedilinkarrow-up1168arrow-down136file-text
arrow-up1132arrow-down1message-squareDo you host your own AI?SuspiciousCarrot78@aussie.zone to Selfhosted@lemmy.worldEnglish · 1 day agomessage-square189fedilinkfile-text
minus-squaretristynalxander@mander.xyzlinkfedilinkEnglisharrow-up3·18 hours agoIt’s not that hard to use llama.cpp directly anyway. Why would I use a wrapper when I can just run a python script?
minus-squareBlackLaZoR@lemmy.worldlinkfedilinkEnglisharrow-up1·edit-25 minutes agoI use LMStudio, because it has quality of life improvements like nice GUI and huggingface search engine. Also they have Vulkan backend that at least on 7900XTX is ~10% faster than rocm (on LLama 3 8b Q4_0 it gets 115Tokens/s vs 105 on rocm)
It’s not that hard to use
llama.cppdirectly anyway. Why would I use a wrapper when I can just run a python script?I use LMStudio, because it has quality of life improvements like nice GUI and huggingface search engine. Also they have Vulkan backend that at least on 7900XTX is ~10% faster than rocm (on LLama 3 8b Q4_0 it gets 115Tokens/s vs 105 on rocm)