I was able to run llms on a 1080. They were admittedly small ones, but a 3090 is enough to be usable. That said, I’m not convinced it’s a good idea to use it for therapy. I expect it’s about as useful as talking into a mirror.
I think I’m at a 3060 or so and it works decently depending on the model. I can generally get away with around 13B, or some 20+ Q4 or so but they get real slow by that point.
It’s a lot of messing around to find something that performs decent while not being so limited as to get crazy repetitive or saying loony things.
Can I run anything on a 3090 or I need a beefier gpu
I was able to run llms on a 1080. They were admittedly small ones, but a 3090 is enough to be usable. That said, I’m not convinced it’s a good idea to use it for therapy. I expect it’s about as useful as talking into a mirror.
Does journaling count? It’s got a small positive benefit according to https://pmc.ncbi.nlm.nih.gov/articles/PMC8935176/
Talking things out to ourselves can often be useful. It could certainly be good for that.
I think I’m at a 3060 or so and it works decently depending on the model. I can generally get away with around 13B, or some 20+ Q4 or so but they get real slow by that point.
It’s a lot of messing around to find something that performs decent while not being so limited as to get crazy repetitive or saying loony things.
3090 is great because it has about the same amount of vram as newer cards, and vram is what determines which models you can run on it.
That’s the best gpu for this usecase. Maybe use 2x if you want the best of the best.