I am running 70B models on M2 Max with 96 GB of RAM and it works very well. As H... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		neop1x on Sept 23, 2024 \| parent \| context \| favorite \| on: Forget ChatGPT: why researchers now run small AIs ... I am running 70B models on M2 Max with 96 GB of RAM and it works very well. As HW evolves, it will become a standard

creata on Sept 24, 2024 [–]

Out of curiosity, what degree of quantization are you applying to these 70B models?

neop1x on Sept 24, 2024 | [–]

Q4_K_S. While not as good as top commercial models like chatgpt, they are still quite capable and I like that there are also uncensored/abliterated models like Dolphin.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact