I would expect that 4060ti to get about 20-25 tokens per second on Mixtral. I ca... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

pitched on Feb 9, 2024 | parent | context | favorite | on: OpenAI compatibility

I would expect that 4060ti to get about 20-25 tokens per second on Mixtral. I can read at roughly 10-15 tokens per second so above that is where I see diminishing returns for a chatbot. Generating whole blog articles might have you sit waiting for a minute or so though.

thrdbndndn on Feb 9, 2024 | [–]

Thanks, that sounds more than tolerable than "more than an hour"!

I also have the 16GB version, which I assume would be a little bit better.

cellis on Feb 9, 2024 | [–]

It depends on the context window, but my 3090 gets ~60/s on smaller windows.

visarga on Feb 9, 2024 | [–]

I get 50-60t/s on Mistral 7B on 2080 Ti

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact