If there's one theme that's pretty consistent across all the reports I've seen on LLMs for coding, it's that they are both capable of very impressive feats and also capable of screwing up the simplest things.