The private repo mention in the comments there is kinda a good one, unfortunately. If someone runs a private gosum/goproxy (relatively common) and amends a tag (hopefully very uncommon but I have personally seen it happen at least three times) then the cache could be wrong because go.mod didn't have to change. Which is Bad™ but it depends on what edge cases they want to handle automatically, vs optimization for the majority case.
For well-behaving/stable/consistent setups I fully agree though, go.mod is both sufficient and better, and those other cases can probably just key off both instead. I think I've seen go.mod to change without go.sum changes (change an unused transitive dependency into a direct dependency), which can lead to your build needing something that wasn't cached because it was pruned in the previous version.
From a relatively naive outside perspective, it sounds like this would be pretty much only ever a self-inflicted probably (where "self" might be an org rather than an individual). What you're describing sounds almost like a force push to a private repo; if you're doing that, you might break things for anyone using it, so the risks should probably rest with you for asking so. "This breaks my setup if I modify history in a way that's expected to be immutable" isn't a super compelling argument for everyone behaving well to have to continue dealing with a suboptimal status quo.
For well-behaving/stable/consistent setups I fully agree though, go.mod is both sufficient and better, and those other cases can probably just key off both instead. I think I've seen go.mod to change without go.sum changes (change an unused transitive dependency into a direct dependency), which can lead to your build needing something that wasn't cached because it was pruned in the previous version.