You can't be sure. There is no silver bullet anywhere in security.

The better approach is to turn that question around and ask what extra security can you gain by not checking the signatures? None? Then it's best to check. Worst case, it doesn't help you at all. Best case, it saves your ass. A win on average, I'd say.