Or only allow signed kernel modules. Aka secure boot.

This doesn't solve all vectors but afaics this will prevent non signed modules from loading.