No, if things get bad enough, locals have a way of terrorizing any stable uses of those infrastructure. Again ask the US and USSR. What the hell are the Chinese going to do, perform a Gaza on the locals?
Not the poor countries that China supposedly "owns". Countries like Pakistan can't even keep the lights on. Why would they care about rare earth magnets? China provides gifts and in exchange Pakistan provides a pinky promise that will disappear as soon as the checks stop. They can't control their borders much less their crazies. And those crazies will terrorize any Chinese that try to start shit and will make it impossible to reliably take advantage of any infrastructure that China may want to use.
