We live in a culture where it matters a lot. We possess biology where it matters a lot. If we were robots which are about to be designed and programmed from scratch, you could ask if it's worth it to make us sexually dimorphic and ingrain it in our cultural programming. But you're too late, it's done already. Pretending it doesn't exist because you'd like it not to won't get you very far.