The reason why having your number out there is a problem is because, if you think about it, most people who use multi-factor authentication, the second factor tends to be a code sent to their phone number. If your phone number is stolen as a result of a SIM swap, which is not as difficult as one would think, for a few bucks, unfortunately, people call people at mobile providers and get them to switch things based on the fact that they go, "I'm sorry.
I forgot my password. This is my phone number, and I just got a new device, by the way, so can you please transfer to my new device?" And then, all of a sudden, you don't get the code. We've had cases where people have lost millions in cryptocurrency because the code was sent to the phone number that had been stolen by a hacker.