What's the purpose of the signal safety number in the signal messenger?
I've read the official page on safety numbers but they are still unclear to me. Without verification, there still is end to end encryption, so what difference does it make? Is it that a phone number is easy to spoof so an adversary could pretend to be someone else but with a safety number verified you would know if it's not really them?
The signal safety number is actually the fingerprint of your public key.
By verifying this, you verify that the public key you are encrypting the messages to, is in fact the same as the keys used by the other party in the conversation.
If not verified, you could be encrypting your keys with a Man in The Middle's keys, which then proceeds to decrypt the messages, read them, and re-encrypt them with the recipient's key. You would be none the wiser.
By verifying that the public keys are identical for both participants, you remove the possibility of a MiTM attack.
It's explained in detail in this blog post from Signal.