Explain RSA signature algorithm.

First a quick review of RSA:

RSA encryption

In RSA encryption, you encrypt a plain text M by raising it to a public key e in a publicly known modulus N:

encryption = Me mod N
To decrypt, you raise the encrypted text to the private key d:
decryption = (Me)d = Med mod N
The private and public keys are designed so that xed = x mod N for (almost) all x.
RSA signature

To sign a message M, you "encrypt" it with your private key d:

signature = Md mod N
To check whether you have actually signed it, anyone can look up your public key and raise the signature to its power:
signaturee = (Md)e = M mod N

If the result is the message M, then the verifier knows that you signed the message.

Conclusion In effect, in verifying a signature, the verifier is performing the same operation as someone who was decrypting the signature using the public key.