In the RSA algorithm, it needs to take two large primes (p and q), then produce n = pq (n is called the modulus). Choose a number, e, which less than n and relatively prime to (p-1)(q-1), which means e and (p-1)(q-1) have no common factors except 1. Find another number d such that (ed - 1) is divisible by (p-1)(q-1). The values e and d are called the public and private exponents respectively. The public key is the pair (n, e); the private key is (n, d). The factors p and q may be destroyed or kept with the private key, because people can break RSA cryptosystem with p and q.

It is difficult to find the private key d with the public key (n, e). But if someone could factor the large integer n into p and q, then the private key d will be found. Hence, the security of the RSA system is based on the hard problem of factoring a large integer.

Encryption
In this case, Alan wants to encrypt a message m and send it to Bobby:

1. Alan creates the ciphertext c: c = m^e mod n, where e and n are Bobby's public key.
2. Alan sends c to Bob.
3. To decrypt the ciphertext c, Bobby use his private key d to find the plaintext m: m = c^d mod n.
4. Since only Bobby knows d, only Bobby can decrypt and read this message.

Digital Signature
In this case, Alan wants to send a message m to Bobby in such a way that Bobby is assured the message is from Alice:

1. Allan creates a digital signature s that: s = m^d mod n, where d and n are Alan's private key.
2. Alan sends m and s to Bobby.
3. Bobby, received the message (m) and signature (s), decrypts the signature with Alan's public key (n,e) to recover the message (m), as m = s^e mod n.
4. Bobby compares the result to the message gets by decrypted the signature (s) and the original message (m). If they are exactly equal, the signature has been successfully verified and Bobby can be sure the message that comes from Alan. If they are not equal, then the message is not originated by Alan or was altered, and he rejects the message.

There are a few possible methods of "breaking'' the RSA cryptosystem.

An attacker would discover the private key corresponding to a given public key; this would enable the attacker both to read all messages encrypted  and to make signatures. It needs to factor the public modulus, n, into its two prime factors, p and q. From p, q, and e, the public exponent, the attacker can easily get d, then the attacker will get the private key. The hard part is factoring n; the security of RSA depends on the hard problem of factoring the large integer n. It should be noted that hardware improvements alone will not weaken the RSA cryptosystem, as long as appropriate key lengths are used.

Another way to break the RSA cryptosystem is to find a technique to compute eth roots mod n. Since c = m^e mod n, the eth root of c mod n is the message m. The attacker can recover encrypted messages and make signatures even without finding the private key. This attack is not known to be equivalent to factoring. No general methods are currently known that attempt to break the RSA cryptosystem in this way. However, in special cases where multiple related messages are encrypted with the same small exponent, it may be possible to recover the messages.