The ".PEM format" does not really exist as a standard. This is more "the PEM format" does not really exist as a standard. This is more "whatever OpenSSL does". Convert key to PEM comes from an old failed standard for Privacy Enhanced Mail (that's what the acronym means). These days, "PEM" really means: some text that looks like: -----BEGIN XXX----- [some Base64 stuff here] -----END XXX----- I.e. a header line that starts with ----- and contains the designation of the type of data (e.g. "RSA PRIVATE KEY"); a similar trailer line; and between these two lines, a binary object encoded in Base64.
For RSA private keys, you will encounter mostly two types of PEM-encoded formats. When the header contains "BEGIN RSA PRIVATE KEY" then this is a RSA private key in the format described by PKCS#1. When the header says "BEGIN PRIVATE KEY" (without the "RSA") then it uses PKCS#8, a wrapper format that includes the designation of the key type ("RSA") and the private key itself. In your case, if you see something that looks like PEM and begins with -----BEGIN RSA PRIVATE KEY----- then it is PEM; just put that in a text file, save it under some name (say "serverkey.pem") and configure Wireshark to use that file as server key. This is described in the Wireshark documentation. Mind some details, though:
Wireshark will probably not be able to read the file if it is encoded in UTF-16 (what Windows somewhat improperly calls "Unicode"). In UTF-16, each character is encoded over two bytes (or four bytes for some characters like Pahawh Hmong). If you are using Windows' notepad, upon saving the file, choose the "ANSI" or "UTF-8" encoding. Knowing the server's private key is not enough to decrypt the data if the client and server use a "DHE'' or "ECDHE" cipher suite. If the client and server agree to use such a cipher suite and you still want to intercept the data, then you must make an active attack (a Man-in-the-Middle) in which you impersonate the server when talking to the client, and the client when talking to the server. This is a lot more work and Wireshark won't help you much there whatever OpenSSL does". PEM comes from an old failed standard for Privacy Enhanced Mail (that's what the acronym means). These days, "PEM" really means: some text that looks like: -----BEGIN XXX----- [some Base64 stuff here]-----END XXX----- I.e. a header line that starts with ----- and contains the designation of the type of data (e.g. "RSA PRIVATE KEY"); a similar trailer line; and between these two lines, a binary object encoded in Base64.
For RSA private keys, you will encounter mostly two types of PEM-encoded formats. When the header contains "BEGIN RSA PRIVATE KEY" then this is a RSA private key in the format described by PKCS#1. When the header says "BEGIN PRIVATE KEY" (without the "RSA") then it uses PKCS#8, a wrapper format that includes the designation of the key type ("RSA") and the private key itself.
In your case, if you see something that looks like PEM and begins with -----BEGIN RSA PRIVATE KEY----- then it is PEM; just put that in a text file, save it under some name (say "serverkey.pem") and configure Wireshark to use that file as server key. This is described in the Wireshark documentation. Mind some details, though: Wireshark will probably not be able to read the file if it is encoded in UTF-16 (what Windows somewhat improperly calls "Unicode"). In UTF-16, each character is encoded over two bytes (or four bytes for some characters like Pahawh Hmong). If you are using Windows' notepad, upon saving the file, choose the "ANSI" or "UTF-8" encoding. Knowing the server's private key is not enough to decrypt the data if the client and server use a "DHE" or "ECDHE" cipher suite. If the client and server agree to use such a cipher suite and you still want to intercept the data, then you must make an active attack (a Man-in-the-Middle) in which you impersonate the server when talking to the client, and the client when talking to the server. This is a lot more work and Wireshark won't help you much there.
