In the 3rd part of the blog series Certificate Authorities were discussed in depth. If anyone is here without reading that post I highly recommend to read that. In this post the whole SSL/TLS handshake in action is practically explored.Before that, the key takeaways from the last part were:
- TLS works on symmetric key
- Symmetric key can be shared via secure key exchange algorithm
- Key exchange can be spoofed if the request is intercepted
- Use of Digital signature for authentication
- Certificate Authority and chain of trust.
In this post a tool named WireShark is used to see the network traffic. I am using a Linux(Ubuntu 16.04) machine and WireShark can be easily installed with the following command:
$ sudo apt install wireshark
Open WireShark as sudo and choose the interface in which the internet connection is served. In my case it is eth0. Then press the “Start capturing packets” button top right corner of WireShark. Wireshark will immediately start capturing whole traffic going through the machine. Now let’s load github.com in a browser. Github uses TLS for all the communication. So it will redirect to https and github.com will be loaded in https. Now close the browser and see what WireShark has got.
Resolving the DNS
This isn’t part of TLS but let’s just see it in WireShark.
I have set Google DNS as my DNS server. It has the address 184.108.40.206. In the image it can be seen that a request has been sent to 220.127.116.11 querying the A address of github.com. By A address we mean the IP of the Github we want to connect.
DNS server responded with the IP of github.com as 18.104.22.168. The blue selection shows the respective part. Now the browser has got the destination IP which will be used to connect to the server.
Initiate TLS handshake
Once the IP is resolved, the browser will request the page via http. If the server supports TLS, then it will respond to the browser by saying a protocol upgrade request. The new location, say https://github.com, will be specified with the port number 443. Browser will then initiate the TLS handshake request. Most modern browsers remember the last connection with the web server. If the last connection was made via https, then next time browser will automatically initiate the https requesting without waiting for the server.
The TLS handshake is divided into various steps as follows
- Client Hello
- Server Hello
- Sharing the certificate and server key exchange
- Change cipher spec
- Encrypted handshake
From here onwards, I will highlight the topic of discussion in blue color in the images.The Client Hello is shown in the image below.
We know that TLS is a protocol implemented above TCP. TLS itself is layer and the bottom layer is called the Record protocol. That means all the data are considered as records. Over the wire, a typical record format would look like:
HH V1:V2 L1:L2 data
- HH is a single byte which indicates the type of data in the record. Four types are defined: change_cipher_spec (20), alert (21), handshake (22) and application_data (23).
- V1:V2 is the protocol version, over two bytes. For all versions currently defined, V1 has value 0x03, while V2 has value 0x00 for SSLv3, 0x01 for TLS 1.0, 0x02 for TLS 1.1 and 0x03 for TLS 1.2.
- L1:L2 is the length of data, in bytes (big-endian convention is used: the length is 256*L1+L2). The total length of data cannot exceed 18432 bytes, but in practice it cannot even reach that value.
In the image, it can be seen that the content type is Handshake, TLS version recommended 1.0 and the length of data as 512. The real data lies in the drop down named “Handshake Protocol: Client Hello”. Let’s go ahead and observe what data is being shared in the Client Hello.
Contents of Client Hello
The following details are shared with the server by the browser.
A list of client supported protocol versions in the order of preference. The prefered one is the maximum protocol version that the client wishes to support.
A 32 byte data in which first 4 bytes represent the current datetime in epoch format. For those who don’t know what is epoch time, it is the number of seconds since January 1, 1970. The rest 28 bytes are generated by a cryptographically strong random number generator(For example, /dev/urandom in Linux). The client random will be used later. For now, just keep it in mind.
This field will remain null if the client is connecting to server for the first time. In the above image, you can see that a session id is being sent to the server. This happens because I have previously connected to github.com via https. During that time server will map the symmetric key with session id and store the session id in client browser. A time limit will be set for the mapping. If the browser connects to the same server in future (of course before the time limit is expired) it will send the session id. Server will validate it towards the mapped session and resume the session with previously used symmetric key. In that case, a full handshake is not necessary.
The client will also send the list of cipher suites which are known to it. The cipher suites are arranged in the order of preference by the client. But it is completely up to the server to follow the order. There is a standard format for cipher suites used in TLS.
Let’s take an example from the list and analyse.
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
- TLS: It simply means the protocol which is TLS
- ECDHE: It is the key exchange algorithm.
- ECDSA: Signing or the authentication algorithm
- AES_128_GCM: It is called the bulk encryption algorithm. The symmetric key encryption algorithm is AES with key of length 128 bit. AES is a block cipher, meaning the input plain text is encrypted in block of fixed length. Each block after encryption is sent in sequence and decrypted in a similar fashion. As per standard a block is AES is fixed to 128 bits. But the input plaintext need not be always a multiple of 128. So we might want to add padding to the last block in order to fix it to 128 bits. Apart from that, to improve entropy some random bits are usually added to the plaintext before encryption. It is called the Initialisation Vector (IV). There are many algorithms to add IV and implement padding on the blocks. In our example Galois/Counter Mode (GCM) is used. Probably it isn’t a good idea to explain GCM mode in detail and make things complex.
- SHA256: The Message Authentication Code (MAC) algorithm. We will talk about MAC in detail.
In order to reduce the bandwidth compression can be used. But there were successful attacks on TLS in which parameters that are sent with HTTP headers can be captured when compression is used. Cookies can be hijacked with this attack and the vulnerability was termed CRIME. As of TLS 1.3, TLS compression is disabled by the protocol.
Additional parameters such as server name, padding, supported signing algorithms, etc can be specified as extensions. Feel free to do a research on the contents specified as extensions.
These are the parts of a Client Hello. It is followed by an acknowledgement from server if it has received the Client Hello. Then the server will send Server Hello.
After receiving the Client Hello server has to send the Server Hello message. The server will check the conditions specified the the Client Hello such as TLS version and algorithms. If all the conditions are acceptable and supported by the server, it will send its certificate along with other details. Otherwise the server will send a handshake failure message.
In the image, it can be seen that the server responded with 0x0303 means that server agreed to use TLS 1.2. Let’s inspect the records in a Server Hello.
Contents of Server Hello
The Server Hello message contains the following information.
We will discuss the important parameters here.
Server will choose the TLS version specified by the client if it can support it. Here TLS 1.2 is chosen
Similar to the client random, the server random will also be of 32 byte length. First 4 bytes represent server’s Unix epoch time followed by the 28 byte random number. The client and server randoms will be used to create the encryption key which we will explain later.
Remember we have sent the sent the supported cipher suites to github.com in the Client Hello? Github picked up the first one from the list. I.e.,
Server store the agreed session parameters in a TLS cache and generate the session id corresponding to it. It is sent to the client along with the Server Hello. Client can key the agreed parameters with this session id. There will be an expiry defined for this. Client will include this id in the Client Hello. If the client connects again to the server before this expiry, server can check for the cached parameters corresponding to the session id and reuse them without the need of a full handshake. This is highly useful since both the server and client can save a lot of computational cost.
There is a downside for this approach when in comes to applications with huge traffic such as Amazon and Google. There will me millions of people connecting to the server each day and server has to keep a TLS cache of all of their session parameters with the session key. This is a huge overhead. In order to solve the issue concept of Session Tickets were introduced. Here, client can specify in the client hello if it supports session tickets. Then, server will create a New Session Ticket and encrypt the session parameters with a private key which is known only to the server. This will be stored on the client and hence all the session data is stored only on the client machine. The ticket is still safe since the key is known only to the server.
This data can is included in the Client Hello as an extension named SessionTicket. In our case this parameter was empty. Because either it was the first time the browser connecting to github.com or the previous session was expired.
If supported, the server will agree on the Client’s preferred compression method. Here, you can see that server responded with a null response means no compression is required.
The server does not send any certificate in the ServerHello message; it sends certificates in the aptly-named Certificate message.
The Server Certificate message
In our case, the certificate message is 3080 bytes long. There’s no wonder it is the server certificate with all the information. The server sends a complete list of certificate in the order of chain of trust. First one of this chain is the server’s certificate followed by the certificate of the Intermediate CA who issued the server certificate. Then the certificate of next intermediate CA and it continues until the certificate of root CA. The server has the provision to not send the certificate of Root CA since the browser, in most cases, can identify the Root CA from any of the intermediate CA. Read the last part of the blog series for a detailed explanation of chain of trust.
In our case you can see that the first certificate is of github.com and the second one is of the intermediary Digicert SHA2 Extended Validation Server CA. Check the id-at-commonName parameter in the following image.
Let’s analyse the contents of a certificate and see how the browser verifies it.
Contents of a certificate
The certificate is sent to the browser and hence we can view the certificate of Github when we visit github.com. From Firefox,
The intermediate CA and Root CA of github can be viewed by clicking on the Details tab.
Let’s understand what are these fields and for what purpose they are used.
Version and Serial Number
Version represents which version of X.509 standard is used. X.509 is the standard used to define the format of public key certificates. There are 3 versions for X.509 and github uses version 3, the latest one.
From RFC 5280, the serial number MUST be a positive integer assigned by the CA to each certificate. It MUST be unique for each certificate issued by a given CA (i.e., the issuer name and serial number identify a unique certificate). CAs MUST force the serialNumber to be a non-negative integer.
Certificate Signature Algorithm and Value
The browser need to know the algorithm of signature in order to verify the signature. If it is signed with RSA, then the same algorithm is necessary to verify the signature. For Github, PKCS #1 SHA-256 With RSA Encryption is used. Which means SHA-256 is used to generate the hash and RSA is used for signing it.
From our last article, the certificate data is hashed with SHA-256 algorithm and this hash is signed with the private key of Github using RSA encryption.
This field holds the details of the authority who issued the certificate. Certificates for Github are issued by Digicert’s intermediate CA.
This field has two values Not Before and Not After. The values are self explanatory. Certificate is invalid if the current datetime is not between these values. Browser will not trust that certificate.
Subject Public Key Info
This field carries the Public key and the algorithm used to generate the public key. This Key is used to exchange keys which we will discuss later.
There are two fingerprints SHA 1 and SHA-256 and both of them are generated by the browser and is never sent to the server. These Fingerprints are generated by hashing the DER format of certificate with SHA 1 and SHA-256 functions respectively. We can verify this by downloading the certificate to our machine and applying the Hash function.
Click the Export button on the left bottom corner of details tab to download the certificate. Save it with .crt extension. And on terminal run the following to generate the fingerprints of the certificate.
$ openssl x509 -noout -fingerprint -sha256 -inform pem -in [certificate-file.crt] $ openssl x509 -noout -fingerprint -sha1 -inform pem -in [certificate-file.crt]
This should produce the same result as what you see in the browser. These values are not part of the certificate, rather they are computed from the certificate. The Fingerprint of Root CA certificate will be hardcoded in the browsers and hence they can be easily cross verified. Apart from this, these fingerprints are mostly used for identifying and organising certificates. Do not mix up Signature with Fingerprints.
The certificate information we discussed here are about the server certificate of github.com. The certificate of Github’s intermediate CA will be also be sent along to the client in the same request and all the above fields apply to that certificate also. You can check that by going to the details tab and clicking on the intermediate CA as shown below.
Server Key Exchange
Followed by the Server Hello and Certificate message there is optional Server Key Exchange. This message is sent only if the certificate provided by the server is not sufficient to allow the client to exchange a pre-master secret. Let’s see why github.com had to send a Server Key Exchange message.
We have seen that github.com preferred the ciphersuite TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 for the session. Which means Elliptic Curve Diffie Hellman algorithm is used by the parties to exchange the key. In Diffie-Hellman, the client can’t compute a Pre-master secret on its own; both sides contribute to computing it, so the client needs to get a Diffie-Hellman public key from the server. (Don’t get confused about the term Pre-Master Secret, we will discuss about it deeply below.) When using Elliptic Curve Diffie-Hellman, that public key isn’t in the certificate. So the server has to send the client its DH public key in a separate message so that the client can compute the premaster secret. This can be seen in the above image. Note that this key exchange is also secured by a signature.
Once the Server Key Exchange is complete, server will send the Server Hello Done message. Client will start to compute the Pre-Master Secret. Let’s see how.
How to compute the Pre-Master Secret
The Pre-Master Secret computation depends on the type of key exchange algorithm agreed upon. When using RSA for key exchange, the Pre-Master secret is computed from the client side, i.e., the browser. The client generates the 48-byte premaster secret by concatenating the protocol version (2 bytes) and some bytes that the client generates randomly (46 bytes). The client is supposed to get these 46 bytes from a cryptographically secure Pseudo Random Number Generator(PRNG). In practice, this means using the PRNG offered by the operating system such as /dev/urandom. Then this Pre-Master secret is encrypted with server’s public and shared, so that the server can later use this to create the Master Secret.
But, in case of Github, as explained above Diffie-Hellman algorithm is used for key exchange. Things are little different here. The server instantly generates a pair of DH private-public keys. The public key is then shared with the client. This is the Server Key Exchange message as explained above. In response, client will also create a DH key pair and share the public key with server through the Client Key Exchange message as shown below.
You can see the client public key being shared. Now, if you understood the working of Diffie-Hellman algorithm you know that the client and server can reach on a common key from these shared public keys. The newly generated key is called the Pre-Master key.
There is an advantage of using Diffie Hellman algorithm for TLS key exchange. Both the client and server generate a new key pair for each fresh session. And the private keys of both client and server will be deleted immediately once the Pre-Master secret is computed. That means the private key can never be stolen afterwards ensuring perfect forward secrecy.
Client Key Exchange
We already discussed above that the client’s DH public key is shared to the server via the Client Key Exchange message. But if RSA was used, then the client will compute the Pre-Master secret by its own as described above, encrypt it with Server’s public key (RSA public key) and send it back to the server through the Client Key Exchange message. Server can then decrypt it with its private key. Whatever algorithm be, at this point, both the client and server have reached on a common Pre-Master secret. Once this is complete client will send the Change Cipher Spec message as shown below.
Let’s go forward and see how the Master Secret is computed from the Pre-Master secret.
How to compute the Master Secret
What all random data do the client and server have now? The Pre-Master secret and the random values shared by the client and server during the Hello message (remember?) Both the parties will compute Master Secret using these values using a PRF (Pseudo Random Function). According to RFC 5346,
master_secret = PRF(pre_master_secret, "master secret", ClientHello.random + ServerHello.random) [0..47];
pre_master_secret – The 48 byte Pre-Master secret computed by both parties.
“master secret” – It is simply a string whose ASCII bytes are used.
ClientHello.random – The random value shared in client hello
ServerHello.random – Random value shared in server hello.
The size of Master secret will be always 48 bytes long. Well, no more confusions so far. Both the parties can use the Master secret to encrypt the data and sent them back and forth. Agree, but the procedures are not over yet. Do you think using the same key on both sides is a good idea? Of course not! TLS uses separate keys for client and server and both of them are derived from the Master secret itself. On other words, Master secret is not directly used to encrypt data. Instead separate encryption keys are used for client and server. Since both parties will have both keys, the data encrypted by server with its key can be decrypted by client with ease and vice versa.
It is not over. TLS have additional security mechanism for symmetric key encryption.
Message Authentication Code (MAC) and TLS Data Integrity
There are two possible attacks an eavesdropper can perform on encrypted data in transit. Either try to decrypt the data or try to modify it. As long as the key as secure, we assume decryption is nearly impossible. But what about data modification? How the client and server know that the received data is not modified by an attacker? As said, TLS does more than just encrypting data. TLS also protects the data from undetected modification. On other words we can say TLS checks for data integrity also. Let’s see how it is done.
When the server or client encrypt the data with Master Secret, it also computes a checksum(hash) of the plain data. This checksum is called Message Authentication Code(MAC). The MAC is then included in the encrypted record before sending it. A key is used to generate the MAC from the record in order to ensure that the attacker in transit cannot generate the same MAC from the record. Hence the MAC is called an HMAC(Hashed Message Authentication Code). The other end, upon receiving the message, the decrypter will separate the MAC from the plain text and computes the checksum of the plain text with it’s key and compares it with the received MAC. If a match is found, then we can conclude that the data is not tampered in transit.
It is necessary to have the Client and server to use the same hashing algorithm to create and verify the MAC. Remember the last part of the ciphersuite that Github agreed upon?
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256. I.e., SHA256 is the hash function used to process HMAC. To improve the security seperate MAC keys are used by the client and server. Let’s see what all are those.
MAC Keys and IV keys
As per the spec, there are 4 keys used to encrypt and verify the integrity of each message. They are,
- Client write encryption key: Used by client to encrypt data and server to decrypt data.
- Server write encryption key: Used by server to encrypt data and client to decrypt data.
- Client write MAC key: Used by client to create MAC and server to verify MAC.
- Server write MAC key: Used by server to create MAC and client to verify MAC.
These key blocks are again generated by the same PRF on the master-key over and over until enough bytes have been created for the keys.
key_block = PRF(SecurityParameters.master_secret, "key expansion", SecurityParameters.server_random + SecurityParameters.client_random);
As you can see, along with the client-server random values and the string “key expansion”, the master secret is also used to increase the entropy of the keys. The PRF can generate keys of arbitrary length. This is useful since the different hash functions have different length by default. In our case SHA256 is used which is 256 bits. But MD5 is used the default length becomes 128 bits.
Apart from this, we know that we are using AES with GCM algorithms, which is a block cipher and it requires a set of bits to use as Initialisation Vector(IV). While discussing ciphersuites we have mentioned that IV is used to improve the entropy of AES encryption. In other words IV helps to generate different ciphertext when the same file is encrypted multiple times. These random bytes are also generated by the same PRF and are termed as client write IV and server write IV. The terminologies are self explanatory. I haven’t researched more on the details of IV because it is a huge topic and beyond the scope of this article.
Generating test data
Both parties have the encryption keys and we are ready to encrypt. But before taking TLS to application layer, like every process, we need to test and verify that the client encrypted data can be decrypted by server and vice versa. In order to do that the client will compute a 12 byte verify_data using the pseudo-random function (PRF) as follows.
verify_data = PRF(master_secret, "client finished", MD5(handshake_messages) + SHA-1(handshake_messages) ) 
Where handshake_messages is the buffer of all the handshake messages. The above is true for TLS until version 1.2. There are slight changes from version 1.2. i.e., length of verify_data depends on the cipher suite and it is not always 12. Any cipher suite which does not explicitly specify verify_data_length has a verify_data_length equal to 12. Also, the MD5/SHA-1 combination in the pseudorandom function (PRF) has been replaced with cipher-suite-specified PRFs. So as per latest spec,
Verify_data = PRF(master_secret, finished_label, Hash(handshake_messages)) [0..verify_data_length-1];
So we have the test data, keys and algorithm to encrypt the test data. All the client have to do is encrypt the test data with AES with the client encryption key (or simply, the client write key). An HMAC is also produced as explained above. Client take the result and add a record header byte “0x14” to indicate “finished” and sends it to the server via the Client Finished message. This is the first message protected by the algorithms and keys negotiated between the entities and the last handshake message sent by the client. Since the message is completely encrypted, WireShark will only see the encrypted content and it calls the Finished handshake by the name Encrypted Handshake Message as follows.
Verifying the negotiation
The server does almost the same thing. It sends out a Change Cipher Spec and then a Finished Message that includes all handshake messages. The Change Cipher Spec message marks the point at which the server switches to the newly negotiated cipher suite and keys. The subsequent records from the client will then be encrypted. Along with that, the server Finished Message will contain the decrypted version of the client’s Finished Message. Once the client receive this data, it will decrypt it with the server write key. Consequently, this proves to the client that the server was able to successfully decrypt our message. Kaboom! We are done with the TLS handshake.
All the encryption will be based on the negotiated algorithm. In our case the algorithm was AES_128_GCM. There is no point in explain it in depth here, because when it comes to some other website the algorithm specified by that server will be different. If you are interested in knowing how each of these algorithms work wikipedia has a list of them. I am also learning Cryptography through TLS basics.
Encrypted application data
So we are in application layer now. If you have an internet connection with moderately good speed, we have passed only a couple of 100 milliseconds. Imagine how much things happened in such a short span of time?
The page I requested was the homepade a.k.a www.github.com. So the plain text request show in the developer tools of Mozilla Firefox is,
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0
Accept-Encoding: gzip, deflate, br
See below how it looks like in the wire:
The first 3 byte 17 03 03 represents the content type(application data) and the TLS version (TLS 1.2).
We are done!
So, that’s it. We are done.
In the next part of the series I will add very few additional things that I couldn’t include in this post. I’d also post structured reference links which can be useful to learn Cryptography in TLS.
I wanted to write a few more words here. The whole article is written on my interest to understand TLS. That means everything I’ve learned/understood are written here as well. This can be incomplete, there can be mistakes, there can be mixed opinions. Whatever be it is please share them in the comment box. I am excited to learn more with all of you!
Here are a few links for more learning