Last week I hit a problem when launching LinkedIn on one of my systems. I moved over to another system and LinkedIn launched just fine. It gave me a perfect opportunity to compare a trace file of a good communication with a trace file when all hell broke loose.
Launching Wireshark on both systems, I hit LinkedIn again and waited until I had connected to the site on both systems. Again, on one system the performance was acceptable and on the other system it was pathetic.
WANT TO LEARN MORE? We offer on-demand, online and instructor-led courses on Wireshark and TCP/IP communications! Check out the links under "Training" on the menu for more information and sign up for our biweekly newsletter to know when future blogs, events, or freebies are announced. Every new sign up also gets five free Wireshark labs!
One of the first things I do when troubleshooting performance issues is to analyze the TCP handshake process. The TCP handshake provides us with some very valuable information, such as:
The Initial Round Trip Time (iRTT) which indicates the path latency between hosts
The Maximum Segment Size (MSS) which defines how much data each peer can receive after the TCP header
If Window Scaling will be used and what the scaling factor should be - both sides of a TCP connection must support Window Scaling in order for it to be used
If Selective Acknowledgments will be used - again, both sides of a TCP connection must support Window Scaling in order for it to be used
On both of the trace files (the good and bad traffic trace files), I applied a filter for packets in which the SYN bit was set on - tcp.flags.syn==1.
Next, I saved the trace files as:
LinkedIn-justsyns-ok.pcapng
LinkedIn-justsyns-bad.pcapng
Note: I also hid some columns so the screen shots would look better in this blog.
There are two key columns that I want in place, as well. These include:
Time since previous frame in this TCP stream (tcp.time_delta) - since both trace files were captured at the clients, measuring the time from the outbound SYN to the SYN/ACK provided me with this information. If I had exported all 3 packets of the TCP handshakes, I would have looked at Wireshark's tcp.analysis.initial_rtt field in the SYN/ACKs.
Stream index (tcp.stream)
Analyzing these two trace files provided me with some interesting information regarding the two hosts on which I was working and the two LinkedIn connection processes.
LinkedIn-justsyns-ok.pcapng (SYNs and SYN/ACKs Only)
If you want to follow along, download this trace file from http://blog-supplements.s3.amazonaws.com/LinkedIn-justsyns-ok.pcapng.
Launching LinkedIn required 12 TCP connections.
It looks like the iRTT values range from just over 10ms to just over 19ms. You can click the TCP Delta column heading to sort this column and easily see the lowest and highest iRTTs in this trace file.
My client is sending those SYN packets. We can see the client supports:
Maximum Segment Size of 1460
Selective Acknowledgments
Window Scaling with a multiplier of 256
LinkedIn-justsyns-bad.pcapng (SYNs and SYN/ACKs Only)
Grab this trace file at http://blog-supplements.s3.amazonaws.com/LinkedIn-justsyns-bad.pcapng.
Let's look at the trace file taken during the lousy LinkedIn launch now. This time I only see 9 TCP connections - I'm not certain, but I might have cancelled the loading process because it was taking just too long.
Ok - this is pretty interesting. Not only are most of the connections running over IPv6 on the system having connection problems, but the iRTT values are really terrible!
The iRTT values range from almost 28ms to over 2 seconds! Needless to say, this isn't looking good so far.
Notice that there are some retransmissions in this trace file. There are a number of TCP SYN/ACKs that have been retransmitted. In another blog, I will follow streams 5, 6, 7, and 8 to look more closely at those retransmitted SYN/ACKs.
Examining the TCP options advertised by the LinkedIn servers, we see the servers support:
Maximum Segment Sizes range from 1220 to 1460
Selective Acknowledgments
Window Scaling with a multipliers ranging from 128 to 4096
So far, this is really interesting. I can see a ridiculously high iRTT when one of my systems loaded the LinkedIn site. High path latency issues would definitely affect the performance.
Always remember to check the TCP handshake packets to determine if there are problems right at the start of the connections.
In the next blog, I'm going to examine TCP streams 5, 6, 7, and 8 a bit more closely.
Cheers!
Laura