It’s great to be at Cisco Live this week. The entire Chappell University team is hanging out at the Profitap Booth 3035 giving away the TCP Analysis architectural poster and the new Wireshark Troubleshooting Cheat Sheet!
If you are reading this the week of the conference, I am also presenting a set of 15-minute TechTalks at the Profitap Booth 3035. Get the schedule here.]
Note: If you are a Chappell University All Access Pass subscriber, you already have access to these sessions - they are in the new Lunch and Learn section of the catalog!
The new 2-sided Troubleshooting Cheat Sheet contains some of my favorite display filters to detect network problems and a series of graphs identifying network issues.
Click the area below to download the Troubleshooting Cheat Sheet and practice files.
Let’s focus on troubleshooting and the contents of the Wireshark Troubleshooting Cheat Sheet in this blog.
First… the golden rule of good network traffic behavior:
“You should only ask for something once and you should get what you ask for in a timely manner.”
This one phrase covers so many areas on which to concentrate when people complain about network performance.
“You should only ask for something once…”
You shouldn’t see network hosts asking for something again and again on a network. This is a definite sign something is wrong.
For example, if a host asks for the same TCP Sequence Number more than once (Duplicate ACKs), this indicates packet loss.
Display filter - TCP: Expert Indications: tcp.analysis.flags
This display filter will show you all the packets that match the TCP analysis flags for Wireshark’s Expert Information system. Try this filter on wwb001-sharking.pcapng.
If you just want to view Duplicate ACKs, use the filter tcp.analysis.duplicate_ack.
Another example would be repeated DNS queries. A host will send duplicate DNS queries when the response did not arrive in time.
When I created the Troubleshooting Cheat Sheet, I ran out of room for display filters. This is one I would add to the Cheat Sheet, if I had another line available.
Wireshark can detect DNS request and response retransmissions using the filter dns.retransmissions==1.
Interestingly, the version of Wireshark on which I am working can detect DNS request and response retransmissions, but it does not have a coloring rule to call your attention to these packets. You can always add a coloring rule for dns.retransmission==1 (shown below).
Note: Interestingly, in the version on which I am working (Version 3.0.2), you can also use dns.retransmission as your filter. The “1” indicates that the condition is true, but right now there is no “0” indicating a DNS packet is not a retransmission. Instead, the field is just missing. In case the dns.retransmission==0 is implemented in the future, consider using the full dns.retransmission==1 filter, not just dns.retransmission.
“...you should get what you ask for…”
You should not see connection or service refusals. For example, a TCP Reset in response to a TCP SYN packet is an indication that something is wrong. Either the service at the target is not running, or something (such as a firewall) is refusing the connection request. Perhaps the client is misconfigured and asking the wrong host for a service.
Display Filter - TCP: Connection refusal or ACK scan
tcp.flags.reset==1 && tcp.flags.ack==1 && tcp.seq==1 && tcp.ack==1
On the Troubleshooting Cheat Sheet, I provided a filter to find these service refusals. Essentially, this filter is based on the fact that the Sequence Number and Acknowledgment Number fields is set to 1 indicating no data has been sent from either host. In addition, this filter looks for the TCP Reset bit and ACK bit set to 1, which we expect to see in a service refusal.
In the image that follows, frame 185 is a “normal” Reset to close a connection. The Sequence and Acknowledgment Number fields indicate that data has been sent from both sides of the connection.
Frames 193 and 195 are TCP service refusals, however. Note that the Sequence and Acknowledgment Number fields are 1 in both of those packets. This is an indication that no data has been sent across this connection.
These TCP resets are connection refusals. You also want to look for service refusals.
The Troubleshooting Cheat Sheet contains four display filter examples to catch DNS, HTTP, SMB, and SIP service refusals, as shown below.
DNS: Error responses
dns.flags.rcode > 0
HTTP: HTTP error responses (codes over 399 indicate client/server errors)
http.response.code > 399
SMB: Error responses (SMBv1/2/3)
smb.nt_status > 0 || smb2.nt_status > 0
SIP: Error responses
sip.Status-Code > 399
“...in a timely manner.”
Finally, you should always pay attention to time when troubleshooting. Sometimes, application traffic is just ugly - perhaps there are numerous requests for something over and over again. True, we don’t want to see a host ask for something repeatedly, but if the total amount of time wasted by these duplicate requests is miniscule, it is most likely not the cause of complaints on the network.
The Troubleshooting Cheat Sheet provides filter examples to detect service response time delays for DNS, HTTP, and SMB.
DNS: Slow DNS response (example: over 100ms)
dns.time > .1
HTTP: Slow HTTP responses (example: over 100ms)
http.time > .1
SMB: Slow SMB responses (example: over 100ms)
smb.time > .1 || smb2.time > .1
Adding these service response time fields as columns makes it easy to determine the fastest and slowest response times in the trace file. In the image that follows, I have filtered on dns.time > .1 (greater than 100ms). By right clicking on the DNS [Time: x] field (only in DNS response packets) and selecting Apply as Column, we can now sort this new column to determine the highest and lowest response times.
We’ve only looked at nine of the thirty-one display filters on the Troubleshooting Cheat Sheet. Download the cheat sheet and supplemental files now and practice detecting network issues quickly!