Lesson 12 - Layer 2 Connectivity Troubleshooting Part 2
Lesson 12 - Layer 2 Connectivity Troubleshooting Part 2
In the previous lesson I attempted to show you how to go about a performance problems we might experience due to the duplex mismatch. It was our first trouble ticket in the series on layer 2 problems.
In this lesson I am going to create another issue to show you how your current skills can be practically useful in diagnosing network connectivity problems.
NOTICE
The steps presented in this lesson are not the ALL possible diagnostics you can do. And they do not have to be done in this specific order. I am merely listing some logical steps which might be useful in order to 'nail down' the root cause of the problem.
Trouble Ticket 2
Some client computers have problems accessing FTP server located in the same network 192.168.1.0/24. This behavior is very intermittent.
We need to collect a bit more data related to this problem. A good starting point might be to take down the following pieces of information:
FTP Server
IP address = 192.168.1.4
MAC address = 00:10:5a:d3:e4:e0
FTP Client1 (with no connectivity to FTP server)
IP address = 192.168.1.2
MAC address = 00:10:4f:b0:b2:fc
Here is one way of doing basic diagnostics.
Step 1
We want to make sure we have layer 1-3 connectivity between the Client1 and the server first (remember 'divide and conquer' method from the previous lesson?).
jr@mandala:~$ ping 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=0.372 ms
64 bytes from 192.168.1.4: icmp_seq=2 ttl=64 time=0.349 ms
64 bytes from 192.168.1.4: icmp_seq=3 ttl=64 time=0.371 ms
64 bytes from 192.168.1.4: icmp_seq=4 ttl=64 time=0.374 ms
It seems we have layer 3 connectivity working just fine.
Here's an interesting question for you. How do you know, you have received the replies from the FTP server in question (192.168.1.4)?
Now, you may have very confused expression on you face as in 'what do you mean?'. Have I not just gotten the reply from it?
Well, let me show you something to address this question.
Step 2
In order to be absolutely sure I got the reply from the FTP server (192.168.1.4) I need to verify my computer's ARP cache (Client1). Recall, that every time a host sends the packets, it encapsulates them in layer 2 frames (here Ethernet ones). In order to do that, the Client must have a valid destination MAC address (00:10:5a:d3:e4:e0) mapped to its IP address (192.168.1.4). If this mapping entry is not found in the APR cache, the Client1 will send ARP request to learn MAC address of 192.168.1.4. Knowing it let's check the ARP cache after we have sent ping packets. Here's what we find in (Pic. 1):
Well, in order to answer the second question, the reasons for this wrong mapping might be different. There could be the device with the duplicate address (same as the FTP server). Then, when the clients send ARP request for the MAC address of the server, this 'rouge' device (not the legitimate FTP server), also responds to the query. And if its answers arrives later than from the legitimate server, the override the legitimate MAC address mapping in the ARP cache. If some other clients get the reply for their ARP query from the 'rouge' device first and then from the FTP server, they create the mapping correctly. That would explain why some computers can still FTP to the server and others can't. Another reason for this wrong mac-to-ip mappings might be ARP poisoning attackin the network (eg. using Ettercap tool).
In this lesson I am going to create another issue to show you how your current skills can be practically useful in diagnosing network connectivity problems.
NOTICE
The steps presented in this lesson are not the ALL possible diagnostics you can do. And they do not have to be done in this specific order. I am merely listing some logical steps which might be useful in order to 'nail down' the root cause of the problem.
Trouble Ticket 2
Some client computers have problems accessing FTP server located in the same network 192.168.1.0/24. This behavior is very intermittent.
We need to collect a bit more data related to this problem. A good starting point might be to take down the following pieces of information:
FTP Server
IP address = 192.168.1.4
MAC address = 00:10:5a:d3:e4:e0
FTP Client1 (with no connectivity to FTP server)
IP address = 192.168.1.2
MAC address = 00:10:4f:b0:b2:fc
Here is one way of doing basic diagnostics.
Step 1
We want to make sure we have layer 1-3 connectivity between the Client1 and the server first (remember 'divide and conquer' method from the previous lesson?).
jr@mandala:~$ ping 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=0.372 ms
64 bytes from 192.168.1.4: icmp_seq=2 ttl=64 time=0.349 ms
64 bytes from 192.168.1.4: icmp_seq=3 ttl=64 time=0.371 ms
64 bytes from 192.168.1.4: icmp_seq=4 ttl=64 time=0.374 ms
It seems we have layer 3 connectivity working just fine.
Here's an interesting question for you. How do you know, you have received the replies from the FTP server in question (192.168.1.4)?
Now, you may have very confused expression on you face as in 'what do you mean?'. Have I not just gotten the reply from it?
Well, let me show you something to address this question.
Step 2
In order to be absolutely sure I got the reply from the FTP server (192.168.1.4) I need to verify my computer's ARP cache (Client1). Recall, that every time a host sends the packets, it encapsulates them in layer 2 frames (here Ethernet ones). In order to do that, the Client must have a valid destination MAC address (00:10:5a:d3:e4:e0) mapped to its IP address (192.168.1.4). If this mapping entry is not found in the APR cache, the Client1 will send ARP request to learn MAC address of 192.168.1.4. Knowing it let's check the ARP cache after we have sent ping packets. Here's what we find in (Pic. 1):
Pic. 1 - ARP Cache Entries
Pay attention to the highlighted entry. Is the MAC address mapped to our FTP server's IP address correct?
NO!
This MAC address does not belong to the server. FTP server's MAC address is: 00:10:5a:d3:e4:e0.
That test proved that getting echo replies to our echo packets (ping) must also be verified in the sender's ARP cache.
So, which device does this MAC address 00:10:0f:a3:3b:e6 belong to? And how on earth did it end up in our Client's ARP cache?
Well, in order to answer the second question, the reasons for this wrong mapping might be different. There could be the device with the duplicate address (same as the FTP server). Then, when the clients send ARP request for the MAC address of the server, this 'rouge' device (not the legitimate FTP server), also responds to the query. And if its answers arrives later than from the legitimate server, the override the legitimate MAC address mapping in the ARP cache. If some other clients get the reply for their ARP query from the 'rouge' device first and then from the FTP server, they create the mapping correctly. That would explain why some computers can still FTP to the server and others can't. Another reason for this wrong mac-to-ip mappings might be ARP poisoning attackin the network (eg. using Ettercap tool).
As for the answer to the first question, if this is not an ARP poisoning attack, you can use your knowledge from the lesson 9 about switching which helps you understand CAM table creation. Then, send uninterrupted ping towards the 'rouge' device and try to trace the mac addresses from switch to switch to find out where the device with duplicate IP address is located. MAC address table should help you find it relatively quickly.
There might be one more question I would like to clarify. Why the duplicate address was not detected by the 'rouge' device? Answer to that question may surprise you. The duplicate address detection uses so called 'gratuitous arp'. This is an unsolicited advertising of the MAC address upon computer startup. If some computers use the same MAC or IP, the newly computer cannot use its IP in the network. Unfortunately, some operating systems may not use this mechanism. They 'trust' what the operator does. If she or he wants this address, there will be no protest on their part.
I hope you are now beginning to see how much you already know!
In my next lesson you are going to analyze the third trouble ticket for layer 2 connectivity. This is going to be the last troubleshooting lesson. The upcoming lessons will describe other tools and interesting technologies such as VLANs, Spanning-Tree Protocol. Once we finish with foundations related to layer 2, we'll start discussing layer 3 technologies and concepts such as IP addressing, subnetting, routing etc.
Comments