I am trying to set up a new failover DHCP server using the Linux isc-dhcp-server package. I had a working failover server that needed to be replaced because it's old. I copied the dhcpd.conf file from the old server to the new one so I'm fairly sure it is correct. But if I shut down the primary server, it does not fail over. I suspect a config problem at the switch -- which I do not own or control. So what I want to do is to sniff packets to see if the DHCPREQUEST packets are even getting to the failover server. Anybody know how I can do that on Linux? My preferred packet sniffer is ngrep but I'll use something else if someone can give me the command syntax. I probably should mention that I need to do this remotely. I am working from home because of the pandemic, of course. -- John G. Heim; jheim@math.wisc.edu; 608-263-4189
On 12/29/20 10:27 AM, John G Heim wrote:
My preferred packet sniffer is ngrep but I'll use something else if someone can give me the command syntax.
I can't give you the syntax, but tshark has filtering options that should let you capture only the DHCP packets. https://wiki.wireshark.org/DHCP I've used it successfully for other purposes.
John, If you can, swap your server address with your old server address. Here is what I suspect is happening: In a normal dhcp request, the packet is a broadcast. All systems see the request. In most advanced (enterprise, university, corporate setups), the switches isolate broadcast traffic and in fact you might not even be serving dhcp for the same subnet you are on. So, enter the "dhcp helper" as Cisco calls it. A router sees the broadcast traffic and forwards it to where ever the admin has told it to go even if the dhcp server is not on the same broadcast domain. I say that only to let you know that this might not work. Here is what I use all the time: tcpdump -i enp1s0 port 67 or port 68 -e -n -vv change the enp1s0 to what ever your ethernet interface is. This will show both sides of a dhcp conversation. The reason I suggested you switch your addresses is that if your switch/router admins have told your dhcp where to go, it will work after you switch the addresses. Hopefully one or the other works for you. -- Kelly Prescott On Tue, 29 Dec 2020, John G Heim wrote:
I am trying to set up a new failover DHCP server using the Linux isc-dhcp-server package. I had a working failover server that needed to be replaced because it's old. I copied the dhcpd.conf file from the old server to the new one so I'm fairly sure it is correct. But if I shut down the primary server, it does not fail over. I suspect a config problem at the switch -- which I do not own or control. So what I want to do is to sniff packets to see if the DHCPREQUEST packets are even getting to the failover server. Anybody know how I can do that on Linux?
My preferred packet sniffer is ngrep but I'll use something else if someone can give me the command syntax.
I probably should mention that I need to do this remotely. I am working from home because of the pandemic, of course.
-- John G. Heim; jheim@math.wisc.edu; 608-263-4189 _______________________________________________ Blind-sysadmins mailing list -- blind-sysadmins@lists.hodgsonfamily.org To unsubscribe send an email to blind-sysadmins-leave@lists.hodgsonfamily.org
Hi, Yep, agree with Kelly on this one, it sounds like a new server is set up but the switch doesn't know about it so will still try to forward DHCP requests to the new server. In this case a packet dump won't show any traffic coming to the new server at all. Andrew. -----Original Message----- From: Kelly Prescott <kprescott@coolip.net> Sent: 29 December 2020 23:21 To: jheim@math.wisc.edu; Mailing list for blind system administrators <blind-sysadmins@lists.hodgsonfamily.org> Subject: [Blind-sysadmins] Re: sniffing DHCP packets John, If you can, swap your server address with your old server address. Here is what I suspect is happening: In a normal dhcp request, the packet is a broadcast. All systems see the request. In most advanced (enterprise, university, corporate setups), the switches isolate broadcast traffic and in fact you might not even be serving dhcp for the same subnet you are on. So, enter the "dhcp helper" as Cisco calls it. A router sees the broadcast traffic and forwards it to where ever the admin has told it to go even if the dhcp server is not on the same broadcast domain. I say that only to let you know that this might not work. Here is what I use all the time: tcpdump -i enp1s0 port 67 or port 68 -e -n -vv change the enp1s0 to what ever your ethernet interface is. This will show both sides of a dhcp conversation. The reason I suggested you switch your addresses is that if your switch/router admins have told your dhcp where to go, it will work after you switch the addresses. Hopefully one or the other works for you. -- Kelly Prescott On Tue, 29 Dec 2020, John G Heim wrote:
I am trying to set up a new failover DHCP server using the Linux isc-dhcp-server package. I had a working failover server that needed to be replaced because it's old. I copied the dhcpd.conf file from the old server to the new one so I'm fairly sure it is correct. But if I shut down the primary server, it does not fail over. I suspect a config problem at the switch -- which I do not own or control. So what I want to do is to sniff packets to see if the DHCPREQUEST packets are even getting to the failover server. Anybody know how I can do that on Linux?
My preferred packet sniffer is ngrep but I'll use something else if someone can give me the command syntax.
I probably should mention that I need to do this remotely. I am working from home because of the pandemic, of course.
-- John G. Heim; jheim@math.wisc.edu; 608-263-4189 _______________________________________________ Blind-sysadmins mailing list -- blind-sysadmins@lists.hodgsonfamily.org To unsubscribe send an email to blind-sysadmins-leave@lists.hodgsonfamily.org
_______________________________________________ Blind-sysadmins mailing list -- blind-sysadmins@lists.hodgsonfamily.org To unsubscribe send an email to blind-sysadmins-leave@lists.hodgsonfamily.org
This theory that it's the switch is still #1, more on that later. But I had already put the new server at the IP address of the old server before I posted my original question. I can't absolutely rule out the possibility that failover was broken before this upgrade. But I don't think so because I was monitoring it with nagios. There's a nagios plugin called check_dhcp that you can run from the command line. It's a really nice way to test your dhcp server. See this page for info on using the plugin. https://serverfault.com/questions/171744/command-line-program-to-test-dhcp-s... I hadn't actually tested failover for years. But when I took down the old failover server, I got an alert from nagios. That is to be expected. But I never got a recovery message when I brought up the new server. So I ran the plugin at the command line and it is acting as if the failover server is down. So then I took Kelley's advice and used tshark to sniff packets. This is the first time I used tshark since my network security class in college. I'd forgotten how nice it is. Way nicer than ngrep. Anyway, tshark showed me that the failover server is receiving packets and responding to them. It takes along time to listen to all the output and there are definitely some differences between what the primary and the secondary servers are sending out. I will have to investigate that. But as I said, the DHCP discover and request packets are getting to the secondary/failover server and it is responding as expected. I am thinking that packets are being filtered on the switch based on MAC address. The switch is probably configured to allow discover and request packets through from any MAC address but block acknowledgement and response packets except from known MAC addresses. Well, that is my theory at least. When I started working here at the Math department at the University Of Wisconsin, I had control over everything, even the network switches. But the campus has taken over the networking so I can't check the firewall rules on the network switch. PS: I don't really mind not having control over the network itself. Now its not my problem when somebody plugs in a laptop that is part of a botnet which happens like once a week. On 12/30/20 6:51 AM, Andrew Hodgson wrote:
Hi,
Yep, agree with Kelly on this one, it sounds like a new server is set up but the switch doesn't know about it so will still try to forward DHCP requests to the new server.
In this case a packet dump won't show any traffic coming to the new server at all.
Andrew.
-----Original Message----- From: Kelly Prescott <kprescott@coolip.net> Sent: 29 December 2020 23:21 To: jheim@math.wisc.edu; Mailing list for blind system administrators <blind-sysadmins@lists.hodgsonfamily.org> Subject: [Blind-sysadmins] Re: sniffing DHCP packets
John, If you can, swap your server address with your old server address. Here is what I suspect is happening: In a normal dhcp request, the packet is a broadcast. All systems see the request. In most advanced (enterprise, university, corporate setups), the switches isolate broadcast traffic and in fact you might not even be serving dhcp for the same subnet you are on. So, enter the "dhcp helper" as Cisco calls it. A router sees the broadcast traffic and forwards it to where ever the admin has told it to go even if the dhcp server is not on the same broadcast domain. I say that only to let you know that this might not work. Here is what I use all the time: tcpdump -i enp1s0 port 67 or port 68 -e -n -vv change the enp1s0 to what ever your ethernet interface is. This will show both sides of a dhcp conversation. The reason I suggested you switch your addresses is that if your switch/router admins have told your dhcp where to go, it will work after you switch the addresses. Hopefully one or the other works for you.
-- Kelly Prescott
On Tue, 29 Dec 2020, John G Heim wrote:
I am trying to set up a new failover DHCP server using the Linux isc-dhcp-server package. I had a working failover server that needed to be replaced because it's old. I copied the dhcpd.conf file from the old server to the new one so I'm fairly sure it is correct. But if I shut down the primary server, it does not fail over. I suspect a config problem at the switch -- which I do not own or control. So what I want to do is to sniff packets to see if the DHCPREQUEST packets are even getting to the failover server. Anybody know how I can do that on Linux?
My preferred packet sniffer is ngrep but I'll use something else if someone can give me the command syntax.
I probably should mention that I need to do this remotely. I am working from home because of the pandemic, of course.
-- John G. Heim; jheim@math.wisc.edu; 608-263-4189 _______________________________________________ Blind-sysadmins mailing list -- blind-sysadmins@lists.hodgsonfamily.org To unsubscribe send an email to blind-sysadmins-leave@lists.hodgsonfamily.org
_______________________________________________ Blind-sysadmins mailing list -- blind-sysadmins@lists.hodgsonfamily.org To unsubscribe send an email to blind-sysadmins-leave@lists.hodgsonfamily.org _______________________________________________ Blind-sysadmins mailing list -- blind-sysadmins@lists.hodgsonfamily.org To unsubscribe send an email to blind-sysadmins-leave@lists.hodgsonfamily.org
participants (4)
-
Andrew Hodgson -
Jason White -
John G Heim -
Kelly Prescott