Intro:
I have faced this issue couple of times now on different NetScaler builds (10 & 11) but used Citrix approach to solve the issue detailed here ( http://support.citrix.com/article/CTX127622 ) . Now that I am stuck at a high profile security customer with no way to maneuver with networking/security requests other than what was originally requested , its time to look into this matter further.
Issue:
Customer has opened UDP 53 to his DNS server from both NSIP and SNIP IP`s never the less DNS name server added on NetScaler keeps on showing status ‘Effective State’ as DOWN and I am not able to resolve any hostname. DNS server is on a different subnet/VLAN from NSIP and SNIP.
Troubleshooting:
Firewall rules where triple checked and everything seemed to be in order. We also had LDAPS opened to the same server within the same ports security rule on the firewall and that service was working fine so the issue lies within the DNS service itself. Obviously all ports are closed except explicitly what is needed and in this case only port 53 UDP was opened for DNS , ping ICMP is closed .
While monitoring traffic on the firewall source SNIP and destination DNS server it seemed that no UDP 53 traffic was shown but rather ICMP ping which was dropped by firewall based on security rules in place. Why would SNIP look for ICMP traffic while the only thing I added is added the DNS server as a name server on NetScaler.
Voila , it seems that NetScaler uses ICMP Ping to determine if name server is up and running ! Even though UDP 53 is open , the name server still shows as DOWN because ICMP requests from NS are being dropped. Since name server is status down because of ICMP block , it wont try to resolve any hostname although again required DNS resolution port is open.
Resolution:
This is a well restricted environment so take into consideration that I cannot ask for additional IP addresses nor can I ask for additional security policies aka. open ICMP which is a No No !!!
1- We need to add a custom monitor to force name resolution traffic to use DNS UDP 53. Go to Traffic Management – Load Balancing – Monitors.
2- Add Monitor , Name it , choose type: DNS .
3- Press on Special parameters and in Query button insert your domain FQDN. You can enter any resolvable host but I like to use the domain name itself so that Only scenario this would fail is if the actual DNS server itself is down. Press Create.
4- Go to Services and press ADD.
5- Name your service, type in your DNS server IP, protocol: DNS, Port:53 and press OK.
6- In the Monitors tab , open the Service to Load Balancing Monitor Binding.
7- Notice that by default the monitor is using ping-default thus this would be in DOWN/FAIL state because ICMP Ping traffic is blocked. Our aim eventually is for this service and later associated virtual server to use customer monitor we created earlier to force monitor to use port 53. Press Add Binding.
8- Choose the custom-dns monitor we created earlier. This will override this default monitor.
9- Now the monitor should change to status UP and Success DNS probes. Press close and refresh, notice status of service.
10- Now we need to create a virtual server that hosts this service. Since we cannot use any additional IPs and Only NetScaler is going to use this virtual server for DNS resolution, we are going to use a non-routable virtual server. Go to Traffic Management – Load Balancing – Virtual Servers – Add .
11- Lets add the service we just created to the new non addressable virtual server.
12- Press Done and check virtual server status.
13- Time to add the virtual server we just created as a name server. Go to Traffic Management – DNS – Name Servers – Add. Choose DNS Virtual Server tab , choose virtual server we just created earlier and UDP protocol.
14- Status of Name Server finally UP.
15- Test if domain diyar.local is resolvable. Ping should not work since ICMP is disabled but this is a simulation in my lab so don’t mind as long as the domain diyar.local is being resolved into an IP , its all working fine.
Conclusion:
Citrix has to work on changing how name servers status is determined to be UP or Down. We have to realize its a bit of a challenge because DNS has to probe some kind of FQDN to determine the status of the name server and Citrix cannot just use any internet hostname or determine your internal hostname schema. That been said Citrix should document how probing is done to begin with and provide a decent solution rather than use MIP which is to some extend depreciated. Anyway hope this helps someone out there. Would love to hear your comments !
The is quite complicated one. I face the same issue where DNS server effective state failed. My dns server is on different subnet. so i have created one SNIP from where the Management IP is defined. Post that it started working. In my scenerio Ping is allowed.
True if you have the luxury to do so that is fine but when you work with very strict security requirements that would not be possible.
Thank you Saadallah 🙂 , it was very helpful.
Thanks Osama.
Maan, thak you very much!
ludicrous, you could do this on any other lb in 2 mins by just assigned a health check to the service. netscaler is so backwards its unreal