DNS server query graphic report

January 7th, 2015 |

Dear my Friend, here I will show you how to organize online monitoring system for DNS queries.

Most time in Telecom Domain business is necessary to measuring the DNS traffic and to check the performance of the service: how many queries has been successfully, how many faulty, what kind of queries the customers sending over different time. Also its necessary to check the type of DNS queries for security purpose to restrict free traffic using over dns-tunneling schemes. I will show how to optimize and turn DNS system to be using only for DNS service and send all required query logs to remote SYSLOG server into Corporate network behind Firewall. By default the DNS query log output is not configured in most systems as it required extra storage space, cause the log file can be growing very fast and it depend from DNS service capacity value – the amount of customer which is using it for Internet address resolving.

In my example of system configuration I will use common name for dns servers: DNS1 and DNS 2 are located in the city A, DNS3 and DNS 4 are located in the city B, DNS 5 and DNS6 are located in the city C. All 6 DNS servers are located in 3 geographical different sites with internet public IP address. All this servers running only dedicated specific build of DNS server for Solaris OS and this servers not allow to run nothing else on it to avoid unnecessary systems performance leakage. We have one management gateway server with SSH access to each of DNS server from Corporate network and this getaway server have connection to internal WEB server in DMZ network over SSH service too.

This is brief schema how the monitoring system is configured:

This is detailed schema of DNS query logging downstream process from Internet public DNS server into Web server in Corporate DMZ network:

DNS query loggin downstream process schema

Now I will give all details steps with full technical information:

1) The DNS server is installed from specific package which is developed for particular platform with extended system tweaks for performance improvement.

All required details you can find in this article “DNS servers for solaris 10 os i386/spac platform“

2) The configuration of DNS server include option for query logging into the SYSLOG daemon

logging{

channel query_log {

syslog local0;

severity info;

print-category yes;

print-time yes;

};

category queries {

query_log;

};

3) The Syslog daemon on each DNS server have next configuration:

local0.info @localhost

4) The SYSLOG daemon should NOT listening on UDP 514 port.

Do next steps:

# vi /etc/default/syslogd

LOG_FROM_REMOTE=NO

# svcadm restart system-log

5) Translation DNS query messages to remote SYSLOG host:

To translate all DNS query messages to remote SYSLOG host through virtual “localhost” using UDP port 514 is necessary to route UDP packages into TCP socket:

# /opt/SUNWsocat/bin/socat -T15 udp4-recvfrom:514,reuseaddr,fork,sndbuf=32768,rcvbuf=32768 tcp:localhost:9000

You can download SOCAT package for SPARC64 and SOCAT x86_64 Platforms.

You can check if socat process is using now UDP port 514:

# lsof -i udp:514
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
socat 2843 root 3u IPv4 0x301575068c0 0t0 UDP *:syslog

6) All encapsulated UDP packages inside TCP packages are going to be forwarded into localhost port 9000.

You can check if TCP/9000 port is open and listening on DNS server:

# netstat -naP tcp|grep “.9000″|grep LISTEN

*.9000               *.*                0      0 400000      0 LISTEN

You can check if this TCP/9000 port is in use by SSH daemon or not:

# lsof -i tcp:9000|grep LISTEN|grep IPv4

sshd    27249 proxy   11u  IPv4 0x30099260e00      0t0  TCP *:9000 (LISTEN)

As you can see the sshd process running with PID 27249. This is correct !

7) SSH Tunnels initiator.

This server is located behind the firewall – inside the corporate network and have only TCP/IP port 22 SSH service access to the remote DNS servers. We will use this system to create 6 bi-direction SSH links between remote DNS servers and internal SYSLOG host server. In our example the SYSLOG server hostname will be “sysloghost” and all DNS servers similar name: DNS1, DNS2, DNS3, DNS4, DNS5 and DNS6.

To create tunneling links – run below commands:

/usr/bin/ssh -q -R 9000:sysloghost:9000 -N -T -f DNS1 -c blowfish

/usr/bin/ssh -q -R 9000:sysloghost:9001 -N -T -f DNS2 -c blowfish

/usr/bin/ssh -q -R 9000:sysloghost:9002 -N -T -f DNS3 -c blowfish

/usr/bin/ssh -q -R 9000:sysloghost:9003 -N -T -f DNS4 -c blowfish

/usr/bin/ssh -q -R 9000:sysloghost:9004 -N -T -f DNS5 -c blowfish

/usr/bin/ssh -q -R 9000:sysloghost:9005 -N -T -f DNS6 -c blowfish

This commands will create TCP/IP 9000 listening port on each DNS servers and bind it back to specific port on SYSLOG host server:

DNS1 TCP/9000 -> SYSLOG TCP/9000

DNS2 TCP/9000 -> SYSLOG TCP/9001

DNS3 TCP/9000 -> SYSLOG TCP/9002

DNS4 TCP/9000 -> SYSLOG TCP/9003

DNS5 TCP/9000 -> SYSLOG TCP/9004

DNS6 TCP/9000 -> SYSLOG TCP/9005

8) SYSLOG TCP -> UDP package translator

On the remote SYSLOG host server for each DNS servers the redirected TCP socket port will be forwarded to the UDP port 514 – to the running syslogd daemon:

# socat tcp4-listen:9000,reuseaddr,fork,sndbuf=32768,rcvbuf=32768 udp:localhost:514

# socat tcp4-listen:9001,reuseaddr,fork,sndbuf=32768,rcvbuf=32768 udp:localhost:514

# socat tcp4-listen:9002,reuseaddr,fork,sndbuf=32768,rcvbuf=32768 udp:localhost:514

# socat tcp4-listen:9003,reuseaddr,fork,sndbuf=32768,rcvbuf=32768 udp:localhost:514

# socat tcp4-listen:9004,reuseaddr,fork,sndbuf=32768,rcvbuf=32768 udp:localhost:514

# socat tcp4-listen:9005,reuseaddr,fork,sndbuf=32768,rcvbuf=32768 udp:localhost:514

You can download SOCAT package for SPARC64 and x86_64 Platforms.

9) The configuration of SYSLOG daemon on host server

The syslod daemon on SYSLOG host server should store all incoming DNS query messages in separate files for each server. We will create dedicated storage pool for all DNS query files under /var/DNSREPORT/ path.

This is an example of config file how is should be looks:

local0.info /var/DNSREPORT/DNS1/log/named/bind-query-syslog.log

local1.info /var/DNSREPORT/DNS2/log/named/bind-query-syslog.log

local2.info /var/DNSREPORT/DNS3/log/named/bind-query-syslog.log

local3.info /var/DNSREPORT/DNS4/log/named/bind-query-syslog.log

local4.info /var/DNSREPORT/DNS5/log/named/bind-query-syslog.log

local5.info /var/DNSREPORT/DNS6/log/named/bind-query-syslog.log

10) SYSLOG to RRDtool data file translator

In our online monitoring system we are using RRDtool based data file to generate PNG graphic output with historic date statistics output. The required application is bindgraph.pl from “Marco Delaurenti” with some tweaks modification to make it able work properly in our particular configuration scheme. Each bindgraph daemon based process will run for each SYSLOG DNS query log file and generate bindgraph.rrd output data file. This is matrix based binary data file with details about required query fields for different date type: minutes, hours, days, month and years.

For each DNS query log file you need run separate bindgraph process:

/var/DNSREPORT/DNS1/bin/bindgraph.pl –logfile /var/DNSREPORT/DNS1/log/named/bind-query-syslog.log –host=localhost –daemon –format=bind92 –daemon_pid=/var/DNSREPORT/DNS1/log/bindgraph/bindgraph.pid –daemon_rrd=/var/DNSREPORT/DNS1/log/bindgraph –daemon_log=/var/DNSREPORT/DNS1/log/bindgraph/bindgraph.log

11) RRDtools data graphics report cgi-bin script.

For generate PNG graphic files from RRDtools data files we need execute separate PERL based cgi-bin script from Web Server. This script will generate time based graphic files and store it in web server over dynamic page.

The bindgraph.cgi CGI-BIN script should be located in cgi-bin folder for each DNS servers sub-website.

You can download the bindgraph package in any Debian or Ubuntu based Linux repository or from BindGraph.

12) WEB Server inside Corporate Network.

The DNS query performance monitoring tools is providing online graphic for all 6 DNS server.
This web server is located inside Corporate network behind the firewall for information security purpose.
You need VPN connection to see online statistics in time when you not in the office network.

To store all web pages, bindgraph binary scripts, DNS query logs, RRDtool data files and web server configuration file on one dedicated store place we will crease in Solaris ZFS pool with related datasets:

# zfs list -r zpool1/DNSREPORT
NAME MOUNTPOINT
zpool1/DNSREPORT none
zpool1/DNSREPORT/htdocs /var/DNSREPORT/web/htdocs
zpool1/DNSREPORT/httpd-etc /var/DNSREPORT/web/etc
zpool1/DNSREPORT/DNS3 none
zpool1/DNSREPORT/DNS3/bin /var/DNSREPORT/DNS3/bin
zpool1/DNSREPORT/DNS3/cgi-bin /var/DNSREPORT/DNS3/cgi-bin
zpool1/DNSREPORT/DNS3/log /var/DNSREPORT/DNS3/log
zpool1/DNSREPORT/DNS1 none
zpool1/DNSREPORT/DNS1/bin /var/DNSREPORT/DNS1/bin
zpool1/DNSREPORT/DNS1/cgi-bin /var/DNSREPORT/DNS1/cgi-bin
zpool1/DNSREPORT/DNS1/log /var/DNSREPORT/DNS1/log
zpool1/DNSREPORT/DNS5 none
zpool1/DNSREPORT/DNS5/bin /var/DNSREPORT/DNS5/bin
zpool1/DNSREPORT/DNS5/cgi-bin /var/DNSREPORT/DNS5/cgi-bin
zpool1/DNSREPORT/DNS5/log /var/DNSREPORT/DNS5/log
zpool1/DNSREPORT/DNS4 none
zpool1/DNSREPORT/DNS4/bin /var/DNSREPORT/DNS4/bin
zpool1/DNSREPORT/DNS4/cgi-bin /var/DNSREPORT/DNS4/cgi-bin
zpool1/DNSREPORT/DNS4/log /var/DNSREPORT/DNS4/log
zpool1/DNSREPORT/DNS2 none
zpool1/DNSREPORT/DNS2/bin /var/DNSREPORT/DNS2/bin
zpool1/DNSREPORT/DNS2/cgi-bin /var/DNSREPORT/DNS2/cgi-bin
zpool1/DNSREPORT/DNS2/log /var/DNSREPORT/DNS2/log
zpool1/DNSREPORT/DNS6 none
zpool1/DNSREPORT/DNS6/bin /var/DNSREPORT/DNS6/bin
zpool1/DNSREPORT/DNS6/cgi-bin /var/DNSREPORT/DNS6/cgi-bin
zpool1/DNSREPORT/DNS6/log /var/DNSREPORT/DNS6/log

The custom web server configuration file will be located in the directory “/var/DNSREPORT/web/etc/httpd.conf“:

ServerRoot “/usr/apache2″
PidFile /var/DNSREPORT/web/log/httpd.pid
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15

StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 0

NumServers 5
StartThreads 5
MinSpareThreads 5
MaxSpareThreads 10
MaxThreadsPerChild 20
MaxRequestsPerChild 0

Listen 4080
LoadModule access_module libexec/mod_access.so
LoadModule include_module libexec/mod_include.so
LoadModule log_config_module libexec/mod_log_config.so
LoadModule log_forensic_module libexec/mod_log_forensic.so
LoadModule env_module libexec/mod_env.so
LoadModule mime_magic_module libexec/mod_mime_magic.so
LoadModule cern_meta_module libexec/mod_cern_meta.so
LoadModule expires_module libexec/mod_expires.so
LoadModule headers_module libexec/mod_headers.so
LoadModule usertrack_module libexec/mod_usertrack.so
LoadModule unique_id_module libexec/mod_unique_id.so
LoadModule setenvif_module libexec/mod_setenvif.so
LoadModule version_module libexec/mod_version.so
LoadModule mime_module libexec/mod_mime.so
LoadModule autoindex_module libexec/mod_autoindex.so
LoadModule info_module libexec/mod_info.so
LoadModule cgi_module libexec/mod_cgi.so
LoadModule vhost_alias_module libexec/mod_vhost_alias.so
LoadModule dir_module libexec/mod_dir.so
LoadModule imap_module libexec/mod_imap.so
LoadModule userdir_module libexec/mod_userdir.so
LoadModule alias_module libexec/mod_alias.so
LoadModule rewrite_module libexec/mod_rewrite.so
User webservd
Group webservd
ServerAdmin eldar@aydayev.com
ServerName ojmgt02.mtn.com.ng
UseCanonicalName Off
DocumentRoot “/var/DNSREPORT/web/htdocs”

Options FollowSymLinks
AllowOverride None

Options Indexes FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all

User webservd
DirectoryIndex index.html index.html.var
AccessFileName .htaccess

Order allow,deny
Deny from all

TypesConfig /etc/apache2/mime.types
DefaultType text/plain

MIMEMagicFile /etc/apache2/magic

HostnameLookups Off
ErrorLog /var/DNSREPORT/web/log/error_log
LogLevel warn
LogFormat “%h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\”” combined
LogFormat “%h %l %u %t \”%r\” %>s %b” common
LogFormat “%{Referer}i -> %U” referer
LogFormat “%{User-agent}i” agent
CustomLog /var/DNSREPORT/web/log/access_log common
ServerTokens Full
ServerSignature On
ScriptAlias /cgi-bin/ “/var/apache2/cgi-bin/”

AllowOverride None
Options None
Order allow,deny
Allow from all

IndexOptions FancyIndexing VersionSort
AddIconByEncoding (CMP,/icons/compressed.gif) x-compress x-gzip
AddIconByType (TXT,/icons/text.gif) text/*
AddIconByType (IMG,/icons/image2.gif) image/*
AddIconByType (SND,/icons/sound2.gif) audio/*
AddIconByType (VID,/icons/movie.gif) video/*

ListenBacklog 8192
ServerLimit 2048
MaxClients 2048
#### DNS1 ####
ScriptAlias /DNS1/ “/var/DNSREPORT/DNS1/cgi-bin/”

AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
#### DNS2 ####
ScriptAlias /DNS2/ “/var/DNSREPORT/DNS2/cgi-bin/”

AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
#### DNS3 ####
ScriptAlias /DNS3/ “/var/DNSREPORT/DNS3/cgi-bin/”

AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
#### DNS4 ####
ScriptAlias /DNS4/ “/var/DNSREPORT/DNS4/cgi-bin/”

AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
#### DNS5 ####
ScriptAlias /DNS5/ “/var/DNSREPORT/DNS5/cgi-bin/”

AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
#### DNS6 ####
ScriptAlias /DNS6/ “/var/DNSREPORT/DNS6/cgi-bin/”

AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all

LockFile /var/DNSREPORT/web/log/accept.lock

Finlay to start DNS online web site you need run apache web server with below arguments:

# /usr/apache2/bin/httpd -f /var/DNSREPORT/web/etc/httpd.conf

This is example of graphic outpu – how it should be looking:

Thanks you very much to use your time to read this article.

All questions – please send to my personal Email

Or use contact form.

Posted in BIND DNS |

Tags: DNS monitoring, DNS query, DNS TPS, query per seconds, TPS, transaction per seond