Logging to Loggly with syslog-ng

syslog-ng is an open source implementation of the Syslog protocol for Unix and Unix-like systems. It extends the original syslogd model with content-based filtering, rich filtering capabilities, flexible configuration options and adds important features to syslog, like using TCP for transport. As of today syslog-ng is developed by Balabit IT Security Ltd.

We copied all that fluff from Wikipedia.

You can use syslog-ng to monitor log files on your servers and forward them to Loggly. A barebones configuration to forward local syslog events and monitor a file located at /path/to/your/file and send it to an input on Loggly listening to port 13337 looks something like this:

source s_all {  
  internal();  
  unix-stream("/dev/log");  
  file("/path/to/your/file" follow_freq(1) flags(no-parse)); 
};  
destination d_loggly {
  tcp("logs.loggly.com" port(13337));
};  
log { 
  source(s_all); destination(d_loggly); 
}; 

There are numerous versions of syslog-ng available for download and packaged for use on different distributions. If your distro already has syslog-ng packaged up on it, you should double check which version you are running:

kord@spartacus:~$ /opt/syslog-ng/sbin/syslog-ng -V
syslog-ng 3.0.5  <--- this is out of date

Loggly recommends running the latest version of syslog-ng if you will be monitoring a set of standalone log files. The current stable version of syslog-ng is 3.1.2. Older versions of syslog-ng may not have support for forwarding to custom TCP ports, or monitoring standalone log files.

While many distributions have syslog-ng available as a package, most default distros aren't using the current version of syslog-ng. It is likely you'll need to do a manual update of syslog-ng to the newest version.

These instructions assume you are running a Debian flavored distro, but should be easily adaptable to other OS builds. If you have alternate instructions for uninstalling or upgrading syslog-ng to a newer version on another distro, using a given package manager, please feel free to login and contribute to this page!

If you are running another syslog solution (including an older version of syslog-ng) which was installed by a package manager such as aptitude or yum, Loggly recommends uninstalling it before you begin installing the newer version of syslog-ng.

Here's an example of uninstalling syslog-ng with aptitude on a Debian box:

aptitude remove syslog-ng -y

All of the syslog-ng versions are available for download on Balabit's website: http://www.balabit.com/downloads/files?path=/syslog-ng/open-source-edition. The source code is also available for building by hand, but we won't cover that here.

If you have a fairly common distribution, you can browse to the binary build directory (for 3.1.2) and download your package: http://www.balabit.com/downloads/files?path=/syslog-ng/open-source-edition/3.1.2/setups. Pick the build best suited for your distro, and then download it (assuming a 64-bit Debian package):

wget "http://www.balabit.com/downloads/files?path=/syslog-ng/open-source-edition/3.1.2/setups/debian-etch-amd64/syslog-ng_3.1.2_amd64.deb"

Still assuming you have a Debian distro, you can do the following to install syslog-ng:

dpkg -i syslog-ng_3.1.2_amd64.deb

Note: Many Debian distributions require some type of syslog service be installed and running. If you run into dependency issues, try uninstalling whatever depends on syslogd first, then install syslog-ng via the binary package and reinstall those packages.

Assuming your installation succeeded, it should have started syslog-ng for you:

Setting up syslog-ng (3.1.2) ...
Restarting syslog-ng: Stopping syslog-ng: Starting syslog-ng:  * 

You can also start, stop and restart syslog-ng using the init.d script:

/etc/init.d/syslog-ng restart

The syslog-ng configuration file should be located in the /opt/syslog-ng/etc/ directory if you installed one of the binary packages:

/opt/syslog-ng/etc/syslog-ng.conf

Edit that file and make sure you have a source called s_all that looks something like this:

source s_all {
  internal();
  unix-stream("/dev/log");
  file("/proc/kmsg" program_override("kernel: "));
};

Forwarding data to another syslog server (like Loggly) requires setting a destination directive which tells syslog-ng where to forward the data it collects. You can reference the name, protocol and port destinations by going to the Input Management page in your Loggly account:

Now take the protocol and port, and put them in a destination entry (this example uses the tcp protocol):

destination d_loggly {
  tcp("logs.loggly.com" port(10997));
};  

Be sure to replace the 10997 above with the port number that is shown on your input page! After you've added that line, put in the log line that tell syslog-ng to forward Loggly the s_all source to the d_loggly destination:

log { 
  source(s_all); destination(d_loggly); 
};

All together, your config should look something like this:

source s_all {  
  internal();  
  unix-stream("/dev/log");  
  file("/proc/kmsg" program_override("kernel: "));  
};
destination d_loggly {
  tcp("logs.loggly.com" port(10997));
};  
log {
  source(s_all); destination(d_loggly); 
};  

Once you are done configuring syslog-ng, restart it:

/etc/init.d/syslog-ng restart

Make sure you've turned on discovery mode on the input you are using on Loggly. Discovery mode is enabled on an input by clicking on the slider switch next to the input on the Input Management page. Once you have discovery mode turned on for the input, send some data through syslog-ng to have it forwarded to your Loggly account:

logger "loggly is better than a bee in your aunt's bonnet"

Give the beavers that run the indexers on Loggly a few moments, then do a search in the shell for part of the string you just sent in:

kordless@geekceo> search aunt bee

Needless to say, there are going to be many times when you're going to want your logs encrypted during transport. This is where TLS comes in. The quick setup will ensure that your logs go to Loggly encrypted, but it will skip the step where Loggly validates *you* (which prevents man-in-the-middle attacks).

You'll need this certificate, which is also provided on your TLS enabled input page (loggly.com.crt) and the intermediate certificate from Starfield called sf_intermediate.crt.

You then need to concatenate both:

cat sf_intermediate.crt loggly.com.crt > loggly_full.crt

loggly_full.crt should go in the ca_dir directory below. Here's a sample configuration destination snippet (assumes the source s_all from above):

destination d_syslog_tls{
    tcp("logs.loggly.com" port(33267)
    tls(peer-verify(required-untrusted)
    ca_dir('/opt/syslog-ng/keys/ca.d/')
)); };
log { source(s_all); destination(d_syslog_tls); };

Of course, you'll need to restart syslog-ng to see your changes take effect - for TLS, you may want to start syslog-ng with the -d flag (for debug) so you can get an idea of what's happening - TLS can be a little tricky to get just right.

The configuration for peer checking is a little trickier than without it, but it's only a couple more lines, so don't worry. You'll will first need to do everything in the Quick TLS Setup section. One caveat is that you'll need a valid cert on your end, and _not_ a self-signed cert:

destination d_syslog_tls{
    tcp("logs.loggly.com" port(33267)
    tls(peer-verify(required-trusted)
    key_file('/opt/syslog-ng/keys/ca-key.pem')
    cert_file('/opt/syslog-ng/keys/ca.pem')
    ca_dir('/opt/syslog-ng/keys/ca.d/')
)); };
log { source(s_all); destination(d_syslog_tls); };

Again, you'll need to restart syslog-ng to see your changes take effect - for TLS, you may want to start syslog-ng with the -d flag (for debug) so you can get an idea of what's happening - TLS can be a little tricky to get just right.

Syslog servers handle syslog events from any services that support logging to the syslog handler on a given box. By default, some services do not log into the local syslog server. To get logs from services like Apache to Loggly, you'll need to tell syslog-ng to monitor the log files that the services generates.

Let's take a simple example where you need to monitor the access.log and error.log files in the /var/log/apache2/ directory on a Debian box. The instructions for monitoring those files with syslog-ng should be placed in the source directive:

file("/var/log/apache2/access.log" follow_freq(1) flags(no-parse));
file("/var/log/apache2/error.log" follow_freq(1) flags(no-parse));

If you apply that to the configuration above, you get something that looks like this:

source s_all {  
  internal();  
  unix-stream("/dev/log");  
  file("/proc/kmsg" program_override("kernel: "));  
  file("/var/log/apache2/access.log" follow_freq(1) flags(no-parse));
  file("/var/log/apache2/error.log" follow_freq(1) flags(no-parse));
};
destination d_loggly {
  tcp("logs.loggly.com" port(10997));
};  
log {
  source(s_all); destination(d_loggly); 
};  

Restart syslog-ng to have the changes take effect, and then hit Apache to generate some events. Jump over into the shell on Loggly and do a search for events with a 200 in them:

kordless@geekceo> search 200
- - [19/Oct/2010:17:22:20 -0700] "GET /rooster/ HTTP/1.1" 200 1516 "http://archives.geekceo.com/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3"

You can get more information about other syslog based solutions on the Logging Configuration page!