Archiving Logs in S3

Loggly stores your logs in a large scale search engine hosted on the Internet. The amount of time we store your logs in our search engine index is called the 'index retention time' which you can set from your pricing tab under your account. Once events in an account reach an age that is older than the max index retention time for your account, the events are removed from the index.

Loggly provides a way to archive logs older than your account's index retention time by writing them to your own S3 bucket. We'll create folders that are named after your Loggly inputs. Logs in your bucket are kept forever, or until you remove them, so you'll always have a copy handy if you need them.

You can create an Amazon S3 bucket, authorize us to write to it, and give us the bucket name, and then we'll write your logs into that bucket from then on.

To set up a bucket for writing, head on over to the Amazon S3 dashboard at https://console.aws.amazon.com/s3/home. If necessary, make yourself a new bucket by using the 'create bucket' button (stick to lower case names separated by dashes. Example: “new-s3-loggly-bucket” Please refrain from using dots and .com for example do not use “new-bucket.s3-loggly.com” this can occasionally throw DNS off, crippling the archive process). There'll be a logging option on the create button modal, but you can just ignore that as it doesn't have anything to do with Loggly.

Once you have the bucket created, select it and click the 'properties' tab. You'll get a nice little window at the bottom with a list of permissions for the bucket. Click the “Add More Permissions” button. For the grantee, enter 'aws@loggly.com'. Check the box for 'List', 'Upload/Delete' and 'View Permissions' and click 'Save' in the lower-right corner.

Back over on Loggly, go to your account page (yoursubdomain.loggly.com/account) and enter the name of your bucket in the form at the bottom. Click submit, we'll make sure we can write to your bucket and will start flinging logs into your S3 bucket as we get them.

There are three options for the format you would like your logs stored as:

• Raw
  • Stores your logs as we received them
• JSON
  • Your logs will be stored as json with the ipaddress, timestamp and inputname. The log events are stored as json escaped text in the event field.
• CSV
  • Your logs will be stored as csv with the ipaddress, timestamp and inputname. The log events are stored as csv escaped text in the event field.

Note: It may take upwards of an hour before you start seeing logs in your bucket.

There are several clients available for browsing your S3 buckets. If you are using OSX, check out S3Hub. At $2.99 on the App Store, it's a heck of a deal.

If there is a Windows client you like, please contribute to the wiki!