Logentries Real Time Analytics

Published on May 2016 | Categories: Types, Presentations | Downloads: 92 | Comments: 0 | Views: 255
of 7
Download PDF   Embed   Report

Using Log Data Streams for Real-Time Analytics: 4 Common Use CasesWeb applications today are part of every IT operation within an organization.Independent software vendors (ISV) as well as enterprises create web applications to support theircustomers, employees and even suppliers. All in all, the goal of these applications is to deliver aservice, complete a transaction, support efficient and effective business operations, and directlyimpact company revenues and operational margins.

Comments

Content

1

Introduction
Web applications today are part of every IT operation within an organization.
Independent software vendors (ISV) as well as enterprises create web applications to support their
customers, employees and even suppliers. All in all, the goal of these applications is to deliver a
service, complete a transaction, support efficient and effective business operations, and directly
impact company revenues and operational margins.
In 2009, Paypal went offline for over an hour due to a network infrastructure failure. At the time,
Paypal was processing about $2,000 in payments every second, which meant its hour offline
resulted in about $7.2 million in lost transactions. According to an Aberdeen research study, this
type of downtime costs over $150K per hour, and the average recovery time is 5.13 hours. In
severe cases such as Paypal’s, IT teams, which include operations and development, need to know
what’s wrong—and fast. In these moments, a team’s ability to collect, analyze, and understand
data in real-time is fundamental to resolving the cause of the problem, taking action and validating
remediation. In this article, we define real-time analytics and demonstrate how log data from
different layers of your systems and application stack can enable real-time analytics and response.

2

What is Real-Time Analytics?
When referring to “analytics,” people often think of manipulating an existing set of
structured data to yield insights. “Real-time analytics” takes this definition a step
further by accounting for the constant appending of new data to the existing data set
and continuously re-analyzing the new dataset for new insights. But for analytics to be
real-time, data needs to be ingested immediately upon creation, delivering results in a
matter of seconds, enabling those interpreting the data to react right away.

To further demonstrate the definition of real-

ultimate source of real-time data, with systems

time analytics, let’s start by comparing it to the

often producing hundreds or thousands of

more commonly known, data batch processing.

log events per second. While a variety of tools

While batch processing can still append new

exist for capturing log data, a tool’s ability to

data to an existing set, it does so in batches

ingest and interpret log events in real-time as

rather than a continuous stream. Batch

they occur is a key differentiator, with many

processing comes with several disadvantages

tools taking anywhere from several seconds to

to real-time streaming. For example, if the data

several minutes to process a log event. A log

being processed doesn’t include timestamps,

management tool that isn’t truly real-time fails

every event in a batch will be assigned the same

to capitalize on the true power that log analytics

timestamp (the date and time the batch process

has to offer. How effective can a system alert

occurred). Batch processing also makes it

be if you’ll still experience several minutes of

impossible to generate immediate alerts off of

downtime before even receiving the alert?

events as they occur. In cases like Paypal’s, or

Tools that are actually real-time can deliver

any other web-based business, batch-process

information within seconds of occurring,

alerting is simply unacceptable.

alerting you to the warning signs leading up to
an issue, improving your chances of identifying,

In the world of operations and systems

diagnosing and resolving problems before they

administration, log data is often seen as the

negatively impact end-users.
3

Four Real-Time Use Cases
Below, we explore four use cases that exemplify why real-time analytics are critical
to performance and user experience, highlighting key capabilities that enable realtime analytics in each layer of your system or application:

1. The Application Layer

as they occur. Using a log analytics tool that

With your developer team preparing for

offers “anomaly alerts” can help you identify

a big push to production, you’re worried

early warning signs of larger issues.

about the possibility of unforeseen issues
deployment.

Regardless of the size of the release, whether

Testing in development will never provide

it’s a minimal viable product released to a

an exact replica of what will happen in

subsection of users or a large release following

production. Therefore, the more you are

a three month sprint, things inevitably go

able to view and monitor your logs in real-

wrong. Logs should be your go-to resource

time, the faster you will be able to address

for investigating and addressing the source

and rectify issues. While big issues may be

of issues that might arise while a real-time

easy to spot, real-time analytics can also

analytics tool should be used for alerting you

help you identify small issues building over

to anomalous activity as it occurs.

immediately

following

the

time that could eventually slow down your
application and user experience. While
batch-processed analytics could only ever
give you a historical analysis of your systems
data, real-time analytics can enable you to
identify anomalous patterns in your data



The easier it is to view and

monitor your logs in real-time the
faster to will be able to address and
resolve issues.

4

2. The Database Layer

database errors stream into the same single

Imagine over the course of several minutes,

view with the rest of your system’s log events

your popular e-commerce application hasn’t

as they occur. Alerts on database errors can

received any orders. Where’s the first place

be generated just as easily as alerts for the

you’d look for a possible issue? You may first

rest of your environment. And tools that offer

check to see if your website is still reachable

custom tagging of specific event types can

from a browser. Then, you may check your

also help you spot database specific errors as

server logs. Or perhaps you check your

they occur.

APM tool? Or a web analytics tool? Are they
all saying the same thing? Or nothing at all?



How much revenue have you lost

When you notice there aren’t any errors in

while guessing where to investigate the

your code and traffic to your website appears

problem?

to have remained steady, you decide to
investigate your database. Only then, after
wasting time investigating other scenarios,

3. The Server/Hosting Layer

do you see your database was improperly

Let’s say your mobile app was just featured

configured in the last deployment and has

on Product Hunt and you’re suddenly

reached its row limit. How many sales have

experiencing a spike in traffic. Luckily, your

you lost while guessing where to investigate?

app runs in an autoscaling environment and
handles the load without issue. When the

analytics,

traffic later subsides and your servers scale

database errors can go undiscovered, often

back, you decide to analyze the distribution of

only realized after a period of noticeable

400 errors over time. But how will you access

inactivity and investigating. When using a

data from the servers that scaled down?

real-time aggregated log analytics service,

If you weren’t sending those log files to a

Without

log-based,

real-time

5

central location in real-time, your data is

in real-time? Many tools only offer email

forever lost. In this scenario, centralizing your

notifications; but what if your team only

logs in real-time is crucial to capturing all

checks emails a few times a day? Alerts are

relevant data.

only as good as your team’s ability to react
to them quickly. Therefore, it’s important to

When dealing with auto-scaling environments,

use services that easily and automatically

real-time analytics also enables you to

integrate with the communication tools

monitor the scaling processes in the moment,

they’re already using, like Slack, HipChat or

ensuring they scale as expected. Without

PagerDuty.

real-time monitoring, you’d have no way of
knowing if a server cluster fails to scale up.

Cross-system, real-time analytics can also

Likewise, if your environment fails to scale

be demonstrated when integrating multiple

down when appropriate, you could be left

monitoring tools. Consider New Relic, which

paying for unused infrastructure.

offers real-time application performance



monitoring: what if one of your application

Real-time analytics enable you

to monitor the scaling process, in
the moment, to ensure everything is
scaling as expected.

metrics recorded in New Relic requires
deeper investigation? Manually switching
from an APM to a logging tool to then search
for related log events eats away at valuable
time. In these cases, plugins that integrate
APMs and log analytics tools for immediate

4. Cross System, Real-Time Analytics

log event correlation enables real-time action.

Generating real-time alerts are an obvious
necessity for reacting to issues as they occur.
But what if your team isn’t seeing the alerts

6

Final Notes
When it comes to leveraging analytics to ensure continuous service delivery and uptime, loglevel data and real-time processing are two requirements for success. At every layer of your
system’s hardware and application stack, real-time analytics enable centralized log collection
and monitoring, easy identification of key events, and instant alerts to the communication tools
used by your team. Thanks to the power of real-time analytics, you can improve your operational
efficiency and prevent downtime, leading to a healthier, stronger business.

About Logentries
Logentries is the leading real-time log management and analytics service built for the cloud,
making business insights from machine-generated log data easily accessible to development, IT
and business operations teams of all sizes. With the broadest platform support and an open
API, Logentries brings the value of log-level data to any system, to any team member, and to a
community of more than 35,000 worldwide users. While traditional log management and analytics
solutions require advanced technical skills to use and are costly to set-up, Logentries provides an
alternative designed for managing huge amounts of data, visualizing insights that matter, and
automating in-depth analytics and reporting across its global user community. To sign up for the
free Logentries service, visit logentries.com.

Cerspere
de platusd
anducim
Start your
free Logentries
trialagnistdfds
today
Get Started with your free logentries account, you will be up and running in minutes.
Free Trial includes access to ALL Logentries features.
*No credit card required!

7

Sponsor Documents

Or use your account on DocShare.tips

Hide

Forgot your password?

Or register your new account on DocShare.tips

Hide

Lost your password? Please enter your email address. You will receive a link to create a new password.

Back to log-in

Close