13 Dec 2012

We are not SIEM

0 comments Permalink Thursday, December 13, 2012
Packetloop is not a Security Information and Event Management (SIEM) system. It's a (parser free) Big Data Security Analytics system and I just wanted to give you 10 ways they differ.
  1. Punishing us with Pie Charts - SIEM's have been punishing us with pie charts and meaningless alerts. Instead of quality analytics they generally throw up a pie chart (High, Medium, Low - Red, Orange, Green) and a spreadsheet (row and column) view of the world. They don't allow you to explore and explain your attack data. Packetloop specializes in security analytics providing multiple ways to quickly visualize, interact, explore and explain security events.
  2. Low fidelity input - SIEM's operate on and are required to parse device log data. They deal with a digest of the original incident as seen by a device, and as configured by the device administrator. Like a cryptographic hash you can't get the original data back after a device has interpreted and produced a log. Packetloop operates on Full Packet Captures and is able to analyse all information in the stream. This provides full fidelity data that can be analysed, stored, searched and filtered, repeatedly and from different perspectives if required. Operating on full packet captures gives your organisation the greatest percentage of options when it comes to detecting and managing a breach.
  3. You can't find incidents you don't log - SIEM is blink and you miss it technology. If you aren’t logging the information you don’t see it. If you don’t have the a signature set to alert to the incident it can’t possibly reach the SIEM. You are faced with a problem of silent evidence. You are never hearing about the attacks that are being perpetrated against your network. Packetloop doesn't rely on parsers and includes multiple sensors to produce indicators and warnings. Packetloop continually scans all network data for Zero Day attacks. Finding attacks in past data is exceptionally valuable for establishing a time line related to a breach or targeted attackers.
  4. Aggregation for scale removes precision - SIEM's aggregate data over time. Security events are grouped by hours then grouped by days, then weeks, then years. The details of the original incident are lost over time and are difficult to analyse. Packetloop stores terabytes of security event data and allows you to zoom from years to seconds with no loss of the original information. Any arbitrary time window can be analysed allowing you to Play, Pause and Rewind network data.
  5. Parsing, Parsing, Parsing - SIEM's are only as effective as the logs they are forwarded and can parse. The configuration of the device forwarding the log determines how much information the SIEM receives. If the log is not verbose enough, or sends information that cannot be parsed the quality of the information is degraded. Packetloop operates on network packet captures taken from live network taps. It requires no parsers and has access to all security and protocol information through deep packet inspection. No configuration needs to be applied to devices, no parsers and no complex integration work.
  6. No ability to share metrics - SIEM's are not built for multi-tenancy and don't provide any ways to share key metrics in the terms of frequency, severity, attack life cycles and trends related to attacks. Within Packetloop almost every metric presented to you has the global version (all customers) presented it beside it. So you can quickly determine whether you are overly targeted or accurately targeted compared to other customers on the platform.
  7. More and more IT instead of Analysts -  SIEM's require experts to deploy, configure and maintain. This generally equates to a number of full time employees to manage the parsers, the flow of information, upgrades, downtime etc. Packetloop operates in the Cloud (and soon on premise) as a turn-key service. Why have employees to maintain the infrastructure when you could do with some  really smart analysts to explore and report on your data? Simply capture your network traffic and upload using the Web, SFTP, S3 Bucket Copy or Send us a Disk. You can encrypt all data and provide a pass phrase prior to us processing the data.
  8. Made for the Cloud - Packetloop was made for the Cloud and for the express purpose of providing greater security to cloud-based assets. Normally customers sacrifice security controls when they deploy systems in the Cloud however through leveraging Packetloop a customer can have greater analysis, detection and incident response capabilities than they currently have on-site using their SIEM. 
  9. Bang for Buck - Packetloop is cost effective and will always provide overwhelming value for money. It's a stand against the expensive solutions we see rolled into customers every day. Solutions that cost $100,000's but provide very little benefit. We only charge you for the data you process and we will launch with a price that makes it an easy decision to upload.
  10. The future is bright - SIEM is a waning technology and due to the low feature space (quality of the input records) it's difficult for it to innovate. Sure there will be SIEM 2.0 which uses Big Data to scale but new insights and analytics will be difficult to conjure. Packetloop can extract information from any part of the network communication giving it a extremely dense feature space. The amount of interesting information is so vast that we are finding new ways to apply statistical analysis and machine learning every day.

6 Dec 2012

What is Big Data Security Analytics? Part One: Visualization

0 comments Permalink Thursday, December 06, 2012

Recently we were showcased in GigaOM's article "6 ways big data is helping reinvent enterprise security". The area of Packetloop they focused on was visualization and the nexus of Big Data, NoSQL and it's ability to power visualisations. This nexus is incredible and I believe will drive a lot of really awesome work. The idea of navigating large data sets effortlessly allows an analyst to explore and explain the data. However I see this as only the first in a set of powerful features that Big Data Security Analytics will deliver.

So what is Big Data Security Analytics?

It's delivering knowledge and intelligence in relation to security events with the highest fidelity and context possible. Knowledge and intelligence drives informed decisions based on real evidence that increases the security effectiveness of an organisation. I also think it's important to view the subject after you remove the benefits Big Data Security Analytics gives you in terms of size and speed. I was challenged to do this recently by Scott Crawford a leading industry analyst and although at first disarming it forced me to explain our work in terms of the subjects in the figure below.

So I was keen to discuss these subjects in a series of blog posts that showcased the real effectiveness of Big Data Security Analytics and what problems Packetloop can solve forgetting about size and speed.


For us Visualization is all about encoding complex and densely featured information so that the best pattern matching system in the world can interpret it. The best pattern matching system in this case is you! We all have difficulty interpreting novelty, outliers, anomalies and trends when faced with a spreadsheet but as soon as we see it visually it's easy. A classic example of this is Anscombe's Quartet. Four data sets that have almost identical statistical features (sum, mean, variance etc.) and also exhibit very strong correlation. However as soon as you visualize the same information you can immediately see the differences in the datasets and the trends within each dataset.

So how does this concept translate to security? In Packetloop's Threats module we have a stacked bar visualization. A great dataset to demonstrate and analyse is DARPA98. When you take away the size of the data set (3 months of time, 171K attacks and 64M packets) it's a basic Low, Medium and High severity visualization. Low attacks are shown in light blue, High/Critical attacks are shown in dark blue and Medium attacks in between those two shades.

Even in this simple visualization I can enable a single feature to help me find outliers. In the first example I am going to track "New Attacks" and see how that changes my analysis. New attacks are attacks that have never been seen before. 

After hiding the Legend drawer you can see that Frequency of Attacks is plotted against the left hand Y-Axis (y1) and the number of  New Attacks is plotted against the right hand Y-Axis (y2). Sharp spikes show periods of time where novel attacks were used against the organisation. They are areas of time I would want to investigate in greater detail. In the figure below you can see a sharp spike in the "New Attacks" line on June 15th. What immediately piques my interest is the fact that there's a number of new attacks in that period but the stacked bar is all Low and Medium attacks. This is a classic example of Severity not always being a good indicator of attacker behaviour and tactics but the "New Attacks" line definitely shows me that either an individual or group of attackers is trying a number of vectors that I haven't seen before in a relatively small time period.

So instead of viewing the information based on Severity (Low, Medium and High) let's pivot the data and view the same information through the Attack lens. In the visualization below each attack is allocated a colour and the height of the bar shows the frequency or number of times the attack was used.  If we focus on the peak of New Attacks on Tuesday the 16th the stacked bar visualization hides the nature and complexity of that period however the New Attacks line clearly shows there is interesting information hidden in there. Why is it hidden? In the first example overlaying novel attacks exposed the fact that severity is not always a good indicator. When we switched to viewing by attacks we still see a large jump in new attacks but they are dominated by the high number of other attacks that aren't new. Viewing the data as a 3 month period where each bar is 1 day is hiding the detail.

Packetloop has the ability to zoom in from years of data to one minute of data instantly. Zooming in from the 3 month view to the 24 hour view the detail is uncovered. In a single hour there a flurry of activity in relation to attacks that the organisation had never seen before.

Zooming in further to look at just 30 minutes from the original 3 months and each minute is laid out clearly. A mouseover allows you to see the Severity, Attack Type and Frequency.

Below our main visualization we have a series of data panels that allow you to see more detail than an annotation can provide. In this case I see the new attack "Finger/execution attempt" has all originated from a single Source IP address ( This is a relatively innocuous attack and is obviously being used for information gathering prior to an exploit being used.

So what other attacks has this single Source IP Address been using, what is this attacker's timeline - throughout the current 30 minutes and then the entire 3 months. The Advanced Filter allows the IP to be selected that filters all attack data based on the Source being

Once filtered you can see that there's a dozen attacks with numerous different attack vectors against a single destination ( within that 30 minutes. All could be classed as methods of gaining specific information on the target. So obviously a lot of work is being done to enumerate the host but as yet there is no smoking gun.

Zooming out you can see the entire attack timeline for this attacker - 21 attacks using 10 different attack vectors over a period of 3 months.

Looking back at the filter I can see that the attacker has hit two destinations and by selecting one then the other I get his attack timeline for each. What's interesting about looking at the entire timeline is I now start to see indicators and warnings related to FTP and warez as well as actual exploit delivery in the form of x86 Shellcode.

If we filter by the destination we can see the attack timeline between the attacker and the specific victim. There is a lot of information gathering on the June 15th and then a break of a week and then a number of warnings related to FTP and warez activity on the 22nd June and 23rd of June.

If we filter by the destination the timeline is totally empty except for delivery of Shellcode over DNS 3 times within 7 minute period on July 23rd.

Zooming back into the exact time period you can see when the attacks were delivered. Two attacks at 12:33am and another occurrence of the same attack at 12:39am.

I will post a video of this soon but to give you a rough idea this analysis took less than a minute or two to perform against a dataset of 64 million packets that spanned 3 months of time.


Big Data and NoSQL enable the ability to process, store and query security event information at incredible size and scale. However Big Data Security Analytics is the intelligence that can be gleaned from this data. This blog demonstrated the strengths of visualization and specific overlays (New Attacks) that allow you to explore and explain the data. Changing the lens to view the same attack data from different perspectives and the ability to zoom from years to minutes and back again makes Big Data Security Analytics an extremely powerful analysis and intelligence tool.

13 Nov 2012

We won the AWS Ninja Award for Innovative Excellence!

0 comments Permalink Tuesday, November 13, 2012
We created Packetloop to provide answers to questions that our customers had. Questions that existing security technologies were not able to answer. Having customers who want to use our software, who see the value in using Packetloop to get clear network intelligence from their data is a reward in itself.

To be recognised by the industry and your peers is an added bonus. Tonight Packetloop received its first award, from Amazon Web Services. The award we received is the "AWS Ninja Award for Innovative Excellence", and it recognises an organisation that uses AWS cloud in a new or innovative ways, pushing the envelope on how cloud computing can deliver tangible business benefits. Packetloop certainly fits the bill, as it utilises the utility power of the Cloud to provide Big Data Security Analytics at a fraction of the price that other on premise solutions cost. We also recently modified our opens source project PacketPig so that it can use AWS Elastic MapReduce. The full blog on how to configure PacketPig for EMR can be found here.

We would like to thank AWS not only for the recognition that this award provides for us, but for giving us a killer platform on which to develop and deliver Packetloop. We also want to congratulate AWS on the unveiling of their new Australian, Sydney based instance which was announced today. This is a great step forward for developers of Cloud hosted services in Australia and we look forward to a long association with AWS.
30 Oct 2012

Packetpig on Amazon Elastic Map Reduce

2 comments Permalink Tuesday, October 30, 2012

Packetpig can now be used with Amazon's Elastic Map/Reduce (EMR) for Big Data Security Analytics.

We've added some sugar around the EMR API to help you start running packet captures through our Packetpig User Defined Functions (UDFs) as easily as possible.

Let's start with a very basic example, pumping a set of captures through the supplied pig/examples/binning.pig.

'binning.pig' uses the PacketLoader UDF to extract IP and TCP/UDP attributes from each packet in each capture. If you look in the script, you'll see the format returned in the LOAD statement.
We want to extract all of these and store them in a CSV file for later analysis.

First let's setup our credentials. Set these env variables in your terminal.

export AWS_ACCESS_KEY_ID=[your key]
export AWS_SECRET_ACCESS_KEY=[your key]
export EMR_KEYPAIR=[name of key you create in ec2 console]
export EMR_KEYPAIR_PATH=[path to saved key you just created]
export EC2_REGION=us-west-1 (optional, defaults to us-east-1)

Now, run the job:

$ lib/run_emr -o s3://your-bucket/output/  \
-l s3://your-bucket/logs/ \
-f s3://packetpig/pig/examples/binning.pig \
-r s3://your-bucket/captures/ \
Created job flow j-33QXAKHCEOXUO

Type lib/run_emr --help for more information but for now, we specify the output dir with -o, the log dir with -l, the pig file with -f and the read dir with -r*.
-w specifies we like to watch.

After a while, you'll see the bootstrap process begin, some packages will be installed, and then Hadoop will start.

At this stage, an EC2 node has been spawned to run the Hadoop master and it's also where the mappers and reducers will run in this example.

It's boring to watch logs, it'd be nicer if we could see more.
$ lib/run_emr -e
j-33QXAKHCEOXUO RUNNING david's pig jobflow
        Setup Pig      COMPLETED                   22s
        binning.pig    RUNNING                   3485s

$ lib/run_emr -x j-33QXAKHCEOXUO
Connect to http://localhost:9100/jobtracker.jsp - hit ctrl-c to stop the ssh forwarder

Do as it says and hit localhost:9100 in your browser and you can look at the Hadoop job tracker which is useful to get a measure of how well you've tweaked your node type and node count.

In my case, I'm looking at 22.64% mappers completed after 1h 14m. That's a bit slow!
The default is to run 1 m1.large instance == 4 cores.

$ lib/run_emr -o s3://your-packetpig-output/ \
-l s3://your-packetpig-logs/ \
-f s3://packetpig/pig/examples/binning.pig \
-r s3://yourbucket/captures/ \
-w -k 20 -t m1.xlarge
Created job flow j-38QAABHC3RXO7

Now we're looking at 20 m1.xlarge nodes == 80 cores.

If you change your mind about the job you can easily terminate it like so:

$ lib/run_emr -d j-38QAABHC3RXO7

All the included Packetpig scripts in pig/examples are mirrored in s3://packetpig/pig/examples.*
If you want to run your own, just change the -f argument to point to whereever your script is.

Here's a video showing how you can use Packetpig and EMR to find Zero Days in past traffic.

28 Oct 2012

News and Updates

3 comments Permalink Sunday, October 28, 2012

Its been very busy couple of weeks at Packetloop leading up to our presentation last weekend at Ruxcon. A lot of work has happened to finalise the commercial release of Packetloop, as we continue to work with our Early Access users, understanding and incorporating their feedback into the platform. The feedback has been great, and its exciting to watch new users explore and understand the user interface for the first time, and even more gratifying to see them understand and exploit the power of Packetloop.

We were lucky enough to be able to take the entire team to Ruxcon to support Michael Baker (our CTO) with his presentation Finding Needles in Haystacks the Size of Countries, and I would estimate there was some 300 people in the room to watch his presentation. I was interested to see the expressions on the faces of some of the countries best security professionals when Michael showed how we can easily process vast amounts of network packet captures (Big Data!), and use our tools to identify previously undetected Zero Day attacks.

Best of all was the stunned silence that came over the room when he showed a couple of visualisations that captured the power of the Packetloop/PacketPig tools, showing security data in a way that people had previously not considered. The entire presentation can be found here on Slideshare, but if you just want to see the visualisations in action, check them out here. Many thanks to those who took the time to come to the presentation, and to also come and seek us out later for further conversations.

Michael is also currently collaborating on a series of blog posts with the team at HortonWorks (whose founders authored Apache Pig). The topic of this series is the use of Pig to perform Big Data Security Analytics. Much of our work in this space has been using our open source platform PacketPig. The first of this series, co written with Russell Jurney (@rjurney), titled Big Data Security Part One: Introducing PacketPig is a great read.

Given our participation in Ruxcon, we took the opportunity to sponsor the Risky.Biz Podcast, who covered both the BreakPoint and Ruxcon conferences last week. Risky.Biz host Patrick Gray did an interview with Michael about Packetloop for the show. The entire interview can be heard here. During the interview and some subsequent conversations, Patrick posed a number of questions around the prospect of businesses uploading their internal data into the cloud for security processing. This is an interesting question, and is worthy of further discussion.

In essence we are advocating a new data source, one that may be seen as higher risk due to its external, 3rd party nature. Obviously we have to offer a value proposition that outweighs this perceived risk. We believe that value proposition is the power of Big Data Security Analytics and the knowledge and intelligence it provides. We should however put some perspective around this risk by keeping in mind that companies already have data in the cloud, such as email, CRM, device logs, or they use online applications for everything from project/document management to financial applications. These all store the resultant data in the cloud. 

There are several ways you can mitigate these risks when using a cloud based solution such as Packetloop, including:
  • Storing your full packet captures in your own S3 bucket and providing us the keys to process
  • Sending us an encrypted drive with the source data
  • Implement the Packetloop onsite appliance (see below).
Of course you can also delete the data you have uploaded once we have processed it, but then you miss out on the full benefits of the Packetloop platform, the ability to search older stored full packet captures for previously undetected Zero Day attacks. An analogy for this sort of retrospective review of older data for previously undetected attacks can be found in sport. Athlete's drug test 'B' samples are kept for up to seven years, and are retrospectively tested to see if they athlete was in fact using a drug that was went previously undetected. This has had a lot of press lately with the downfall of a certain multiple Tour De France winner. Bruce Schneier wrote a great article for Wired this week about this very topic, and the power of being able to look into the past for answers. The parallels to Information Security are very real. It's our belief that over the next 5 years full packet captures will become the standard for logging and analysis of all data, including sensitive data, as this is the only way you can currently use existing IPS to produce indicators and warnings about potential threats, and its the only way you can Play, Pause and Rewind your network data.

We do however acknowledge that for some organisations, their data classification or regulatory position will simply prevent them from using our cloud based security analytics service. We understand this, and that is why we intend to create an on premise appliance version of Packetloop straight after we release commercially in the cloud. Ultimately, we are trying to provide the best Big Data Security Analytics tools in the market, and we will let you choose your level of involvement.
21 Oct 2012

Finding Needles in Haystacks @ Ruxcon

0 comments Permalink Sunday, October 21, 2012
Yesterday I was in Melbourne presenting "Finding Needles in Haystacks (the size of countries)" at Ruxcon. If you are looking for the latest version of the slides they are here - [PDF] [Slideshare]. It was an awesome conference with high quality presentations. Special thanks to Chris Spencer and the Ruxcon panel for selecting our CFP.
I was a little concerned about how it would be received as 'Big Data' hasn't really penetrated the security world yet. However that fear was soon dispelled and I think our visualisations really helped to reinforce the concepts.

The Worldwide Attack Globe received a great response. It showed almost 1 Million attacks over a 12 day period. This was a real world dataset from an early customer of Packetloop's.

The Worldwide Attack Globe can also be used to show/filter different data types. In this example I demoed how TOR endpoints can be plotted on the globe and then I zoom in on a very persistent attacker from the Republic of Ireland.

One of the concepts I wanted to focus on was that of data fidelity. Big Data tooling enables the ability to maintain full fidelity from years to minutes. Further to this sometimes it's seeing data in a different way or seeing it animate that brings on the discovery and knowledge. This was shown in the 'Full HD - Play, Pause and Rewind' demonstration.

Thanks again to everyone who attended and filled Room 1. Also thanks to all those who took time out to chat with us and share ideas.
18 Oct 2012

Teaming up with Hortonworks for Packetpig Blog Series

0 comments Permalink Thursday, October 18, 2012

Recently I connected with Russell Jurney @rjurney on Twitter after he posted a couple of tweets related to Packetpig. Russell works for Hortonworks a Big Data Platform company founded by Alan Gates one of the developers of Pig and author of Programming Pig.

I had been following Russell after reading his datasyndrome blog which is an awesome reference for people keen to learn about Big Data and how Pig, Hadoop and NoSQL databases like Cassandra and Mongo can be linked together in pipelines.

Soon after this Russell asked if I would like to collaborate with him on a series of blog posts on the Hortonworks blog. The first of which came out recently.

So check out "Big Data Security Part One: Introducing Packetpig" - I hope you enjoy the post and the series!
28 Sep 2012

Packetloop Early Access

0 comments Permalink Friday, September 28, 2012

We are almost there! The last month has been spent on internal testing and refining the first release of Packetloop. We have now started an Early Access phase, providing access to a handful of our customers and industry experts to get some initial feedback and also to watch how they interact with the platform. We are very keen to see whether it answers all of the questions they have, that are not answered by current security platforms. So far the response has been very positive, with our Early Access users commenting on how easy Packetloop is to use and more importantly get results from quickly.

Once we have completed this Early Access phase, we will release the product to general access starting with the Threats module. This will allow you to create an account and explore Packetloop using the sample data sets we have provided. These data sets will give you a great insight into Big Data Security Analytics, and how Packetloop can help you gain clear network intelligence from your network data. At that point we will also make our upload features available, so if you wish to upload one of your own packet captures, you will be able to do so using our low monthly pay as you go fee structure. You can jump on our website and sign up to be notified when we release.

In October we will be in Melbourne at Breaking Point, and Michael will be presenting at Ruxcon. His talk, Finding Needles in Haystacks (the size of countries) is a continuation of the research into Big Data Security Analytics presented at BlackHat Europe earlier this year. If you are going to be at either event, look out for us as we are more than happy to discuss Packetloop's approach to Big Data Security Analytics with you in more detail, or give you a personal demo. We hope to see you there!
8 Jul 2012

Security Visualization

0 comments Permalink Sunday, July 08, 2012
Security threat data is complex. In most environments there is always a constant stream of new threat data. Traditional security solutions have their own way of classifying and presenting this data. Firewalls tell you what’s been allowed in and what has been blocked. IPS/IDS solutions will rank threats into 3 or 4 levels of severity. Security Information and Event Management systems (SIEMs) will attempt to bring some sense to these fragments of data by correlating similar events and providing you a summarised version of the attack. Reporting on this data generally means reviewing a series of rows and columns, grouped by event severity. To interpret the data you still need to watch these rows and columns for the important events and then analyse and respond to them. Blink and you will miss something important.
Packetloop Main Visualization
All this data is really showing you, if you see it before it rolls off the screen, is that an event occurred. It doesn’t tell you anything detailed about the event, nothing about the actual cause of the incident. At a glance, you can’t decide if you need to be more concerned or whether you are still secure. You can’t supply this data to your Board or executives to support your decisions because it doesn’t tell you anything about the trends. Is today different to yesterday or is this week different to last week? What deviates from normal?What sort of data do you want to use to inform your Board?

In developing Packetloop we wanted visualization to be at the heart of the platform, to provide context to help you interpret the data. To be able to glance at a screen and understand in an instant what is going on using simple visual cues, without having to spend time interpreting rows and columns of threat data. Instead of a spreadsheet style interface we believe our users will enjoy this.

Packetloop Main Visualization
The main visualization dominates the screen and the product will ship with the ability to toggle between line, stacked bar and bubble visualizations. If you’re blessed with a large display you extend the visualization full width or enter full screen mode which has been especially designed for data panels. The title bar gives you context and provides different high level information within the different views e.g Overview, Source, Destination, Attacks and Location. Data panels below the main visualization provide more details and trends. Drag the visualization left and right to navigate through time or use the calendar to set the date range you want to analyse. You can use your touchpad or mouse scroll wheel to zoom in and out from years to minutes instantly.

It is also important to represent graphically the more complex relationships in the data, the relationships that help you ascertain what the attacker did and what was accessed or stolen. Packetloop provides you with additional insight into the attack data including:

  • New Attacks - sources, destinations and attacks that have never been seen before.
  • Distinct Attacks - how many distinct destinations did a source attack, or how many distinct attacks did a source use against a destination.
  • Looped Attacks - identifying zero day attacks that were not previously detected when replaying packet captures.

We also view the Threat data through a series of different views staring with a high level Overview that can then be broken down into views by Source, Destination, Attack and Location. We can also provide additional context in relation to attacks such as the the use of anonymous proxies.

Where an attack physically originates from is increasingly important to track as this can be used to formulate high-level access and monitoring policies to deploy on your security devices that protect your assets. Packetloop can plot the location of attackers down to country and city, representing this on a global map, showing volume of attacks via different colours. This simple visualization provides a powerful yet easily understood snapshot.

Packetloop Locations Visualization

Packetloop is designed to help you establish a visual baseline for what is normal traffic. It allows you to view data from different angles or contexts and to zoom in and out of massive amounts of data. You can see the rates of change in data, and the deviations from your baseline. When you hover over the area of the visualization in question, a summary of the attack detail is provided without requiring further in depth analysis.

There is no point in storing Terabytes of data for future analysis, unless you are going to perform the analysis. Packetloop provides a comprehensive filtering capability that allows you to quickly trace an attack via any part of the conversation detail, linking different attributes together to refine the filter.

Packetloop Linked Filter
For example you could see quickly what hosts an attacker has accessed, and then you could identify one of these hosts, and establish whether anyone else had attacked it using the same vulnerability or other vulnerabilities in just a few clicks. The advanced filter capability also offers an intuitive search language that allows you to quickly construct searches for specific attributes in the packet capture data.

As we finalise the preparations for our Beta release, we will publish some more screenshots and videos of our visualizations that better explain what I have discussed above. With Packetloop, we aimed to create the Big Data Security Analytics platform that we would want to use. Hopefully you will be as excited as we are with what we’ve created!
29 Jun 2012

Zero Day Attacks – Your security past, present and future

1 comment Permalink Friday, June 29, 2012
The problem with current security enforcement solutions is that they only enforce the policy that is applied to them. They detect and block what they know to be bad based on vendor published signatures. They only alert and report on what they have been told you are interested in seeing. These solutions are incapable of detecting and blocking Zero Day attacks whether they are targeted or not. On an an ever decreasing timeline prevention fails.

Current security enforcement technologies are heavily focused on real-time. When they fail to alert using signature technology the moment is lost. If you blink you will miss it and never know that there was a targeted Zero Day attack against your network. The promised alert was never triggered and the information and context of the event is lost forever.

How does it make you feel when you hear or read about a newly discovered vulnerability or Zero Day attack? Do you get that uncomfortable feeling, wondering if you are affected? Do you truly know what is in your network?

Keep in mind the timeline of a vulnerability remembering that the timeline is accelerating.

How long Zero Day attacks are in circulation for before a vulnerabilities Disclosure Date is difficult to ascertain. However we know that Zero Day attacks are used and used quite often (RSA, Stuxnet, Flame etc). From our recent consulting research (Vulnerability Data Analysis) the time period prior to the Date of Disclosure can be considered the Zero Day attack window. The time window between a generally available exploit and the Disclosure Date is almost zero. In the example of Metasploit project there is enough data to state that working exploits are available on the day of disclosure.

This is a significant departure from the general view that IPS will provide some measure of coverage until a patch is released. It’s fair to say that not many vendors can truly detect or prevent Zero Day attacks. So how do you measure and review your exposure until you can apply the new IPS signature or patch? Furthermore how do you know that the remediation work performed protects you from this specific vulnerability going forward?

The reality with IPS/IDS and Next Generation Firewall vendors is that you won’t know. Until the new signature arrives, you won’t know whether this vulnerability has been exploited in your environment. Assuming the worst has happened and you find that you are vulnerable and there have been exploits, what is your next step?

Device logs may show hints of the exploit in the abscence of an IPS alert, but only at a point in time, and then only at the control points where you have deployed the security device. In some cases you may not even be logging the type of traffic (see my previous post Dealing with Data Explosion) required to understand the actual event. You won’t have enough of the data you require to understand the actual event.

It’s not the kind of problem you want to have and to be honest these are the kinds of problems we are designing Packetloop to solve. Aside from the high level goal of visualising opaque networks in respect to Threats, Sessions, Protocols and Files we want to find Zero Day attacks, identify the breach, establish the time window, determine what was stolen and provide full context of all these events to our Customers. We want Packetloop to play, pause and rewind the actual traffic that was part of the breach and provide as much context as fast as possible. A platform that allows you to easily sift through Terabytes of data and unlock the complex relationships that exist within network traffic streams and how they change during a successful breach.

By unlocking the power of Big Data and applying it to Security Analytics, you will also be able to gain a valuable insight into how your security systems performed during the breach, and where you need to strengthen your defences. There's only so much money to go around. Apportioning your security budget is difficult especially when you are making decisions without any data. Packetloop shows you your threat landscape and allows you to direct budget to where it offers you the most effective security. Packetloop gives you the Security Analytics you can use to make better decisions giving you clear intelligence for your network

Join our Beta release in the coming months and understand the power of Packetloop. We are focused on solving these problems and providing you with the information you need.
22 Jun 2012

Dealing with Data Explosion

0 comments Permalink Friday, June 22, 2012

Data storage capacity requirements in today’s enterprises are increasing at an alarming rate. According to CSC research, average enterprise data capacity will need to grow by 650% in the next 5 years. This is driven by increased user connectivity and an organisation’s dependence on the information these users create and exchange. Couple this with today’s mobile user requiring ubiquitous access to their data from any platform, anywhere, anytime and we can see why organisations are struggling to keep up with the storage requirements for this data explosion.

Security data storage has always had its own challenges. Firewalls, IDS/IPS and Vulnerability Assessment systems produce an ever expanding amount of device log data that is invariably stored for a period of time (up to 3 months) which may stretch out to years to meet policy or regulatory requirements. But do these stored logs help you when you have a security incident? Will these reams of logs be enough information to understand the incident, the breach, the exposure? Would you be able to perform the necessary forensic analysis on these logs? How often have you witnessed security professionals only logging blocked traffic, when the traffic you are really interested in is what is actually being passed by your security devices into your environment. This is the traffic that contains the serious threats worth worrying about.

In my previous post on whether  big can data solve the unfulfilled promise of network security, I discussed the traditional logging and reporting paradigm, and how it doesn’t allow you to reproduce incidents with enough fidelity to detail the breach, the time the intruder was inside, the systems they accessed and the data they stole. Device logging  doesn’t give you full range of options, and it may not even alert you to an incident. The only way you can truly assess the security of your network is to analyse full packet captures of your traffic, you are presented with a new and interesting challenge. However a single gigabit network can transport terabytes of traffic a day. How and where do you store full packet captures (weeks or months) of your network traffic.

The only true representation of your data, is the data itself. The only way you are able to play, pause and rewind attacks completely is to store an entire copy of all the traffic.
The Cloud offers extremely low cost, high capacity storage which is perfect for short term storage of this sort of data. It offers secure upload and encryption, and it can be replicated and distributed if required. You only pay for what you use, and for how long you use it. Coupling full packet captures with Cloud storage makes perfect sense. You capture the data, upload it, and let someone else store and process it for you.
Packetloop accesses the full fidelity of the data. It gives you play, pause and rewind It has access to all events and can replay them any time with new insights to find blended and sophisticated attacks or exfiltration. It scales, it's focused on providing executives with the metrics and overviews they are looking for (dashboards) but powerful enough to track and trace incidents.

Packetloop is designed to leverage Big Data to perform analysis of terabytes of full packet captures. Scalable to handle the data on your network now and into the future. Shouldn't you be giving your organisation the best chance of detecting intruders, containing the incident and remediating with the best evidence and information?
15 Jun 2012

Big Data: Can it solve the unfulfilled promise of network security?

0 comments Permalink Friday, June 15, 2012
We have now had nearly 20 years of vendor promises that if you buy and deploy this software or that appliance, you will be protected. And yet breaches are still occurring at an increasing rate, and organised crime relating to online systems is at an all time high. What went wrong?

The vendor’s assertions that their systems will detect, block and report all manner of threats are only partially true. No one system can do all of this with 100% accuracy. So we have continued to purchase and install a range of complementary systems under the guise of “defense in depth”, but in reality to cover the weaknesses or gaps found in our other systems. And still properly motivated individuals will probe your defences looking for, and quite often finding, a way in.

Current thinking dictates that we will block everything that doesn’t match an access policy, and then we will deploy a range of real time threat detection systems to deal with threats we are seeing in that accepted traffic. Finally we will add some sort of correlation system to consolidate and present the findings. We generally deploy this technology at the most obvious locations such as our Internet or partner links, remote access points, or around publicly accessible systems. We then need to employ a handful of skilled people to administer these systems, interpret the reports and alerts, and react to the findings.

This approach has a number of weaknesses;

  • Detection systems that find “zero day” attacks usually can’t do so until they have an updated signature.
  • The volume of information is large and accelerating
  • Single threat systems have their own error rates, which include false positives and true negatives
  • It only shows you what is happening right now, at this moment in time
  • It only shows you what you have asked it to show you, based on what the vendor can detect, collect and store.
  • It doesn’t allow you to query the data in any other way, to infer any other relationships.
  • These systems are biased to the vendors view only, which is carried through to the logs and therefore the reporting.

Most importantly, it doesn’t tell you anything about how secure you actually are. It doesn’t deliver any sort of baseline or metric that you can use for comparisons. It doesn’t tell you whether you are overly targeted. Assume for the moment the worst happens, and you are compromised. This current approach to security logging does little to assist you in determining when and where an attacker gained access, and it certainly does not help you identify what else this attacker did after gaining access. Compounding all of these problems is the increasing use of encrypted protocols to mask an attackers actions.

The team at Packetloop have worked in the IT Security industry for the past 15 years, and have seen the above problems replayed constantly with our customers. Our consultants have configured countless threat detection and SIEM systems applying everything we have learnt. We have done our fair share of security reviews and incident responses but have always been plagued by the question, “What aren’t we seeing here?”. There is always that nagging doubt that we are not seeing the entire picture, and that logs will only show us what was detected at a particular security check point in the system, and not the full conversations the attacker has with multiple systems once they gain access.

We were being challenged with these same problems by CISOs. In the face of increasing security spend, they were being challenged by their Boards who wanted to know:

  • How secure is my organization?
  • How does my organization compare as a target with similar organizations in our sector?
  • Is my organization really maintaining our compliance and regulatory obligations?
  • How effective is my organization's security spend in relation to our level of security?
  • Does my organization have the means for fully understanding the extent of an attack?

We asked ourselves what was the best form of data or logs to analyse in order to answer these questions confidently. The only answer we kept coming back to was full packet captures, taken from the most trafficked areas of the network. The packet capture gives us the exact context of the attack, and it allows us to investigate in a multitude of directions using the original data rather than a vendors logs from a specific network location. Most importantly, because Packetloop stores an exact replica of the data, we can replay it over and over again through a multitude of different threat systems, and we can test old data for the presence of newly identified threats.

At the same time as deciding that full packet captures would be our data source we were also looking at what tools we could use to find the sort of evidence we were after in such a large amount of data. All of the currently commercial tools that could offer some of what we were after required huge investments in capital for probes, collectors, consolidation servers and dedicated storage arrays for holding the data. What we needed was an engine that could process and store the huge amounts of data, but did so cost effectively, presenting the data in such a way that I didn’t need a doctorate to understand the findings.

This is where Packetloop came from. The simple premise of building the tool that we would want to use in our own security consulting business, and that could answer the questions and problems posed above. Packetloop is focused on Big Data Security Analytics, using the efficiency of the Cloud to store mass data. Packetloop aims to deliver high quality business intelligence, in an easy to understand format. Over the next couple of posts we will share some of the more detailed analysis of these problems and the thinking behind how Packetloop has been created to solve these issues.
5 Jun 2012

Finding Needles in Haystacks the Size of Countries

0 comments Permalink Tuesday, June 05, 2012
There has been a lot of interest in our presentation from BlackHat EU 2012. To save you having to download the complete video you can watch it here.

21 Mar 2012

Packetpig - Open Source Big Data Security Analysis

1 comment Permalink Wednesday, March 21, 2012
Packetpig is an open source project hosted at Github that allows full packet captures and device logs to be analysed. We describe it as Big Data Security Analysis - a way of analysing and applying Network Security Monitoring principles to big datasets.

Packetpig is made up of a series of Pig Loaders (Java Classes) that exposes packets captures so they can be analysed at massive scale;
  • PacketLoader() - opens packet captures and provides access to TCP, UDP and IP headers e.g. Source IP Address, Source Port, Destination IP Address, Destination Port.
  • SnortLoader() - wraps the Snort Intrusion Detection application allowing packet captures to be analysed across a Hadoop Cluster. The loader analyses packets and returns signature, priority, message, protocol and Source IP/Port, Destination IP/Port.
  • ConversationLoader() - links packets to their conversations or flows. The conversation start and end, the way the conversation ended, the number of packets, their size and delay can all be extracted through this loader.
  • DNSConversationLoader() - provides additional functionality for the deep packet inspection of DNS conversations.
  • HTTPConversationLoader() - provides additional functionality for the deep packet inspection of HTTP conversations.
  • ConversationFileLoader() - allows file metadata and files themselves to be extracted from conversations. The file name, extension, libmagic information as well as MD5, SHA1 and SHA256 hashes are returned through this loader. In addition the actual files themselves can be extracted and dumped.
  • FingerprintLoader() - a wrapper for p0f that allows it to operate across a Hadoop Cluster.
  • PacketNgramLoader() - extracts data from each packet in a conversation and breaks it into an N-Gram. Unigram, Bigram and Trigrams are most commonly used however any integer can be passed to the loader.
Google WebGL Globe of Snort Alerts
Loaders are called in Pig files written in PigLatin. Multiple loaders can be used to analyse data. For example you may want to take all sources of attacks and see whether their operating system matches their user agent. This would involve using the SnortLoader(), FingerprintLoader() and HTTPConversationLoader().

Firstly you would parse all packet captures using the SnortLoader() to find the distinct Source IP addresses linked to Snort attacks. Secondly you would parse all packet captures using the FingerPrintLoader() (a wrapper for p0f) that would provide information on the operating system using passive analysis. Thirdly you would parse all HTTP conversations using the HTTPConversationLoader() to extract the User Agent field from all conversations. Finally you would join the data together on the Source IP address to output the analysed data linking attackers to their operating systems and their user agents.

SSH Trigrams Visualised in 3D Space
The Packetpig Loaders are the building blocks for analysing full packet captures. There is nothing stopping you from also integrating device log files if required. The Packetpig project also includes a 3D Globe, World Maps and Line Graphs for time series and NGram visualisation.

All of us at Packetloop hope you enjoy the project and we are happy to accept pull requests if you wish to contribute.

15 Mar 2012

Blackhat Europe Finding Needles in Haystacks (the size of countries)

0 comments Permalink Thursday, March 15, 2012
At Blackhat Europe 2012 I unleashed the subject of Big Data Security Analytics and Network Security Monitoring. The presentation was "Finding Needles in Haystacks (the size of countries)" and you can find the slides on Slideshare or download the [PDF].

I knew the audience wouldn't be familiar with Big Data technologies such as Map/Reduce, Hadoop and Pig but they have a keen sense for the changing nature of attacks - that they are becoming more subtle, complex, blended and frequent. We only need to look at 2011 and the major companies that were exploited in that year.

During the talk I showed the "Let's Enhance" video and stated that it was a good metaphor for security analysis. It juxtaposes the hollywood detective with our understanding of the real world. In terms of Security it makes you think of the context you need to find structured attacks against your network. In security we are dealing with a problem of scale and accuracy. Charged with finding needles in haystacks we can barely correctly capture security events. This is why the video is so funny.

These Hollywood 'analysts' have almost magical tools that afford them capabilities we could only dream of as security analysts. They are -
  • Enriching data when we constantly face a  loss of resolution and fidelity.
  • Playing, Pausing and rewinding events but we have one chance and then it's gone.
  • Exploring data in vector space, building context and entropy but we are looking at isolated and disconnected events.
  • Focused on detection however the security industry is still heavily focused on prevention.
  • Investigating  events after they have happened but we are geared towards preventing an unknowable future state.
  • Operating on a complete copy of the event when the best we can often summon is a log or correlated log store.
  • Using algorithms to process features and vectors from data which is a subject that is not even being looked at in terms of security.
So I proposed taking the core concepts from Network Security Monitoring (NSM) and combining it with Full Packet Capture (FPC) and Big Data tools to provide the ability to investigate incidents at mass scale.

We delivered this as an open source Big Data NSM tool called @packetpig, you can find the Github Repository here.

Packetpig can analyse packets at terabyte scale. The data analysis language (like a query language) of Pig lends itself nicely to exploring terabytes of full packet captures. The beauty of Packetpig is you can write a query on your laptop against a small sample of data and then execute the query on the cluster against months or years of traffic captures. Packetpig also comes with a large number of examples.

Packetpig is the first Big Data security tool, it's open source and available for anyone to use. It combines big data analysis with some pretty stunning visualisations. I demonstrated a number of these during the presentation. They included the Google WebGl Globe displaying 420,000 snort alerts across approximately 12 days of full packet captures. I also demonstrated the full capabilities of Packetpig in the areas of threat analysis, traffic analysis and payload analysis including an awesome way of visualising trigrams using an NGram Cube. All of these features will be showcased on the blog over time.

Analysing large data sets gives security analysts new capabilities and this was demonstrated towards the end of the presentation when I used BitTorrent seeders and leechers to triangulate the source of attacks to confirm what IP addresses were common to individual attackers. This involved finding distinct attackers out of 420K individual events (3 Billion packets) and matching it to 180,000 Seeders and Leechers we tracked across Piratebay's Top 100 Movies, Music and Books.

Thanks to everyone that attended the briefing and also those who stayed back to ask questions, discuss their own situations and problems and the capabilities of Big Data Security Analytics.

You can follow @packetpig on Twitter but also download and use the code on your own traffic captures!