The term “Big Data” is definitely a leading contender for the marketing buzz-phrase of 2012. On November 11, 2011, a Google search on the phrase “big data” got 252,000 hits. By April 4, 2012, that had number increased by a factor of 5,000, to 1,390,000,000 results. And for good reason A 2010 IBM/MIT Sloan Management Review survey of 3,000 executives across 30 industries from 100 countries reveals that 60 percent of respondents have more data than they can effectively use. An IBM study of 1,700 CMO’s from 19 industries and 64 countries further exposes this issue with 71 percent saying their organizations are unprepared to handle the explosion of big data. The pace of data acquisition is accelerating and everyone is just trying to keep up. While IT talks about "speeds and feeds,” and data miners tend to talk about techniques, CMOs need to know how to effectively use data and analytics. This article lays out some of the basic IT and analytic concepts around Big Data, and provides concrete examples of how to use Big Data infrastructure to drive results. In late 2011, Gartner released a report identifying the top ten strategic technologies for 2012. On the list were the following four Big Data technologies:

1. Next Generation Analytics
2. Big Data
3. In-Memory Computing
4. Cloud Computing

These all go together to solve the information challenges faced by marketers when they adopt the technologies that enable truly personalized communications, product recommendations, entertainment, and advertising. At the end of each section there are keywords with brief explanations to help you find more information.

Next generation analytics

Next generation analytics simply refers to analysis methods that can run in real time against Big Data. Statistical methods like regression do a lot of computation that takes time on even the fastest computer. They also require that the data meet specific requirements; the one everyone is familiar with is a normal distribution. On the other hand, newer data mining methods are much faster and some of them do not require that the data meet specific requirements. All of the real-time analytic methods are based on data mining algorithms. The importance of high speed next generation analyses is in the ability to adapt what marketers do in real-time to what the mathematical models tell us our customers want and need. Data mining methods are most often discussed by expert analysts in terms of associations between "things," searching for shapes in the data, measuring distances between objects or looking for places where relationships change. This corresponds to what marketers have to key off of when we tailor an offer. Has the kind of product or the frequency of purchase changed for an individual customer? Does "Doug buying 'X' mean that we ought to offer him 'Y' before he goes offline?" Embedded analytics is a descriptive way to think about some elements of next generation analytics. The analytics are embedded in software that is always running on the compute infrastructure. This is an absolute requirement for making sense out of massive amounts of data. Done the old fashioned way – taking a sample of the data, having an expert analyze the data, formulating the results, and working out a set of decision rules can take anything from two days to two weeks. Two days to two weeks just does not cut it when we want to send someone a follow-up email the day after purchase or show them another offer before they go offline. Marketers have to consider everything from promotions, contacts, and purchases to factors like paydays and economic news, all of which affect purchasing behavior. Having an analytic result two weeks later just doesn’t keep up with the flow of data. The analyst is always either getting further and further behind if they are trying to analyze every day, or they are taking snapshots that give an incomplete picture of the “sales space.” From an IT/analyst standpoint, these high-speed analytics require high-speed access to data. The more complete the data the better. And, of course, more complete data is "Bigger." The next three topics talk about aspects of "Big Data" and how IT and analysts are dealing with it. Keywords include but are not limited to:

• “R” – an exemplar of a flexible high-speed analytics package whose code can be put directly on a computer to spit out fast results
• IBM’s InfoSphere BigInsights – an example of a high-speed analytics package tied closely to data
• High speed data mining – which pulls up algorithms and mathematics

BIG DATA

For this discussion, Big Data refers to amounts of data that cannot be handled by the standard relational data structures, such as the star schemas that were developed commercially in the 1980s and became popular with corporations in the 1990s. One new approach provides for much denser storage of data by allowing us to put different kinds of data in the same column of a table. Instead of having to query hundreds of columns, where it takes time for the computer to find and load the data from each column, we can use fewer columns to speed up data access. Another approach reads data by columns, not rows. For example, when we want a file of email addresses, the old methods had to read each row, find the column with the email address, then write that email address to the new file. Column-oriented architectures just read the whole column of email addresses at once. In one engagement, we reduced the time to create a marketing database from 25 hours to 14 minutes in a test of a column-oriented architecture. As a marketer, there is no way you can keep up with daily changes in what each customer wants if it takes 25 hours out of a 24 hour day just to create that day's data. Keywords for "Big Data Architectures" include but are not limited to

• Hadoop
• Greenplum
• Cloudera

IN-MEMORY COMPUTING

In-memory computing refers to processing data at high speeds in the memory associated with a computer’s CPUs instead of reading and writing data to disk. In-memory computing is a component of being able to process Big Data in real time. Reading and writing to disk is slow compared to using the high-speed “Core,” or “RAM,” or “processor” memory that is connected to the CPUs or Central Processing Units of the computer. The key here for marketers is speed, since a large retail chain may process as many as 500,000 sales an hour. While that’s a lot of data to read, it is even more if we want to personalize a follow-on communication timed to coincide with when each customer is likely to be ready to buy a targeted product again. Keywords include but are not limited to:

• SAP and in-memory
• Microsoft and in-memory
• Intel and in-memory

CLOUD COMPUTING

Already familiar to most, cloud computing refers to tying a large number of large computers together into a much larger virtual computer. The cloud represents the massive cloud or collection of data these computers can support, some of it processed in-memory. It is becoming common for a cloud computing system to link 10,000 computers together. This aggregation of hundreds or thousands of computers makes a lot of in-memory data storage available at a scale and cost that cannot be touched by the massive computer systems that were popular as recently as five years ago. For example, Amazon Web Services (AWS) has launched ElastiCache, which is designed to allow enterprises to speed up their Web applications by allowing them to retrieve information from "a fast, managed, in-memory caching system, instead of relying entirely on slower disk-based databases." Because of the volumes of data to be processed (a petabyte of data is becoming common), a large number of computers are necessary to store the data, move it around, clean it, aggregate it, and report on it. Catalysis expects marketers will typically use results from data stores that exceed two petabytes within four years. Why? Because the more we know about our customers the better we can serve them. Keywords include but are not limited to:

• SalesForce and cloud
• Amazon and cloud
• Google and cloud

HOW IT WORKS – AN EXAMPLE OF MARKETING RULES

There are a couple of ways to build out an environment that uses embedded analytics on Big Data in a cloud computing environment that leverages in-memory computing. A best practice is described here. An initial set of big analytics is done to establish some starting marketing decision rules (sometimes called business rules, but these are much more complex). Marketing rules are tested until the system returns satisfactory results. For example, we could set a rule that has us offer heavy jackets for sale during summer to people who had previously purchased outerwear in June, July, or August. In November, there is a negative correlation between decreasing temperatures and increasing sales of heavy jackets so our rule may say that we always offer heavy jackets as the temperature being to decrease. In winter we'd offer jackets to those who previously bought outerwear when it was cold. These are just a few options; there are potentially thousands or even millions of possible rules. While these rules might produce results, an even more effective approach is to use a set of meta-rules to decide which rule to activate when. In our example, a meta-rule might tell us to:

1. Start with using the associations between the sales of different items by average outside temperature. This is the default rule.
2. If they have bought outerwear in a particular month, the meta-rule overrides the default recommendation and offers outerwear in that month.
3. If they bought a kid's jacket in that month, use external data to check if there are children in the household, then offers kid's outerwear because their child may have outgrown their last jacket and needs a new one.

The more data used, and the more specific the rule, the higher the potential return. After you develop the potential rules, each rule is "valued" by the probability and expected amount of the sale for each possible item to be offered. Then just the top three or four best offers are sent to each customer. If we want to highlight 20 products, the highest valued rules are shown first. Each customer gets a different set of offers tailored to them, increasing the retailer's relevance and their sales. Specially coded high-speed algorithms are needed to even approach analyzing the data. The algorithms have to be designed explicitly to run in massive computing environments. The data may be housed in a cloud computing environment and the analytic calculations may be performed in-memory to speed the formation of the results that in turn are used to create the marketing rules that are built into code that is executed on cloud servers.

CONCLUSION

Implicitly, the burgeoning technology of massive data and high speed analytics to inform customer communication will affect each and every marketer in the near future. There are already studies indicating more than a 200% increase in ROI for companies that effectively use big data. Leading edge marketers are engaging with Big Data and big analytics right now. We are using cloud services like Amazon, data stored in Hadoop, and analyses performed in R to segment our customers, identify associations between what we do and how our customers react, and predict customer future value to optimize marketing spend on a customer-by-customer basis. While the short-term investment in infrastructure, talent, and time may seem daunting, the long-term payoff will be huge.

ABOUT CATALYSIS

For nearly 20 years, Catalysis has specialized in the digital integration of award-winning marketing campaigns that drive connected, measurable results. Our clients include Microsoft, Moss Adams, Banner Mattress, Thunder Valley Casino, BabyLegs, and WineBid.

For more information, contact info@catalysis.com or visit our website at www.catalysis.com.

The information contained in this publication is general and is for informational purposes only. Catalysis makes no warranties, express or implied, in this material.