Growth Series BLOG

Its a Premium BLOG template and it contains Instagram Feed, Twitter Feed, Subscription Form, Blog Search, Image CTA, Topic filter and Recent Post.

All Posts

Extracting Insights from Big Data – Building a Knowledge Architecture

This Articles originally appeared on CMSWire.com on March 15, 2016.

Every CIO has heard about the promise of Big Data – gleaning new insights from customers, markets and competitors, speeding time to market, detecting anomalies and security outliers and providing the foundation for machine learning algorithms that form the precursor of Artificial Intelligence.  The challenge is getting from “here” to “there”.  There’s lots of promise and potential but there are still many challenges and roadblocks that stand in the way.  This Articles discusses the “knowledge architecture” of the enterprise and ways to develop and apply it in the short term that will lead to new capabilities in the long run.

“Big data is like teenage sex:...

: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it...” Dan Ariely, Author of “Predictably Irrational” 

I might not go so far as to say no one is doing it--many organizations are getting value from big data.  The ones who are “doing it best” are the ones that understand their objectives, have identified the questions they want to ask, and have done a good job of architecting their data and process to produce and act on insights.  This begins with identifying what data you have and understanding its value.  The challenge is that most organizations are not aware of the full scope of data sources at their disposal and don’t consider how that data can be applied in order to produce value for the enterprise.  Consider the data “exhaust” thrown off by marketing applications. Marketing integration suites capture a huge variety of behaviors and interactions.  If that information about the user’s “electronic body language” is not instrumented into content processes and engagement strategies, it goes to waste.   

Double Loop Learning – Pulling Levers versus Changing the Levers to Pull

The information doesn’t need to go to waste, but in order to use it well, some questions need to be answered: What you are trying to measure? Would you know it if you saw it? What would you do if you had it? These answers can be found in the data indicators related to the customer lifecycle and the learning and adaptation that result when acting on the data. These can be accomplished through “double loop learning”- the mechanisms of looping insights back to the organization for action.  The first learning loop is the cycle of data collection, observation and intervention and then cycling back to data collection in order to see if the intervention had the desired impact.  The second loop is a larger review of the process and macro objectives where lessons learned and interventions can change the hypothesis and data collection.  The objective is the same; however, the lessons learned are taking place at two levels – macro and micro.  At the micro level, we are making small adjustments to content, organizing principles, product groupings, associations, cross sell rules, upsell configurations and “pulling levers” based on the data. At the macro level, we are changing our hypothesis and the types of interventions that take place at the micro level (“changing the levers” that we are able to pull).   

Clues from the User Vapor Trail

The “vapor trail” of user behavior data can tell us that a piece of content, a relationship or merchandising/shopping attributes are not effective – they are not leading to clicks, conversions, or downloads or whatever behavior or response we are trying to achieve.  There might be several smaller adjustments to the content and product data relationships or ways to surface the content where the impact can be measured.  However, if an entire program or engagement process is not effective, then the metrics collected might lead to a significant revamp that requires development, testing, quality review and deployment.  This process can take months from start to finish.   The clock speed and intervention action of these loops is vastly different. 

These scenarios depend on identification of a hypothesis about the correlation of observed data and the action.  But what if we don’t have a clear understanding of the outcome or correlation?  There can be more variables than we can easily comprehend or the volume of interactions can be too large to make such changes manually.   

Adjusting content, product data and promotions for large enterprises at scale requires more automated approaches, but these approaches still require metrics, objectives, and hypotheses as part of the feedback loops that machine learning algorithms can leverage. 

Back to Big (and Large) Data

 But what about the definition of big data? We have all heard the Volume, Variety and Velocity definition.  What does that really mean when it comes to extracting value?   

There are large data sources and big data sources. Large data might be transaction processing output from a large retailer.  Such data is well structured, of good quality and with a defined schema – that is, all of the elements are understood, consistent, and normalized so that apples to apples comparisons can be made.  In contrast, big data that streams in from a variety of sources might contain different definitions of a customer – one defined as a family and the other as an individual. In that case, it will be hard to compare results.  Moreover, in a big data scenario, the volume could be so great that conventional hardware and software takes too long to perform analyses or becomes too expensive to scale, even if the data is clean and well-structured.

Big data technologies help address this particular aspect by running the analyses on lower cost commodity hardware and splitting processing up into smaller jobs.  But in general, the big data that people are really interested in is not well-formed, defined, clean, or consistent.  The most interesting insights and new discoveries come from analyzing and combining disparate sources and processing these in a way that looks for patterns and correlations across data sets. 

Time for a Nice Swim in the Data Lake

For example, weather and traffic condition impact a retailer’s sales.  Those data feeds are coming from different systems in different formats, or potentially from different sources within a particular system. The systems may have different designs and conventions for naming the data elements.  So-called “data lakes” allow these disparate types and structures to be stored in a repository without a predefined structures that traditional systems such as data warehouses require.   We can keep adding data sources that consist primary of sensor data or text information – Tweet streams or remote traffic monitors – and use algorithms to process and analyze the information and look for patterns.

The other characteristic of big data that we need to be concerned with is how fast the data changes. Sensor data might stream all day long versus sales results that are processed at the end of the day as a batch. If the goal were to correlate sales with senor data in real time, the velocity of the sales data would increase. (Real time versus batch Data for a single product category changes more slowly than the cumulative transactions across all categories.  There is an increase in the amount of data (the volume) but also an increase in how quickly it changes when sampled in real time.    This is the velocity of the data.  Considering that sensors can throw off continuous streams of data (perhaps monitoring not just vehicle traffic but pedestrian flows) and that there can be hundreds or thousands in a target region, and the velocity continues to increase.

Enter Omnichannel

An omnichannel campaign objective might be to understand the impact of traffic and weather on in store and mobile promotions and customer segments.  The data might include sensor data from stores measuring pedestrian traffic, clickstream data on the retailer’s web site, and mobile data from third parties correlated with anonymized demographic data.   This data is coming in very quickly, with new data points generated every second. We now have physical and behavioral data combined with traffic, weather, pedestrian traffic, mobile phone data, and we have that across dozens of demographic segments (the granularity of this data is scary – e.g., down to looking at “trend-setting soccer moms without college degrees interested in crafts,”) and we now add in our promotional campaigns  for the spring. 

What do we make of such a data stew? (Ah, a new buzzword – data stew).  This is the point at which the data has to be processed in some way.  Contrary to popular belief, big data may be messy but we do have to do something with it to make sense of the patterns.  A set of hypotheses must be developed in order to select particular information for analysis.  If we are looking for conversations about our products, there needs to be a definition of the product and the variants in how people will describe the product – including misspellings.  If we want to know positive or negative sentiment, the system has to determine if there are variations specific to product characteristics that people will call out in a positive or negative way. 

Since different systems will define sensor data parameters differently, those also need to be reconciled.  The ways we process the data are part of a family of algorithms called machine learning.  Machine learning looks for patterns, classifies data and content, and predicts patterns based on new data sets. But we need to know what questions to ask or know what to look for.  In this way, big data is no different from BI and traditional analytics. It is possible to ask for anomalies or to look for outliers or to find patterns but what outliers and patterns?  The conditions of low foot traffic on sunny days?  The makeup of pedestrians on cloudy days when we are running our clearance sales?  The characteristics of people who use promotions sent on iPhones who dislike the competitor products? The rate and conditions of product failures in harsh conditions?  The frequency of products purchased as a part of high margin solutions correlated with customer segment and promotion strategy?  The questions and possibilities are endless.

Our Lake can get Swampy

In order to make the data useable it cannot stay in the lake forever - unless the data is organized and extracted, the data lake will become a data swamp. Just as with BI and data warehousing, we eventually need to have reference data – the standard names of products, markets, customer types, promotion types, demographics, classes of outliers, classifications of data types and conditions, security and privacy constraints and types of learning algorithms and processing models.  Data and metadata need to be catalogued and defined with the correct history, lineage, ownership, usage rights, and source information, and quality and accuracy assessments. 

We need to organize the data in catalogues with meaningful business terminology.  We need to harness the power of big data and machine learning with organizing principles that are the foundation of all analysis.  

This is not a trivial task.  In some cases, it is left to technical teams and in others to business teams, but they may not collaborate effectively or have a view of the larger picture.  A contextual enterprise architecture is the scaffolding and framework for knowledge in the organization. Increasingly, that knowledge is gleaned from diverse sources and hidden in streams that continually flow through the organization. 

Big data needs to be smarter, and contextualizing it adds the intelligence.   Business users are looking for insights.  Insights come from understanding patterns in data and seeing causal links and connections.  We can start to predict the behaviors of customers and employees and the performance of products and promotions in the field by finding the cause and effect relationships between the things we have control of and the things we want to influence.  The starting point is consistency of language, concepts and terminology – the knowledge architecture –supported and maintained by governance and ownership.

Earley Information Science Team
Earley Information Science Team
We're passionate about enterprise data and love discussing industry knowledge, best practices, and insights. We look forward to hearing from you! Comment below to join the conversation.

Recent Posts

Designing AI Programs for Success - a 4 Part Series

Recorded - available as on demand webcast AI is plagued by inflated and unrealistic expectations due to a lack of broad understanding of this wide-ranging space by software vendors and customers. Software tools can be extremely powerful, however the services, infrastructure, data quality, architecture, talent and methodologies to fully deploy in the enterprise are frequently lacking. This four-part series by Earley Information Science and Pandata will explore a number of issues that continue to afflict AI projects and reduce the likelihood of success. The sessions will provide actionable steps using proven processes to improve AI program outcomes.

The Missing Ingredient to Digital Transformation: Scaling Knowledge Communities and Processes

The holy grail of digital transformation is the seemingly conflicting goals of high levels of customer service and pressure to reduce costs. “Digital Transformation” has become an all-encompassing term – in a piece in this column about customer data platforms, I asked whether the term has lost its meaning: The phrase “digital transformation” can mean anything and everything — tools, technology, business processes, customer experience, or artificial intelligence, and every buzzword that marketers can come up with. Definitions from analysts and vendors include IT modernization and putting services online; developing new business models; taking a “digital first” approach; and creating new business processes, and customer experiences. The overarching objective of a digital transformation program is to improve end-to-end efficiencies, remove friction from information flows, and create new value streams that differentiate a company’s offerings and strengthen the customer relationship. Having assisted large global enterprises with building the data architecture, supporting processes, and governance for multiple digital transformations, in my experience, there are two broad classes of initiatives that seem to get funding and others that miss the boat in terms of time, attention, and resources.

4 Reasons B2B Manufacturers need Strong Product Data

There are many manufacturers who have started to take the leap forward in the digital space, but there are still a great number who rely solely on their distributors to manage their product data. We are going to look at 4 key reasons why its so important that manufacturers own their product and dedicate the time and resources to build it out.