Three Data Challenges Shaping the Future of Analytics

By Laura Shores - May 5, 2020

During our recent webinar, “The Future of Analytics: Accelerating Operational Decisions,” we discussed three primary data challenges that modern companies face. Jad Naous from Imply and Peter Bailis from Sisu explored how these challenges are shaping data architectures and solutions, and forcing companies to adapt their analytics strategies.

In the discussion, Jad explained what’s motivating companies to change their ways. He said, “The ability to make decisions at the speed and volume of daily activity is critical to winning in this market.”

The three data challenges that are shaping the future of analytics include:

  • Too much data, too little time
  • Our BI tools are fundamentally incompatible with the kinds of data we collect 
  • Companies don’t have a model for what great data-driven decision making looks like 

Companies lack the time and resources to properly analyze the datasets they’re collecting

“Even though we have more data available today, data usage over time is actually declining. That’s because people have a fixed amount of time.” – Peter Bailis

This first data challenge is a self-inflicted one. Companies are amassing data on every event and step in the business process. They’re efficiently aggregating terabytes of data inside modern data warehouses – but they’re now sitting on a goldmine of information they can’t use. 

These organizations struggle to find real strategic value in their data because they don’t have the time or resources to look at every factor collected in the data. It’s an unfortunate truth that handicaps most teams: the more data we collect, the less of it we use. We’re leaving a lot of data ROI on the table.

Peter explained that there has always been fundamental inequality between available data and analytics resources, but there’s a critical change that’s happened over the past few years. “Historically, people were cheap and data was expensive,” Peter said. “The new normal we’re seeing with the rise of cloud data warehouses is that the inequality has flipped. Data is cheaper than ever to collect and store, and people’s time has become more valuable.” 

When you’re collecting massive amounts of data, the core question becomes how you prioritize people’s attention? When a change occurs, are you able to not only detect it, but also to dig in and comprehensively answer why the change occurred in order to take action? 

Even Modern BI tools are incompatible with the data we collect 

The data collected across many industries is both wide and granular. For example, by the time you click “subscribe” to a monthly box service, the company has already collected information on your acquisition channel, your demographics, the time you’ve spent on their digital properties, and much more.

If you’re a subscription business, you need to track all these factors to understand what’s driving changes to your core metrics. Traditional BI & dashboard tools were not designed for this type of data, and require extensive data cleaning and aggregation in order for users to visualize trends in their business metrics.

Peter succinctly described the predicament companies find themselves in as they simplify their data for use with their current tools. He said, “To unlock the value of this data, we need data infrastructure and tooling purpose-built for the kinds of information we capture today. We can’t afford to spend 60% of our precious time ‘dumbing down’ our data for yesterday’s BI tools.”

Additionally, given the amount of data prep required, the information presented in traditional BI tools is static. This simply doesn’t support an environment where decisions are being made continuously. 

Everyone aspires to be data-driven, but most don’t know what that looks like in practice

As we explored what data-driven organizations really look like, Jad explained that many companies don’t know what insights they’re missing from their data. Often, they don’t ask complex questions because they don’t know the true capabilities of their data.  

Jad said, “For enterprises, the biggest issue is that companies don’t even know what questions to ask, or they don’t have high expectations of their data.” 

Data-driven decision making is the result of human intuition paired with data. In true data-driven cultures it’s not about the insights you’re generating, it’s about the action that results from them. Most companies fall short on taking action, and don’t effectively use data to inform every key decision. 

“The business is going to keep making decisions and no one wants to make a bad one.” – Peter Bailis 

As more and more data is available to the organization, it’s no longer an option to simply rely on “gut feel,” especially when there’s an option to check your hypotheses based on what’s really happening in your business. What we see for the future of analytics is a world in which the operational analyst is empowered to make decisions, and an increasing amount of the rote, routine, repetitive work of data exploration is increasingly automated.

Listen to the full virtual Fireside Chat here, and hear more from Jad and Peter on the effects of digital innovation on operational decisions, the role of the analyst, and unlocking value from your company’s existing data and institutional knowledge. 

Read more

What's Remote Got To Do With It? Making Analytics More Collaborative

Listen in on the virtual fireside chat we had with three analytics leaders with deep experience building successful, highly collaborative, remote-friendly teams at GitLab, Fishtown Analytics, and

Read more

451 Research:
Business Monitoring Tools Using Machine Learning to Analyze Metrics

You need fast answers about what’s happening in your business. Get a complimentary copy of 451 Research’s latest report on the opportunity to achieve immediate, detailed visibility into your data and critical KPIs.

Read more