Self-Service in the Golden Age of Data

“Trust is the cornerstone of a data-driven business, so if your users don’t trust the data, that’s going to be a real problem,” said Eric Kavanagh, host of DM Radio Deep Dive, in the webinar Leave IT Alone – The Vast Value of Self-Service.

Kavanaugh set the stage for Kenneth Cunanan, Product Marketing & Analytics Manager at Looker, and in a separate interview Elena Rowell Product Marketing Manager at Looker. Kavanaugh, Cunanan, and Rowell talked about where we’ve been, how we got here, and the importance of trust for Self-Service Analytics.

History of Self-Service Analytics

Self-Service Analytics allows business users to access data necessary for decision making, which was a simpler process before Big Data, said Kavanaugh.

“In the old days, meaning, five or ten, and even twenty years ago, there weren’t that many systems that you’d be pulling data from – maybe 20 whole sources of data pulling into a Data Warehouse. Now, we’re talking hundreds, and even thousands of sources of data.”

In the late 1980s, Business Intelligence tools were created “to provide a way for business users to finally to access data,” he said. Data Governance was built in:

“If I was a business user and I was accessing data, I knew that it was the single source of truth, that was centralized, and also that was curated and monitored by the IT team because it was the IT team that actually built and managed these tools.”

The database technology was very slow, however, and Elena Rowell added that they were also limited in other ways. “Any question that a user wanted to ask had to be predefined by an analyst. And when they started to ask questions that they didn’t realize they needed to ask, they were limited by that architecture.”

The First Wave of Business Intelligence (BI): Monolithic but Accurate

The First Wave consisted of monolithic stacks like Cognos, MicroStrategy, and Business Objects among others. “It was during the early 1990s, so these systems were developed and evolved in a much different world of databases than today,” said Cunanan. They focused on transactions and operations that required inserts, updates, and deletions of records – not Analytics. “They were also expensive, so utilization management was a key focus for most enterprises,” he remarked.

Rowell agreed:

“In order to do deeper analysis of the data, small slices had to be removed from the databases, transformed, and put into specialized BI cubes and caches that were part of these monolithic stacks. The slices only allowed for specific sets of questions, with many constraints that required help from IT.”

As a result, most resources were focused only on mission-critical information. Ad hoc queries were mostly the purview of C-level executives. Everyone else had to wait in the “data breadline” that never ended.

The First Wave wasn’t without advantages though. Because data was so locked down, “the answers you got were accurate – the data was easier to govern,” she said. But, every change or modification took months. The systems were designed to maintain consistency, but that made them inflexible and meant they couldn’t keep up with business needs that evolved faster and faster.

The Second Wave of BI: Self-Service Causes Chaos

The personal computing revolution of the late 1990s gave business users a growing comfort and familiarity, creating a “demand for more data-driven decision making,” said Cunanan, where “everything got blown apart.”

Self-Service BI tools emerged that could run locally on a PC, allowing users to “side step the constraints of those traditional database models,” but this also led to a loss of Data Governance. Instead of having one centralized Business Intelligence tool, “you have an explosion of workbooks and spreadsheets, and it becomes very hard to manage.”

Beginning around the year 2000, “some people saw this pain and thought ‘huge market opportunity!’” said Rowell. That was the advent of the Second Wave of BI: Self-Service. If business users could take care of their own data needs, they wouldn’t need to rely on data people or those slow, expensive databases at all.

Or so people thought. Unfortunately, the reality was a little different.

According to Cunanan, “when a business person was working with data they understood intimately, these systems worked fine. But when the data was complex or unfamiliar, bad things happened.” The Second Wave tools were basically “Excel on steroids.” They lacked the governance of the First Wave tools, so any data that you didn’t personally understand was out of reach for reliable analysis.

The idea of “Self-Service BI” set out to make data professionals obsolete but, ironically, “instead made them even more critical,” commented Rowell. All the data curation and cleanup that used to live in the earlier monolithic stacks now had to be done manually each time a business user wanted to load data into their Self-Service BI tool. “The supposed solution for the breadline only made it worse,” said Rowell.

The process of putting data quickly and directly in the hands of business users began to erode trust in the value of the data. “If you had a bunch of people coming to a meeting, they all did their own separate analysis, but they’d all come to the meeting with different numbers,” remarked Cunanan. It became even worse as the cord was further cut from the central Data Warehouse – business logic was lost, key metrics became more difficult to report on, Metadata Management became unmanageable. “The Second Wave of BI is characterized by this lack of standard definitions and metrics. Even the data itself became unreliable,” said Rowell.

Negative Effects of the First Two Waves

According to both Cunanan and Rowell, there are two primary negative effects of the first major Waves of BI. Though these

• Data Breadline: “The data team has too many mouths to feed,” said Rowell. Overtaxed analysts had to prioritize work for company executives and therefore everyone else had to wait until later. As a result, employees couldn’t get critical answers when they needed them, thus slowing the company’s progress. “Companies try to cope by hiring more analysts,” she said.
• Data Brawl: “Data, and understandings of what that data means, fragment,” said Cunanan. This meant each analyst had to invent their own metrics and analyses. When teams would come together to make a plan, they would often end up with completely different numbers, and therefore would spend hours (or longer) debating over information before they could even start to formulate a data plan. “This caused data brawls,” said Cunanan, or arguments that many times would never be agreed upon and thus cause further fragmentation of company progress.

These issues, along with those both of them mentioned earlier, made it necessary for a revolution in Business Intelligence, but it has taken many years for the technologies to catch up with the need, along with the formulation of processes to make it all possible.

The Third Wave of BI: Self-Service with Data Governance

“That brings us to the present,” said Rowell. “Most people have been either stuck in a breadline or a brawl (or maybe even a brawl in a breadline).” The key constraints that defined those first two waves – slow, expensive databases – is gone.

The Big Data revolution brought a lot of hype, but one thing of value has delivered is “fast, cheap analytic databases (many of which even migrated to the Cloud),” commented Cunanan. “These MPP Data Warehouses and SQL-on-Hadoop systems are so fast and so cheap that you no longer have to extract your data to analyze it.” Enterprises can now do their Analytics right in the database. Such an enormous technological shift is the cornerstone of the Third Wave of BI.

Today’s databases are so cheap, companies can put all their data – from every SaaS application, every custom tracking code, and every transactional database – in one place and query it at will said Cunanan.

These databases are so fast, organizations can transform data after it’s in the database, “eliminating much of the heavy lifting that was required of traditional ETLs.”

Such a radical shift has vastly shortened the time it takes to get from a new question to an answer. According to Rowell:

“All the data is already in one place, it’s always fresh, and just one tool integrates data from different sources and governs the meaning of that data. That means data professionals can stop answering one-off requests and leverage the Data Platform to curate a model.”

A model isn’t static like a report, it’s a dynamic environment that lets non-technical users ask whatever questions they want, knowing that the platform will translate their question into the correct analytic query and deliver the right answer.

“Data Platforms of today allow business users to explore in a truly self-service way,” she said. “With Data Governance baked right in.”  It is now the business processes dictating the tools, rather than the other way around. “That is a revolution,” she said.

 Technology Won’t Solve Everything

Even with such seismic shifts though, IT is not going away. The advancements in technology are hugely important, but according to Cunanan,

“You have to take the knowledge and expertise of IT and codify it. As we learned from the Second Wave, a world without IT is chaos. IT’s knowledge needs to be baked into the tool the rest of the company uses.”

Cunanan believes that self-service creates relationships between IT and users. ‘Air dropping’ a BI tool for your business users and walking away will not empower them.

“Analysts are quite expensive and having talented analysts sit at their desk and copy and paste and adjust SQL queries is not an efficient use of that power,” Rowell added. “Plus, the end users are not served either, since they’re stuck waiting on those analysts for the data they need”

With the proper Data Platform, business users can ask and answer their own questions, Cunanan said, but that doesn’t mean analysts don’t have a role.

“They’ll still be the expert in that tool and the data sources. If I’m a business user and I’m curious, for example, ‘What question do I want to ask?’ That’s what the IT analyst is there for – to help train people not only on the tool itself, but also on the effectiveness of answering your own questions through that tool,” he said.

Once everyone is accessing the same, verifiable single source, trust can be built. Building trust is not about getting it right all the time. It’s about identifying when there’s a wrong answer, or the KPI is wrong, or the metric is wrong, so analysts can trace that back down to a single line of code in a model.

How Looker Facilitates True Self-Service

“Looker is a Data Platform that makes it very easy to provide Self-Service Analytics to your users while also keeping Data Governance,” Cunanan said.

Rowell added that Looker provides, “Freedom to slice and dice in a governed environment that’s analyst approved. The secret sauce of Looker is the modeling layer that sits between the analyst and the user.” That layer allows analysts to make data accessible to users, giving them freedom to explore and manipulate metrics however they want, she said.

According to Rowell Looker gives the freedom to think through data in an intuitive way, similar to an online search.

“You Google something and you end up on an article that then leads you to a blog, that then leads you into the store that sells what you were actually looking for. Being able to do that with your data – to ask one question and continue to iterate on that question and dive deeper – is what I would say sets Looker apart, that exploration and hunger for information.”

Corresponding Podcast

The podcast had guest speakers: Rick Sherman of Athena IT Solutions; Elena Rowell of Looker; Douglas Jardine of AirDNA; Jeff Burk of Dell Boomi; and Andrew Jeung of Thoughtspot. The speakers, along with Kavanaugh, discussed the complexity of forces causing dynamic change within the data landscape today, and why such technologies as Self-Service Analytics are becoming ever more important.

Photo Credit: Sergey Nivens/Shutterstock.com