Data, SQL, and Analytics: The Next Wave

Looker has taken the next evolutionary step in Data Analytics. Using SQL as a base, they have reimagined how Analytics are done, and rebuilt the process of Analytics from the ground up. In doing this, they have developed an approach to Analytics using proven long-standing concepts, by combining them with a flexible Data Platform and the latest technologies.

The need for such an approach has been around for many years, Looker understands that need and so created their platform to address it in the most efficient and straightforward way possible. Their platform allows everyone in an organization to access the data they need, answer the key questions about that data, and be able to understand those results.

Looker offers a modern Data Analytics platform that provides the data needs for an entire company. Looker’s Chief Data Analyst, Daniel Mintz discussed the power of Analytics in a recent webinar:

“I think the reality of most Business Intelligence tools, and even most Analytics tools, is that they were architected for a very different world. Ten years ago, databases were incredibly expensive and they were quite slow. So, they couldn’t possibly have been architected for this world, because no one really saw this complexity coming.”

When asked about Looker during a separate interview, Kenneth Cunanan, a Product Marketing & Analytics Manager at Looker responded:

“Databases in the past were very complicated, and they required a specialized team of engineers to troubleshoot and operate. And so, as a result, it was really hard to work with them at any sort of scale, that a business, especially a growing business, needs.” 

The rapid rise in database technology has allowed for more centralization in some cases, and with it has come a renewed appreciation for the value of a single source of trustworthy data. With the understanding that SQL is still the primary language of databases for most organizations, those practices are now better at helping professionals to optimize their decisions, processes, and systems. But, SQL for all its power has remained largely inaccessible to business users. Looker has taken the power of SQL and created an effective Data Analytics system that removes that final barrier to make that centralized data accessible to everyone.

Codd, ETL, and the Old World

In the opening of the webinar, Mintz asked, “How did we get here?” The answer is essentially, “By standing on the shoulders of giants.” In the 1970s, Edgar Codd created the relational database model, “To separate the machines that stored the data and processed the data, from the applications that actually ran on top of that data,” he said.

This allowed programmers to build applications without needing to know how the data was organized in the machine itself. Then Donald Chamberlin and Raymond Boyce built on Codd’s model, inventing Structured English Query Language, later called SQL, “a standardized language that looked a lot like the English language,” Mintz remarked.

It was SQL and the relational database model that was the foundation for long-standing Business Intelligence, and associated Data Warehouse/ETL, practices used by organizations for the past few decades.

The traditional way of doing Business Intelligence (and Data Warehousing) over the years was always ETL, which worked when data sources were manageable. But with the proliferation of sources and an ever-increasing volume of data, the ETL process simply can’t keep up anymore, Mintz said. He added that he used to wonder,

“When are we going to stop the madness of all this ETL? Of moving data from this system to that system, from that system to this system. Over the course of 10 or 20 years, in a large organization that has lots of data sources, ETL scripts become like spaghetti mayhem.”

Looker works with organizations to clear up that mayhem and move into the next wave of BI and Analytics, while providing a system grounded in the work of those original “giants.”

Shifting Paradigms: From Then Until Now

Mintz said he has focused his career on how people react to data and the best ways to use it in achieving their goals. Mintz has a passion for using Analytics to solve the world’s toughest problems. During the webinar, he described the changes that have taken place in the way data is dealt with. In the past due to the extensive costs of Database Management and Business Intelligence it was necessary:

“To essentially shrink your data because you couldn’t possibly deal with the data and all its complexity, all its size, and that was in a world of much smaller data than today. So of course, everything you did was about shrinking that data because you didn’t want to buy another server to put in your server farm because it would cost you a million dollars and it’d be trucked in on an 18-wheeler.”

That was the old world “that these things were architected for,” but it is a much different world than today with modern Data Lakes and other new technologies.

“Databases today are incredibly fast. They’re almost unthinkably fast, and they cost pennies.  Sometimes they cost less than pennies. Most of the big Cloud vendors will give you some free queries each month.”

The Hadoop revolution, along with the extreme lower costs of commodity hardware has changed everything. With data volumes continuing to grow exponentially, there needed to be new technologies that could keep up with that growth in data volumes, while also allowing for real-time solutions, advanced BI, self-service, and a host of other new paradigms.

That certainly all started with Hadoop,” said Mintz. “Around ten plus years ago or so, and continued through with Amazon Redshift and Google BigQuery, Snowflake, and Vertica.”

There are two essential transformational shifts that are driving the industry today said Cunanan. “The first is technological and second there’s a cultural shift.” Those older databases were optimized for writing data really quickly back to memory, that was their primary purpose. They weren’t optimized for analytics. The new generation of technologies have alleviated that first problem. The cultural shift has brought business users into the world of data; they now want access to the data assets within their organizations, and they want it immediately.

Looker has now stepped forward with a platform that allows developers to use their existing SQL skills within an environment that can handle both tasks with ease and efficiency not seen before. “We’ve created a platform that deals with both of these major shifts,” he said. Looker opens up the world of analytics to everyone in a business.

The Looker Solution

Describing Looker’s platform, Mintz said that it’s grounded in the idea Self-Service Data Analytics without having to be so technically proficient and with a “friendlier interface.” It also allows analysts and developers to go into the system and do their work. LookML further builds on this by making SQL modular and friendly so analysts can easily manage business logic:

“We make everything very readable, it is not a scary machine–generated SQL that’s impossible to parse, it is pretty straightforward and easy to understand.  One of the key concepts of Looker is LookML, and its reusability and extensibility.”

Now anyone can ask sophisticated questions of the data using familiar business terms. Mintz said:

“We’re doing it all via the SQL which is really nice. And it works for business users and developers. If someone is in developer mode they can make changes that only affect them. The reason why we ground in SQL is because we want to speak the language that data analysts are already speaking. We want to make that language version control friendly and dynamic.”

This ability to “interrogate what’s going on with the data,” to change it on the fly and maintain agility is “really critical to being able to sort of learn what’s going on with your data for real,” said Mintz. That is the core of what LookML allows organizations to accomplish.

Looker’s Data Platform also allows businesses to access real value from their data. It does this by creating an Agile Modeling Layer immediately above SQL processes.

“I love watching technical folks get it, because they do,” remarked Mintz. “But I also love seeing the business users getting it, because they do as well.” According to Mintz, there are really three major groups using their system: Analysts who speak SQL, but are really happy that Looker is doing so much of their work for them; the developers who still want to be deeply involved in SQL with queries and coding; and the non-technical data consumers who have so often only lived in Excel and now have dashboards that give them access to all of the data without having to “wait in line for the data results anymore.”

“Everyone now has a place where they can ask a question and get the data back instantly, then ask a follow–up question and get that data back as the follow–up question instantly. It’s expanding the understanding what the data means for everyone in an organization, I just see over and over how the discoveries they’re making are driving their businesses forward.”

Group Discussion Podcast

There is also a related DMRadio podcast, which had additional guests who expanded on the SQL revolution topic. The guest list included:  Ragu Thiagarajan from Datameer, Ramon Chen from Reltio, Mike Finley from AnswerRocket, Eric Mizell from Kinetica. The host was Eric Kavanagh, who is also CEO of the Bloor Group, and the analyst Mico Yuk, CEO and Founder of BI Brainz.

Photo Credit: Kalakruthi/Shutterstock.com