The story of data: How did we get here?

We field stories every day about data. From new data platforms to deeper levels of insight fueled by artificial intelligence to the inevitable privacy breaches, data is in the forefront of martech news — and beyond. In fact, it was difficult to complete this series because the story of data keeps evolving every single day.

One thing often left out of all those conversations? That when we talk about data, data points and datasets, we are talking about human beings. We are talking about their qualities, their preferences and in many cases, what they do nearly every minute of their day. And in some situations, we’re talking about them either without their permission or with dubious permission.

It won’t surprise anyone to know that we have more data at our fingertips now than any other point in history. So even though the news continues to change rapidly (see Facebook’s recent debacle with Cambridge Analytica for one example), we decided to share the story of data … so far.

We hope it provokes a lively conversation — one that will continue for a long while to come.

Data, data and more data. Today there is more data on more people than ever before. IBM has estimated that each day we create 2.5 quintillion bytes of data (that’s 1,000 to the power of 6: a lot). But how did we get here?

Data collection started with commerce

Companies have always collected data on their customers. Merchants kept track of who bought what when and used that data to inform how they would sell. First, it was by observation, simply noticing details about a customer, such as when they shopped and what they bought.

Businesses would take the information gleaned from observations and execute their marketing plans accordingly. But, though they could make correlations between their ads and customers’ in-store behavior, it was still rather speculative. Once it was possible to keep track electronically, business owners started to collect specific information about their consumers and make more informed predictions. Data-driven marketing was on its way.

Companies started talking directly to customers

Omer Artun, CEO of customer data platform AgilOne, says that data collection moved into high gear with the advent of direct marketing.

“The starting point was direct marketing many decades ago,” Artun said. “This was the original context/platform for data-driven marketing. It was persona-based and utilized customer transactions for mainly direct mail purposes. The data was only looking backward and only collected when a sale happened.”

Julia Stead, VP of marketing at voice marketing cloud Invoca, agrees.

“It began in the 1980s with direct marketers wanting to take their businesses to the next level with data-based personalization,” Stead said. “The earliest direct mail campaigns were sometimes inartful, if not spammy. Thankfully, the internet and mobile phones have advanced the practice in recent years, and we are truly starting to realize the power of understanding customers and creating personalized experiences for them.”

Digital advertising opens the floodgates

Once the internet became popular in the 1990s, so did online advertising. Whereas conversions were historically considered in-store sales, suddenly companies considered a wide range of desired actions to be conversions — including a click to learn more, downloading a coupon or buying online. Marketers could see how many people clicked on their ads and tie in-store behavior more closely to that activity. Once customers went online, advertisers could use cookies to gather a wealth of additional data, including the amount of time they spent looking at certain products, where they were previously and if they returned.

Karl Van den Bergh, chief marketing officer at translytical data platform DataStax, explains.

“Companies began collecting all sorts of data because they quite frankly didn’t know what may end up being useful — or when or how!” Van den Bergh said. “Many went overboard in collecting data that has since proved meaningless to improve customer experience, expand markets or offer new services. Others disappointed (and angered) their customers by poorly protecting that data and letting it into the hands of criminals via data breaches. Companies also collected data from a variety of inputs and sources: so what was captured at a point of sale system may not integrate with web or social data — or even old-school data from direct mail.”

Big data gets, well, bigger

As the number of data points an advertiser could collect on an individual customer multiplied, the total amount of consumer data swelled. Marketers started referring to this massive dataset as “big data,” a buzzword that not only fueled the frenzy for more collection but sparked the creation of countless companies. Many of these companies would help businesses collect and analyze data and would build models to predict future behavior. Eventually, these companies started to implement machine learning algorithms into the process, building massive predictive graphs which marketers would then use to pinpoint customer behaviors and personalize marketing in ways never seen before.

AgilOne’s Artun says that companies use data to keep pace.

“In the digital era, we have data before a sale is made, therefore giving marketers a chance to influence the outcome. This data is very large (browse, call center, email data is 100x bigger than transaction data), very transient (a few hours or days later, it become almost obsolete), very diverse (not just products, but behavior, reviews, preferences, surveys, call center tickets, etc.). The rich information in the new datasets the digital era opened up requires not only timely processing but also analytical rigor to find patterns in the data — hence the rise of predictive/machine learning approaches,” Artun said.

How much is too much?

In 2018, consumers are simultaneously offering up more data than ever before and expressing more vocal concern about how it’s handled. There were a record number of major data breaches last year, and new, more restrictive data privacy and usage laws are on their way, including the European Union’s General Data Protection Regulation (GDPR). Companies are sharpening their focus on how the data is used, rather than how much they can collect.

Mike Herrick, SVP of product and engineering at Urban Airship, says that companies need to show customers that they can be trusted with their data.

“Digital made data currency,” Herrick said. “Its speed enabled marketers to go beyond a view of historical performance to affecting the future. So businesses and every one of the 5,000 martech solutions out there started collecting data, some without apparent purpose, appropriate transparency or user consent. The response has been a rise in ad blocking, and growing data and privacy regulations, while machine learning has continued to advance, fed by this data fuel. Today, for all of these reasons, it’s more important than ever for brands to earn the right to utilize customer data to improve and enhance brands’ interactions with customers.”

So where are we now? Simple answer: sitting on a ton of customer data. Companies are emerging every day with new ways to collect, analyze and use this data — as well as new, comprehensive privacy laws, such as GDPR, which put restrictions on how we do that.

Here are links to the entire The Story of Data series:

Part 2: Where are we now?
Part 3: Who owns it?
Part 4: Will it ever be truly secure?
Part 5: Where are we going?