Small and Wide Data is Important and Relevant: Is the Era of Big Data Coming to an End?

Whether reading Greek philosophy or listening to songs on the radio, we’re often reminded that the only thing that stays the same is that everything changes. In the realm of research and analytics, one of the most important changes currently influencing individuals, corporations and even politics is a shift in focus away from the concept and capabilities of big data.

If the era of big data is ending, what is taking the place of such a powerful and influential practice? With computing power increasing exponentially and advances across the information technology space, one might think that the next evolution would be bigGER Data!

Somewhat surprisingly, industry leaders believe that the opposite is true. According to a May 2021 research report from a leading technology advisory firm Gartner, small and wide data is what the top academics and analysts focus on for the future. Gartner experts say that 70% of Organizations Will Shift Their Focus From Big to Small and Wide Data By 2025.

In this article, we will first introduce the concept of small and wide data, together with some real-life examples of how practitioners are using them in the market. Let’s also dig into how small and wide data differs from big data and why this change is expected to remain long into the future.

It’s clear that this trend away from big data and towards small and wide data is more and more critical for companies to understand, and we’ll give you the top reasons why.

What is Small and Wide Data, and How is Big Data Different?

Old Big Data

Sometimes it’s easiest to start with what is already familiar — big data. Ever since computer hardware and software became sufficiently powerful to deal with unimaginably huge data sets, scientists and mathematicians have run meaningful, academic statistical analyses, and investors have embraced “more, faster and better quality” data.

As the internet age continued to produce vast reams of data, such as the seemingly infinite amounts gathered and stored by technology companies operating search engines (like Google), social media platforms (like Facebook) and computerized financial exchanges, the appeal of big data grew and grew.

Key characteristics of big data, such as volume, frequency, and variety, lead to its use being somewhat limited to building bigger picture ideas. It’s great for visualizing a particular market trend or understanding the distribution pattern of its components. In other words, big data is an excellent way to figure out whether you are looking at a tree or a building.

Suppose you are an AI developer. In that case, big data can show you the percentage of companies using AI In their market tech stack or the ratio of companies using it for generating leads. It becomes quite a valuable piece of information to understand the level to which the market is interested in AI-based software, doesn’t it?

Small and Wide Data. What is it?

In contrast, small and wide data is better at picking out more specific information and distinct insights from individual data components and drawing valuable comparisons. To use our tree/building example, it’s more about looking at the leaves on the tree or focusing on a particular room in a building as a means of understanding not only what the thing is, but how it works and why it’s there in the first place. But this is just a vague explanation to get us rolling. Let’s look deeper.

Wide Data

Wide data allows the analyst to examine and combine a variety of small and large, unstructured and structured data. In comparison, small data is focused on applying analytical techniques that look for useful information within small, individual sets of data.

Specifically, wide data is all about tying together disparate data sources across a wide range of sources to come up with meaningful analysis. Consider this example from a systematic trading strategy. Based on data gathered from asset price movements, asset valuation factors etc., a systematic investment strategy can be created. That is, just a few simple factors across a wide range of data. To simplify, the best trading strategy, based on such a wide data analysis, could be to buy assets that have experienced a rising market price AND have low fundamental valuation AND intending to hold such assets for a 1-3 month period.

A real-life example of the usefulness of wide data comes from the Target department store in the U.S. They looked at customer purchases across their stores alongside big data behavioral analytics that showed the likelihood of consumers who buy a certain subset of products to purchase another basket of goods soon after. Wide data showed them the spending habits of families expecting a baby so that they were able to build an effective marketing strategy around it.

Specifically, they noticed that customers who stocked up on such items as bulk cotton buds, unperfumed soaps, and lotions were highly likely to become buyers of diapers, infant clothing, and cleaning products a few months down the line. They initiated a successful email strategy promoting such products to the previously defined subset of existing clients. Very cool!

Small Data

Small data, rather, concerns collecting and analyzing data sets sourced within individual organizations or based on individual problem-solving examples. Simply put, small data can be thought of as the opposite of big data, and therefore small data is not captured or easily extracted in a helpful form from big data sets.

Let’s look at the famous “dirty sneakers” insight that LEGO executives realized could enhance their customers’ user experience —small details matter. When a customer informed them that he got a lot of satisfaction from knowing that the wear and tear of his favorite skateboarding shoes indicated that he was a regular and proficient skateboarder, LEGO concluded that customers value small and specific details.

As a result, they changed the trajectory of their production from larger, combined LEGO pieces to smaller, individual and more detailed components. Customers rewarded this change and reported higher satisfaction with their products.

How is Small and Wide Data Better?

To draw a simpler conclusion, strategic objectives outlined from wide data should directly benefit the organizational decision-making process. Notably, such findings are often missed if relying solely on big data analytics. Small data is more human, company-specific and can be effectively leveraged to affect valuable decision making.

Again, using our tree/building example, it is essential to consider the information available from looking at the different rooms, utilities, etc. Additionally, by combining such insights with another data source, such as describing different types of buildings, the what, how and why of a building becomes more precise and meaningful.

Data analytics, including those related to AI development, need to rely on data sourced recently and in smaller amounts. Additionally, large-scale data collection typically associated with big data approaches, including collecting vast amounts of data for analytical purposes, is challenging for many organizations.

Even if big data is available, the costs, time and energy to implement conventional supervised machine learning can still be tricky or impossible. In addition, decision-making by humans and AI has become more complex and demanding, requiring a greater variety of data for better situational awareness.

It is clear that there is a trend towards the benefits of small and wide data that rely on effective data analysis of company-available data. Reducing the volume of data is required to implement meaningful, result-oriented strategies or recognize and target the value inherent in less structured and diverse data sources.

Such a shift in direction from big to small and wide data affords individual companies the ability to recognize a tree from a building and act strategically based on such research-based initiatives. Such practices will be based on objective data-driven analysis rather than theoretical or more soft-factor research approaches often relied upon by project, product, and sales and marketing teams.

Conditions change, and information becomes less reliable or obsolete. While big data allowed companies to collect and store massive amounts of data, many of the end-users of such data sets have already found such content unreliable and ofdecreasing value for implementing their strategic goals.

What are Big Data’s Main Constraints?

Now, let’s look at two critical factors disrupting the data analytics industry that relies heavily on big data.

The Regulatory Environment

In response to privacy concerns, there is an increasingly negative stigma surrounding the use of big data. It is now more than ever associated with invasive marketing techniques and disruptive advertising.

A perfect example of this is the recent high-profile case of Facebook and Cambridge Analytics, as documented in the Netflix documentary “The Great Hack”. Ultimately, the result of such concerns and criticism has resulted in three American states introducing new Data Privacy legislation prohibiting the use of more private data sources, such as Facebook accounts, for sales and marketing purposes. In this particular example, the companies involved took big data a step too far — to influence election results.

On a more consumer-specific scale, this ties in with the privacy “invasions” that individuals are starting to accept less and less, such as targeted online advertising and direct email marketing campaigns, for example. The key outcome is that organizations now face legal hurdles that make big data less effective and efficient.

Big Data Anomalies

Just as retailers often exclude such extreme data patterns as Black Friday sales numbers, or tweak investment strategies to ignore Black Friday-type market data, big data can be crippled by events that skew their data set. Recently, the global Covid pandemic has skewed and warped data sources so much so that they can make big data analytics meaningless.

For example, nobody could have predicted such a strong trend towards online sales for companies like Amazon shopping or Deliveroo fast food delivery, and the likelihood that it would influence sales and marketing so acutely. As any data analyst knows — data in, conclusions out. Data sets corrupted so severely have been exposed, and their predictive potential has been limited.

So while adjusting big data analytical techniques to account for such crazy events like the Covid pandemic can be crammed into data analytics, small and wide data-based approaches offer a more pragmatic solution for analysts and programmers.

How Can Businesses Benefit From Small and Wide Data?

We have yet to mention that working with big data incurs a few costs that can be a significant hurdle for even large, multinational companies. So if your budget is limited for any reason, acquiring big data sets and exploring how they may work for you might just be out of the question. Aside from paying for the data itself, working with it is quite tricky and especially expensive if you lack the expertise internally.

Luckily, the trend towards small and wide data works in your favor! Aside from what we’ve already discussed, here we summarise a bunch of benefits that working with small and wide data can bring to your organization.

Stay More Personal

While big data shows big picture trends and correlations and is very general, small data focuses on what drives each customer, prospective client and even your employees. Instead of relying on their gut instinct or hoping that big data will apply to your company specifically, you use this data to adapt your efforts to the requirements of individuals. For example, here are some ways you can get more personal with your clients using small and wide data:

• identify key segments

• communicate personalized offers

• tailor marketing campaigns for each segment or customer

• know the best channels to engage with them

Get Real-Time Insights

Big data needs time and resources to process. By the time you get it, it’s already history. And if valuable insights aren’t quickly and easily discovered, it’s unlikely to add value anytime soon. In contrast, small data is always at your fingertips, allowing fast or even real-time decision-making possible. Here’s what you can do with it:

• understand triggers that make your clients buy

• improve the process of lead generation

• change the way you market your products

• adjust your marketing strategies in real-time

Manage It Easily

Small data is simple, manageable, easy to get, and can be worked in real-time. Even paying close attention to specific fields in your CRM system equates to small data that you can access and action. So with small data, you can:

• respond to customer actions proactively

• do without a team of experts to work on huge data sets

Wrapping Up

The era of big data provided researchers, companies, industries and even entire countries with important information that will continue to affect decision-making and understanding well into the future. But regarding its importance for organizations who think actively about applying data analytics to the operations, products, projections, and so on, its limitations are starting to show.

Luckily, small and wide data has also proven its effectiveness in dealing with similar problem solving and decision-making applications. And the fact that it is more specific and manageable makes it a great compliment to a wide variety of information, including things that big data has taught us.

Don’t get overwhelmed by technical terms and data analytics technology. You’ll be surprised what’s sitting around unused in your tech stack. Go for it!

More useful content on our social media: