By now we have all heard how powerful Big Data can be for retailers. Drawing together product information, inventory levels, customer profiles, buying patterns, and even weather forecasts, retailers can use predictive analysis to make better decisions.
However, the size, variety and complexity of these giant data haystacks is increasing everyday, creating technological challenges that only the leading analytics platforms are keeping up with.
Databases are key to filing data in ways that can be easily accessed for a variety of analytical demands. Traditional, “relational databases” are organized in a simple table format with columns and rows. Each column is an attribute of that type of entity and each row is a distinct entity. An entity might be a SKU, or a vehicle, or a shopper for example. One column contains “keys” that uniquely identify each entity in the list, like a model number, license plate or email address. With this intuitively simple organization, these ‘relational databases’ allow retailers to analyze the data using straightforward, if inefficient algorithms.
Data starts to become Big Data
But in the years since the relational database was conceived, the quantity of data has increased by perhaps a million times. The first gigabyte hard drive appeared in the 1990s. Now consumer analytics firms have databases over 1 million times that size, measured in petabytes, with data on every man, woman and child who has ever used a credit card.
In addition to hundreds of demographic and behavior metrics for hundreds of millions of customers, a key driver of this growth is the amount of real-time data coming from the Internet of Things: real-time inventory levels, moving item locations, customer traffic patterns, sales conversion ratios, environmental Info, smartphone app interactions, video data – even the weather – the list is endless.
As the information volume increases, the number of sources grows and the structure of the data becomes more fluid and changeable. The cost of storing this content with traditional databases rises faster than the amount of files and quickly becomes unaffordable and unmanageable – as the IT team says, “it’s not scalable.” What’s needed? A new technology.
Say “hello” to NoSQL
Along comes the so-called NoSQL, whose name derives from the reference to the structured query language (SQL) used with traditional databases. NoSQL removed the ordered rows and columns of tables, giving each item its own container with searchable attributes. NoSQL creates a giant digital haystack of “unstructured data” containing needles of actionable intelligence. Unlike relational databases, whose cost growth outpaces the growth of data, NoSQL scales linearly with the amount of data, can be expanded dynamically without a major redesign of the system, and is much faster when the dataset is large.
While NoSQL ensures availability, speed, and flexibility, large datasets need sophisticated IoT platforms such as Mojix’s ViZix to be useful. ViZix consumes massive streams from the cloud, performing real-time complex event processing on the live streams, and then stores it in the NoSQL. From there ViZix enables users to analyze the historical data, performing description, diagnosis and prediction for human users.
ViZix helps users make sense of the data stored in the NoSQL database. If this data, or “Needle” is the haystack, then ViZix provides the magnet.
