You want big data? You can’t handle big data…

Lori Mac VittiePrincipal Evangelist

Current infrastructure cannot handle the projected explosion of data, connections, and traffic as the world moves into the digital economy.

We talk about the digital economy but we don’t often dive into the specifics of what that really means. It means paper trails and phone calls are being replaced by audit trails and e-mail. It means invoices and customer files are being phased out by e-receipts and database records.

Our migration to digital data will one day be viewed with the same historic impact as the move from stone tablets to papyrus. With each evolution in the methods we, as human beings, use to record data, we’ve made it cheaper, faster, and more efficient. Which means we’re not only creating but stockpiling data at rates never before seen in human history.

There is no end in sight. The first signs of the digital economy appeared when we could capture in those oh-so-efficient 1s and 0s massive amounts of data. But it was still mostly manual. Data entry and log files provided us with the bulk of the data collected in the early days of the Internet.

Now we know how to collect data automatically. With tiny transistors and robotic assistance, we’ve compressed the computing power of entire buildings down to the size of a pinhead. And we put it in everything. Light bulbs. Watches. Motion sensors. Cameras. Water filters. Cars. Refrigerators. Watches. Phones. Teddy bears. Toys.

If there is something that isn’t Internet-enabled and thus part of the growing Internet of Things, I’m not sure what it is. And yes, I’ve heard about that adult toy, too. No need to share, this is a G-rated blog, after all.

But existence doesn’t mean usage, and we’re just touching the edge of the explosive growth of things. That means over the next few years there will be more devices on the Internet. Many, many more devices, all transmitting and receiving far more data than the Internet has ever seen. Industry projections estimate that there will be 50B connected devices worldwide by 2020 – or 7 devices for every person on earth.

I can count three of mine from where I’m sitting right now. And that doesn’t count the rest of the house. Here’s some handy statistics, in case you still aren’t convinced by a glance around your own desk.

Cars alone are getting connected to cellular networks faster than anything else, according to statistics compiled by Chetan Sharma Consulting for the second quarter of this year. Counting all U.S. carriers, about 1.4 million cars got connected to cellular networks in the quarter, compared with 1.2 million phones and less than 900,000 tablets.

By eliminating manned travel of pallet trucks distribution centers, Giant Eagle increased productivitybetween 10 percent and 30 percent. Each vehicle has sensors to gather data. And all of the data—such as the size of the load picked up, how long it takes to deliver the load, where the vehicle is in the facility, vehicle downtime—are uploaded to the Seegrid Supervisor application.

Chick fil A instituted temp sensors for their freezers and refrigerators to ensure food safety. Previous methods of monitoring were manual, and required visual checks of each thermometer, a process that had considerable risk when restaurants were closed and no one was watching. Now sensors monitor round-the-clock and report in hourly, potentially saving the estimated $15,000 to $20,000 in inventory replacement costs should a cooler malfunction go unnoticed.

I have many more examples, but at this point, it’s pretty clear that bigdata is being generated, transmitted, and stored every second of every day across the globe. And a significant amount of it is not consumer-oriented. It’s what the industry calls ‘industrial’ and it’s geared toward monitoring operations and processes more than anything else. There are millions of sensors everywhere today, and each is generating data.

Lots of data.

There are background updates as things report in, check for new instructions, and exchange collected data. Each thing has an app to which it chats on a daily basis. That app is somewhere, out there, in a data center or in the cloud. No matter where it might reside, the important thing is that, well, it exists. Because there is infrastructure and services in between the thing and that app, and it isn’t ready for the explosive growth.

Don’t take just my word for it. A Forbes article cited a a Bain & Company survey of 325 global companies and found that “59% believe they lack the capabilities to generate meaningful business insights from their data. In another Bain survey of 250 companies, 85% said they will require substantial investments to update their existing data platform, which includes consolidating and cleaning data, simplifying access and rights management, and improving access to external data sources.” [emphasis mine]

At the core of this explosion of data (that’s “big data” for those of you playing buzzword bingo today. You’re welcome) is, of course, the data platform. That’s the database, and there’s a ton of commentary and expert analysis for and against NoSQL (non-relational) databases like MongoDB and Heroku as well as those for an against SQL (relational) databases, like Oracle and MySQL and Microsoft.

But what is often ignored in discussions about bigdata and dashboards and operational insight gathering is that there is a whole lot of infrastructure underpinning the scale of and access to those data sources. There are few line of business owners or C-level executives who are hunched over a keyboard writing their own queries to extract value from the data. That’s done by apps, too.

If there is one truism about the digital economy it is likely this: there is an app somewhere responsible for gating access to data.

And that app needs infrastructure. It needs security, it needs scale, and in a world where things can’t tell time, it needs round-the-clock availability.

Today’s infrastructure was built for the web. It was built for people, really, and apps. Both are still critical, but now we have the added burden of simultaneously dealing with the scale, security, and availability requirements of things. We need to re-evaluate what that means to our networks, and our infrastructure, and to the operational processes that scale and secure the apps and devices that reside in between users and apps. And by users I mean things and systems, too.

We need to rethink how we manage authentication and authorization. How we prioritize and process requests and responses. And how we secure our most critical business asset (that’s data) from being exposed to the wrong person (or thing).

If we’re going to handle “bigdata” and all it entails, we have some serious work to do.

About the Author

Lori Mac VittiePrincipal Evangelist

More blogs by Lori Mac Vittie