What is Big Data?

There’s a quote near the top of Philipp Janert’s instant classic Data Analysis Using Open Source Tools: [I]t seems that for many people in the tech field, ‘data’ has become nearly synonymous with ‘Big Data.’ That kind of development usually indicates a fad. The reality is that, in practice, many data sets are ‘small,’ and in particular many relevant data sets are small. He was writing in 2010 — and, oh, what a fast five years it’s been. Janert goes on to say that classical statistics was built to perform inductive operations – start with a subset of a mess of information and draw conclusions about the mess. Big Data puts the mess in our midst, which is a mixed blessing. As Janert says: “Big Data makes it easy to forget the basics.” But there’s no avoiding it now:  the fad is not fading. The rush of outrage greeting my recent summary post about AdWeek in NYC titled “’Big Data Is a Big Distraction’: Notes from #AdWeekXII” put me on notice. Never mind that I was quoting someone else (i.e., not myself) and was simply reporting the ad industry’s reaction against last year’s Big Data hysteria – a reaction against hype and not substance. Let us admit to ourselves the obvious: We need to walk into the light, amigos. Big Data is a big reality. So what do marketers need to know about it? What follows is a primer on the topic for the interested beginner. It’s based on a recent research report I published called “Understand Big Data Basics for Marketing” (Gartner subscribers enjoy here). It’s not – I mean not – for the white-coated, square-eyed crowd down there in the clean room. So: Big Data Basics for Marketers. What is Big Data? Let’s keep it simple. Big Data is data that is so big it won’t fit on a single machine. It has to be spread over many machines. And it can come from anywhere, so it might be in strange and exotic formats. And it’s coming fast. These ideas of size, road speed and formats are captured

Continue reading