Big Data is the breaking news story in the IT world; here's the picture - sometime soon company managers will have an enormous amount of information at their fingertips, they will be able to see everything and optimize everything.

What could be better?

Within the past month, two very senior, astute individuals contacted me about this – one a senior IT industry analyst, and the other a senior editor of a major business publication. They both had the same question: If Big Data actually becomes available, what will be the consequences?

My answer – Big Data offers big opportunities, but it carries the very strong likelihood of creating really big headaches in three areas: (1) paving the cowpaths, (2) managing at the right level, and (3) driving without a roadmap.

Let me explain.

Paving the cowpaths

Remember RFID?

A few years ago, RFID was all the rage. The idea was that it was becoming technically and economically possible to affix an RFID tag to every item moving through a supply chain. (An RFID tag is a small passive label that essentially emits a precise identifier when it is hit by an electromagnetic field.) Armed with this capability, managers could know the identification and location of every item in their company, and even those flowing into and out of their companies.

I remember discussing this with a former student, who is a senior operating executive of a Fortune 20 company. His reaction was similar to mine: What ever would you do with all that information?

In fact, several years ago I co-authored a column, Are You Aiming Too Low with RFID? in Harvard Business School's Working Knowledge. In this piece, I joined with Sanjay Sarma, co-founder of MIT's Auto-ID Lab, and John Wass, CEO of WaveMark, to argue that the biggest danger with the flood of RFID information was that almost-irresistible temptation to focus on "paving the cowpaths."

Here's how we put it. "One of the most exquisite challenges of living in Boston is navigating the labyrinthine maze of streets in the downtown area. This part of town is the oldest part, and the streets follow the original paths formed by settlers driving their cows to pasture. Traffic flows poorly because the city fathers simply paved the cowpaths, making the ineffective more efficient. It's much easier to navigate Back Bay, a part of Boston with grid-like streets, built on landfill centuries later."

In the article, we outlined a number of highly focused, high-value analytical uses for the "Big Data" that RFID could produce, and counseled avoiding the large-scale applications that simply automated routine activities.

In the absence of this disciplined, strategic approach, managers are in grave danger of utilizing Big Data to pave the cowpaths, further entrenching existing practices and rendering the possibility of developing sweeping paradigm-changing initiatives more and more difficult.

This will almost inevitably occur because in the capital budgeting process, the tactical payoffs from paving the cowpaths will be clear and easy to measure, while the payoffs from far-reaching strategic changes in the business will be hazy and unmeasurable.

Managing at the right level

This past weekend, I was reminded again of the Big Data question when I re-read Wired for War, a terrific book by P.W. Singer which traces the robotics revolution and the use of robots in  twenty-first century conflict.

In a particularly telling chapter, Singer describes how the real-time video feeds from drone aircraft – Big Data – led to the systematic leadership problems that I call "managing at the wrong level."

Over many years, improved communications technology has enabled commanders to command increasingly at a distance from the actual battles. This has led to a very effective management structure in which top commanders focus on strategy and personnel, mid-level commanders on operational initiatives, and local officers on tactical issues. This parallels the leadership structure of most effective companies.

However, the widespread availability of drone aircraft information feeds has led to serious and systematic command and leadership problems. The ability of top commanders to see battlefield video feeds in real time has rapidly increased the centralization of command and led to an explosion of micromanagement.

Crack for Generals

Singer relates, "Too frequently, generals at a distance are now using information technology to interpose themselves into matters that used to be handled by those on the scene and at ranks far below them. One battalion commander in Iraq told how he had twelve stars worth of generals (a four-star general, two three-star lieutenant generals, and a two-star major general) tell him where to position his units during a battle."

Singer continues, "An army special operations forces captain even had a brigadier general (four levels of command up) radio him while his team was in the midst of hunting down an Iraqi insurgent who had escaped during a raid. The general, watching a live Predator video back at the command center…ordered the captain where to deploy not merely his units, but his individual soldiers. ‘It's like crack for generals,' says Chuck Kamps, a professor at the Air Command and Staff College. ‘It gives them unprecedented ability to meddle in mission commanders' jobs.'"

This direct meddling by military leaders has led to the rise of what Singer calls the "tactical general," as the line between timely supervision and micromanagement has became blurred. Officers in the field lament what they call the "Mother may I?" syndrome which has come with these new technologies.

Moreover, power struggles are common when the feeds are available to multiple command groups.  Singer notes, "At its worst, this pattern can lead to the battlefield versions of too many cooks spoiling the meal. A marine officer recalls, for example, that during an operation in Afghanistan, he was sent wildly diverging orders by three different senior commanders. One told him to seize a town fifty miles away. Another told him to seize the roadway just outside of town. And the third told him, ‘Don't do anything beyond patrol five miles around the base.'"

But, the biggest problem with top-level micromanagement in the military – just like in business – is the huge hidden opportunity cost of failing to manage at the right level: a leader ignoring the critical issues of high-level strategy and organizational capability because he or she is so caught up in real-time micromanagement. This causes two very big, related problems.

First, the top managers fail to plan for the future. For example, in business, vice presidents should primarily be focused on defining and developing the company as it should be in three to five years, since that is the time it takes to develop a new set of capabilities. Their other critical responsibility: coaching and developing the next generation of leaders.

In the absence of this hierarchical discipline, the company is in grave danger of getting mired in the present, and falling further and further behind.

Second, when top managers – or generals – take over tactical decisions, the lower-level leaders cannot develop their skills. Instead, they must be empowered to act with initiative, even if it means making a few mistakes along the way. No – especially since it means making a few mistakes along the way, since false starts and errors are a natural and necessary part of doing anything significant and new.

The answer? Singer calls it "enlightened control," a concept he credits to the great Prussian generals of the nineteenth century, whose ideal was that the best general gave his officers the objective and left it to them to figure out how best to achieve it. He cites the commanding general who so trusted his officers that the only order he supposedly issued on the eve of the Prussian invasion of the Danish province of Schleswig was, "On February 1st, I want to sleep in Schleswig."

The action question for managers: Will Big Data be "crack for your vice presidents," or will they have the insight and discipline to double down on "enlightened control."

Driving without a roadmap

One might ask: But won't Big Data let a company's managers optimize everything? After all, every manager will have, theoretically, the information needed to get everything right. And if a company's managers optimize everything, won't the company be great?

This question embodies one of the biggest false assumptions in thinking about Big Data.

The glib answer is that if it weren't for the humans, this premise might actually happen. Let me explain.

When I think of Big Data, an analogy comes to mind. Imagine that you were living decades ago, at the time of the invention of the automobile. Assume further that all of a sudden, all the dirt roads and tracks were paved. What would you do? Where would you go?

The obvious answer is that either you would stay local or you would be paralyzed in the face of the enormous number of opportunities. In fact, you would need a roadmap, so you could see how to get to different places. Beyond that, you would need to understand the nature of the destinations so you could decide where to go, since you couldn't get everywhere in one lifetime.

Further, if you had a number of different drivers, it is likely that each would head in a different direction, since each would go after the goal that he or she thought best. If these drivers had to coordinate, like the managers in a company, what would happen? The result would be chaos.

The problem here is two-fold. First, a company can't do everything, because it takes significant time and resources to manage the change required to harvest any IT-based initiative. And second, the initiatives have to be coordinated and focused on the right long-term strategic goals to be effective. If the availability of Big Data encourages a massive flock of independent tactical initiatives, it will do more harm than good.

The problem with low-hanging fruit

This raises an important related problem. Managers have an almost overwhelming tendency to focus first on opportunities that are near to hand, and have quick, visible payoff. Sometimes these are called "low-hanging fruit."

The problem is that these relatively small, parochial projects will absorb the organization's resources and capability to change, even while they give the illusion of progress. The huge opportunity cost is losing the opportunity and ability to focus on the really important initiatives with the really big long-term payoffs.

Think of it this way. The analogy breaks down because the big money is not in harvesting fruit more efficiently, but rather in changing the location of the orchard and the type of trees you plant.

This dissipation of effort, with its focus on a large number of small, incremental projects – rather than on the smallest number of game-changing, high-payoff initiatives – is the ultimate danger of Big Data.

Keys to Success

Big Data offers great opportunities – and huge dangers. How can you navigate toward the benefits while avoiding the hazards?

The key is management insight and discipline.

The true promise of Big Data is to make your company better, not just to make parts of it more efficient. To accomplish this, you need one part technology to nine parts vision and great management.

Jonathan L.S. Byrnes, Senior Lecturer at MIT, is an acknowledged authority on profitability management. His extensive experience spans virtually every industry, including healthcare, transportation, software, retail, financial services, distribution, and others.

www.islandsofprofitbook.com