Give Technical Experts a Role in Defining Project Success

Below the waters of Stockholm Harbor lie the ghostly ruins of the 17th century battleship Vasa. Commissioned by Swedish King Gustavus Adolphus, the ship was a technological and military marvel in its day, boasting two gun decks with a total of 48 24-pound shot cannons — a major improvement over ships that typically had a single gun deck with 12-pound shot cannons. Its arrival threatened to upset the balance of power in the Baltic Sea and deliver control of the lucrative trade routes of northern Europe.

The vessel’s actual military service was embarrassingly short-lived, however. On its maiden voyage on Aug. 10, 1628, a gentle harbor breeze capsized the vaunted ship just a few hundred meters from its Stockholm dock, in full view of the crowds of cheering Swedes who had come to see its grand debut.

The loss of the ship was both a financial catastrophe and a political embarrassment. When the king’s privy council convened a full inquest, the shipbuilders pleaded innocence — they had built the ship to specifications approved by the king himself. How could they possibly be at fault?

Based on my experience working with hundreds of companies, the same problems that plagued Vasa’s construction — suboptimal communication between technical and business teams and the lack of joint ownership of outcomes — are common when companies try to build their Big Data capability. To overcome these issues, leaders need to absorb three key lessons about how to manage the inherent tensions between defining technical requirements and achieving valuable business outcomes.

Let technical experts help define success. Business leaders need to understand how to communicate with technical managers in a way that clearly conveys business needs but doesn’t impose technical judgments that they are not qualified to make. This can be tricky, because apparent business requirements can have hidden technical requirements that are difficult to identify. The Swedish king thought of the second deck on his battleship as being a military feature and did not consider the structural instability that the higher gun deck created. To avoid issuing impossible orders, business leaders need to get buy-in from their technical experts while establishing business goals in the first place.

Align compensation with outcomes. Many subject-matter experts have non-variable pay. When there are performance bonuses, those are more likely to be tied to peer-reviewed technical achievements than business ones. This perpetuates a technical culture focused more on technical virtuosity than on delivering business value. The onus is on business managers, as “principals,” to create compensation packages that incentivize technical experts, their “agents,” to prioritize the business interests of the company over the desire to achieve technological breakthroughs solely in the name of advancing technology.

Balance free speech and power dynamics. These challenges become more complicated when a technical manager is subordinate to a business manager. It is the responsibility of business managers to create an environment and culture where their employees feel safe communicating their concerns, especially complex technical ones that aren’t necessarily intuitive for nontechnical experts and that often defy pithy explanations.

For example, managers often directly suggest what kinds of features to use in a machine-learning model, rather than allowing the data to speak for itself. While suggestions are extremely useful, using features that have no signal can significantly degrade model performance through the introduction of spurious correlations. Experienced data scientists know that proper, data-driven (not boss-driven) feature selection is one of the most important aspects of model prediction.

Unfortunately, navigating power dynamics is one of the hardest areas for managers. A recent study found that over half of all employees wished their bosses either listened or communicated better. It can be difficult to raise an issue with a busy boss, even harder still when you need his or her undivided time and attention to really explain the problem. In the case of the Vasa, it certainly did not help that “the boss” was a formidable monarch with the intimidating nickname of “The Lion of the North” because of his battlefield exploits and tactical genius. Managers need to listen to the concerns of their employees and remember back to when they needed to raise an issue with a leader and felt the (perceived, if not actual) threat to their own nascent careers.

We live in an era that’s much more technologically advanced than the 17th century, and yet many of the lessons from that time are just as relevant today. Both subject-matter experts and general managers have a role to play as agents and principals in ensuring the success of complex technical projects. Subject-matter experts need to take ownership of business success in addition to technical success, perhaps accompanied by a shift in compensation structure. General managers need to bring technical stakeholders into the decision-making process, leveraging their expertise and giving them genuine ownership of critical business decisions.