IBM Watson for Oncology

After IBM’s Watson system won the US quiz show Jeopardy!, the company set out to apply it to medicine, and cancer care became its most visible ambition. IBM launched its Watson Health business in April 2015 and, according to reports, spent around $4 billion on acquisitions to build it.¹

Two well-documented episodes show how far the reality fell short of the promise.

MD Anderson: $62 million and not ready for patients

In June 2012, the University of Texas MD Anderson Cancer Center began a project to build a Watson-based advisory system for cancer care. A 48-page audit by the University of Texas System, reported in February 2017, found that the project had cost more than $62 million.²

The audit described a contract with IBM for the first product that was meant to cost a fixed $2.4 million over six months. It was extended twelve times, and IBM’s fees reached $39.2 million. The project’s pilots had used the hospital’s older patient record system and had not been updated to work with its new one. The project ended in September 2016, and the audit stated that the system “is not ready for human investigational or clinical use, and its use in the treatment of patients is prohibited.”²

Watson for Oncology: “unsafe and incorrect” recommendations

IBM’s main oncology product was trained by IBM engineers working with doctors at Memorial Sloan Kettering Cancer Center. In July 2018, the health news site STAT reported on internal IBM slide presentations from June and July 2017, prepared by a Watson Health executive and shared with the division’s management.³

According to STAT, the documents cited “multiple examples of unsafe and incorrect treatment recommendations.” They largely blamed the way the system had been trained: on a small number of “synthetic”, hypothetical cancer cases rather than real patient data, so that its recommendations reflected the preferences of a few specialists for each type of cancer rather than “guidelines or evidence”.³

The end of the venture

In January 2022, IBM agreed to sell Watson Health’s healthcare data and analytics assets to the investment firm Francisco Partners. The terms were not disclosed. Earlier reports had suggested IBM was seeking around $1 billion.¹

Why it failed

The promise ran ahead of the evidence. Watson was marketed as a transformation in cancer care before there was clinical evidence that it improved decisions or outcomes. Hospitals bought into a vision, not a proven product.

Training on hypothetical cases embedded a narrow view. Synthetic cases are quick to create and easy to control. However, a system trained on how a few experts would treat imagined patients learns their preferences, not the variety of real patients, settings and treatment guidelines it will meet.

Integration was underestimated. At MD Anderson, a tool built around one patient record system could not simply move to another. The everyday work of connecting AI to clinical systems and workflows proved as hard as the AI itself.

Lessons for leaders

1. Ask for outcome evidence, not demonstrations. Before committing, ask a supplier what independent evidence shows that the system improves results in settings like yours, and treat the absence of evidence as an answer.
2. Ask what the system learned from. The data behind an AI system defines what it knows. If it was trained on synthetic or narrow data, expect it to struggle with your reality.
3. Put a ceiling on open-ended commitments. A six-month, fixed-price contract became a multi-year programme through repeated extensions. Set clear stage gates, and be prepared to stop at them.

Questions to ask in your organisation

1. For our largest AI supplier commitment, what independent evidence do we have that it works in an organisation like ours?
2. Do we know what data our suppliers’ AI systems were trained and tested on, and how closely it matches our own?
3. How many times has the scope, budget or timeline of our flagship AI programme been extended, and who decided each time?