Miracle clinic
The tonic is water. So why does everyone get better?
You run a small clinic. Twelve people walk in with bad pain, take your tonic, and come back a week later. Then you sit on a fund board that hires and fires managers. Each time you say what did the work.
Why the water seemed to work
People do not go to a healer on an average day. They go on a bad one. Pain swings up and down round each person's usual level, so a bad day is usually followed by a better one, with or without help. Statisticians call this regression to the mean. Add the hope that comes with any treatment, and almost anything you hand over will look like it worked.
In 1799 John Haygarth, a doctor in Bath, tested a fashion of the day: Perkins tractors, a pair of metal rods sold for five guineas a set that were drawn over the skin to pull out pain. He made fake ones from wood and painted them to look like metal. He drew them over five patients with chronic rheumatism. Four said their pain eased. The next day the same patients got the real metal tractors and reported much the same relief.
The wood did nothing. What Haygarth had that the tractor sellers did not was a comparison.
The same game in a trading room
Amit Goyal and Sunil Wahal followed the hiring and firing of investment managers by 3,400 pension plans, foundations and endowments from 1994 to 2003. The plans hired managers after large gains over the market. After they were hired, those gains did not continue.
Then the two researchers looked at the cases where a plan fired one manager and hired another. If the plans had kept the managers they fired, their returns would have been no different from what the new managers delivered.
A board that hires on last year's results is a clinic that only sees patients on their worst day. The change looks like it worked, because the extreme year was always going to fade.
When a change seems to work, ask: compared with what?
Sources
- John Haygarth, Of the Imagination, as a Cause and as a Cure of Disorders of the Body; Exemplified by Fictitious Tractors, and Epidemical Convulsions (Bath: R. Crutwell, 1800). Account of the test from C. Booth, "The rod of Aesculapios: John Haygarth (1740–1827) and Perkins' metallic tractors", Journal of Medical Biography 13 (2005), and the James Lind Library record.
- Amit Goyal and Sunil Wahal, "The selection and termination of investment management firms by plan sponsors", Journal of Finance 63 (2008), 1805–1847.
- The patients and managers are made fresh in your browser. Each patient has a usual pain level and walks in on a day scored 7 or more. Each manager's yearly return is pure chance with no edge. The tonic and the hiring change nothing.