This article is adapted by Terry Bresnick from a presentation given at the INFORMS Practice Conference in April 2010.
What I am writing about here is an “aha” moment I had a few years ago that really made me look hard at what I do as a decision analyst. Now, we all know that decision analysts and operations research practitioners are supposed to objective, content neutral, and unbiased elicitors of information. We elicit in an unbiased fashion, analyze according to sound theoretical principles, and report findings clearly and accurately. It’s a no-brainer – my perspective was very clear for many, many years. At least it was until I was confronted with a very personal, family-related life and death situation. All of a sudden, I found that the line that I couldn’t cross was a bit blurry; the role I needed to play went beyond analyzing and reporting accurately. What I thought was a very clear perspective, changed for me overnight.
So what was the circumstance that shook my world? In October of 2008, my 39 year old daughter, Michele had been suffering from a very bad sinus infection for a few weeks, and her doctor had put her on a very strong antibiotic with side effects that required her red blood cell count to be monitored weekly. During one visit, he told her that there was something unusual and that he wanted her to see a specialist at Johns Hopkins Medical Center. After a biopsy, he gave us some very bad news. She had a relatively rare and very nasty strain of acute myelogenic leukemia (AML) known as an FLT3 mutation which doesn’t respond well to chemotherapy and radiation therapy. After the shock had set in, we peppered the doctor with many questions, the most obvious of which was the nature of her treatment and more importantly, her prognosis. He told us that she would need an initial course of chemotherapy, followed by a bone marrow donation, followed by more chemotherapy. Fortunately, Johns Hopkins is one of the best centers in the world at this form of Leukemia, and they had developed their own unique protocols that had “done well” for patients with this disease. I talked to the doctor about getting a second opinion; the doctor responded that he was admitting her immediately, would have a bed ready for her by 9:00AM and that she would have been finished with her first chemo treatment by 5:00 that afternoon. The white count was rising so quickly, that if her internist had not sent her to Hopkins when he did, she had days, or at most weeks to live. As to the prognosis, he told us that he believed Michele had a “good possibility” of making it through this.
As a decision analyst, I understand the importance of precise language in explaining probabilities to clients. While encouraged by the words “good possibility” that the doctor used, I wasn’t satisfied that I really understood what my daughter was facing. So reverting from my role as highly distressed parent to clear-headed decision analyst, I pressed the doctor for what his words meant in probability terms. As many of us have come to find, doctors generally don’t like to talk in terms of probabilities. But I was persistent, and he finally said that survival probabilities for this disease were around 20%, perhaps as high as 30% - not exactly what I had calibrated to the words “good possibility”! I pressed further – was the 20% a population statistic, or was it conditional for 39-year old, otherwise strong and healthy females? What was the probability of getting her into remission to even get to the point of bone marrow transplant? What were the odds of finding a donor? The questions just kept coming. And the more I asked, the more he tried to put me off. After all, my daughter and her family needed to know exactly what she was facing so we could properly evaluate courses of action. I assured the doctor that this type of decision making was I did for a living, and that I could best be of help by understanding the probabilities and consequences. Although I was hardly unbiased and neutral in this case, I still believed I could help her make any decisions she had to make by “eliciting” the required information. At this point, the doctor could sense my frustration, and he took me aside, out of earshot of my daughter. He then explained to me his reasons for avoiding my questions, and when he was done, a light bulb came on. And it was a light bulb that shed a new perspective not only on the effects that my role in trying to play decision analyst for my daughter could have, but on the effects that my role as a decision analyst could have on all of my clients. So what was it exactly that the doctor said to me?
The doctor told me that he had dealt with many patients in similar circumstances, and his experience and several other studies clearly showed that for patients with the same a priori survival base rates for patients with AML FLT3, the ultimate survival rates in are higher for patients who have a positive, optimistic attitude about their circumstance, who believe that the hellish treatments they are about to go through will do the trick, and who believe that they will survive, than for those who go into a funk of depression, who start to plan for their death, and who don’t believe they will make it. The psychological attitude can affect the probability of survival. In other words, the very act of eliciting and reporting on probabilities accurately can actually change the probabilities of survival in a case such as this! As Michele’s father, I clearly want to increase the chances of her survival. As her decision analyst, I want to play my ethical role in helping her to have all of the information needed to determine the right course. Yet these two fundamental objectives were clearly in conflict.
When you think about it, this revelation that the act of elicitation can actually change the probabilities of the outcomes isn’t a revelation at all. I was taught that probability is a state of information; it includes all relevant information, to include information on the physical state of things. Any time we do a diagnostic test or other data gathering, we change the state of information and thus have the potential to change the probability along a path in our decision tree. Let’s assume that a patient appears at a critical care facility with serious symptoms that could be indicative of a serious Disease A, a critical Disease B, or some very minor condition. The physician can choose to not treat, treat for A, or treat for B. Depending upon what the patient really has and what treatment the doctor selects will result in a patient outcome that is either of low consequence, serious consequence, or death-resulting.
But the doctor can probably do better by choosing to conduct a non-invasive diagnostic test that may make it clearer whether the patient is suffering from A, B, or something else. Of course, the test is not perfect, so we must incorporate various misdiagnosis probabilities and consequences into the model as well as the cost of the test. As we gain more information, we can expect the probabilities of patient impact states to change as a result of the diagnostic test.
Now, this is very basic decision analysis, and there should be no surprise that by changing the state of information, probabilities on the consequences change. But let’s take it one step further. What if the diagnostic test is invasive rather than non-invasive? For example, when a patient presents at an emergency room with chest pain, doctors sometimes perform an angiogram to look in to the heart and gather information to determine the probability that the patient has had or will have a heart attack. But there is something different about an angiogram than a non-invasive test such as an MRI. The procedure itself can initiate the heart attack. Not only can the probability change because the state of information has changed, but because the physical condition has changed as well as a result of the test.
In this context, what if we think of the routine probability elicitation process by a facilitator in a different way? The characteristics of the elicitation process are very similar to those of a diagnostic test in medicine – probe the situation for additional data to change the state of information. The interesting question is whether we are more like a non-invasive diagnostic test or an invasive diagnostic test? Are we more like an MRI that reports the situation or more like an angiogram that can alter the physical properties of the situation as well?
When I try to answer this question in the context of the admonition given to me by my daughter’s doctor, the revelation occurs. By eliciting more information from the doctor and reporting accurately on it to my daughter, I was clearly having an invasive effect, and could actually be having a negative impact on her survival.
So now we come to the heart of the dilemma – do I report the probabilities accurately and take the risk that she will take on a pessimistic attitude and decrease the probability of survival further, or do I use vague words like “good possibility” to bias her towards a more optimistic outlook that can increase the chance of survival? As a father, this was an easy choice; like the doctor, I would do everything in my power to give my daughter a survival edge. If I am going to have an “invasive” effect, let me bias it in a way that works in her favor. I can live with that and still believe I am an “ethical” decision analyst.
In the bigger picture, what I learned from my daughter’s situation has carryover implications for what I do in my work where I don’t carry the same biases towards pushing the decision in favor of the “client”. As hard as I try to be neutral and not impact a decision as a decision analyst, that may not be possible. The very act of elicitation, as we have seen, can impact the decision. And this isn’t a phenomenon that only occurs in medical decisions. When I go into a client firm to help with strategic planning, I am now more aware that the very act of my eliciting information from workers may cause them to behave differently. What if my very presence in the company leads some to believe incorrectly that there is something negative going on in the company and causes some key performers to put their resumes on the streets or “abandon ship” to get ahead of a perceived bad outcome? I am now intervening in the decision and affecting it, rather than analyzing and reporting on it -clearly troublesome in the context of a decision analysis code of ethics.
So what happened in my daughter’s case? I listened to the doctor and did everything I could to help her have a positive attitude. She was successfully put into remission after a nasty course of chemotherapy, an anonymous donor was found, she had a bone marrow transplant on 4 March 2009, she was married in August of 2009, and in March 2010, her one year post-transplant bone marrow biopsy was free of any traces of Leukemia. She recently had her 4-year biopsy, and it was perfect. If at the 5-year mark she is still free of Leukemia, she will have passed the critical threshold for AML and will be considered “cured”.