Assisted dying and evidence-based law-making: a critical analysis of an article’s role in New Zealand’s referendum | OPEN ACCESS
View Article PDF
Assisted dying (AD) is a significant social policy issue, so reliable evidence to inform law-making is critical.1 Society expects health policy2 and healthcare3 to be evidence-based. This expectation should extend to law-making.1
In September 2020, New Zealanders will participate in a binding referendum to determine whether the End of Life Choice Act 2019 (EOLCA)—legislation permitting AD already passed by Parliament—will come into force.4 A recent article published in the New Zealand Medical Journal by Winnington and MacLeod (WM article)5 raised concerns about the potential impact of this law. Further, the WM article proposed that the “evidence from this study must be factored into the New Zealand debate before the referendum on the [EOLCA]” (p21).
This article responds to the suggestion that the WM article be evidence in this debate. Our goal is to undertake a critical analysis of the evidence it purports to contribute, to determine its reliability and therefore probative value. We conclude that a combination of the nature of the study, flaws in its research design, its use of data to draw conclusions, a failure to outline limitations and inadequate engagement with social science literature make the WM article unreliable. It should not be considered as evidence in deliberations about the EOLCA.
Methods
Criteria to assess reliability
We applied three criteria to assess reliability. The first is a pyramid of evidence developed by one of the authors (JD). This pyramid draws on existing models for assessing evidence and graphically depicts how reliability increases as it ascends the pyramid, but is adapted for the evidence most commonly used when making law about AD and the sort of external review that can occur (Figure 1). Reliability is assessed both by determining the level of the pyramid at which particular evidence sits, and by considering whether it has been externally tested and the nature of that testing—whether by peer review or review by courts, non-partisan parliamentary committees or expert panels.
The second criterion is the principles that guide the conduct of social science research. We are conscious of different approaches in quantitative and qualitative research6 but there is broad acceptance across different research traditions of the following principles: study design (including data collection and analysis) that is appropriate to research aims; rigorous use of selected research methods; and fair presentation of results including only conclusions sustained by data and acknowledging appropriate limitations.
The third criterion for reliability was whether factual claims made in the WM article were defensible in light of available social science literature. We inquired whether factual claims were based on literature that was reliable (informed by some of the above considerations), up to date and fairly represented the field’s state of knowledge.
Review process
Two authors (BW and LW) reviewed the WM article multiple times and compiled a list of possible failures to meet the reliability criteria. These were grouped according to issue type and written up. These critiques were reviewed by other authors with an invitation to add new areas, revise or remove existing ones. All authors endorse the critical analysis below.
Results
Summary of Winnington and MacLeod article
The stated aim of the WM article was to consider “the possibility of consequences … for families left behind, communities and society as a whole” should New Zealand legalise AD (p18). To address this aim, a single semi-structured interview was undertaken with a person who had experience with AD in a country where AD is lawful. The interviewee’s perspective was from being married to a sibling of the person who chose AD. Thematic analysis of this interview identified three key themes: potential expectations that people would seek AD when unwell and possibly facing a life-threatening illness; stigma for individuals using AD and their families; and the potential for AD legislation to produce contagion (not defined in the article but we understand refers to the notion that AD may activate others to seek AD who would not otherwise do so) (p18).
The article calls for further research including “to investigate whether a contagion effect of AD is possible (or even probable)” (p22).
Criterion 1: reliability pyramid
As research based on a single case study, this study falls into the second bottom row of the pyramid (Figure 1). On this basis alone, the reliability of such evidence is limited. In addition, some conclusions or claims are not based on the data or go beyond what the data could support (see examples below), and would fall to the bottom row: anecdotes and opinion.
Figure 1: Adapted reliability pyramid of evidence for assisted dying.
Although the nature of the study means it is of limited reliability, some external testing adds reliability; the article is published in the peer-reviewed New Zealand Medical Journal.
Criterion 2: reliability in terms of principles of social science research
Study design not appropriate for aims
The article’s aim was to consider “the possibility of consequences … for families left behind, communities and society as a whole” should New Zealand legalise AD (p18). In terms of study design, a single interview is not capable of meeting this ambitious aim, even with the qualifier “possibility”. This is particularly so in relation to consequences for the wider communities and society as a whole. Such a method might shed light on family experiences, albeit in a very limited way with only one interview, but it cannot reliably inform about broader community perspectives.
Results extended beyond what data reasonably supports
The WM article’s analysis identified three key themes (abstract, p18):
• “the potential for assisted dying becoming an expectation for others to pursue when unwell and possibly facing a life-threatening illness;”
• “the notion of stigma being associated with the individual using assisted dying legislation and the family left behind;” and
• “that there may be the potential for such legislation to produce a contagion effect.”
We accept the interviewee discussed these three issues. But a threshold concern is whether all themes are supported by the data. It is possible that an interview with a member of an extended family with an experience of AD could, with relevant limitations articulated, produce findings about stigma within a family.
However, it is not justifiable to present the other themes as results from these data. Both relate to wider, societal-level trends rather than individual experience. The experience of one person could not, from a scientific perspective, be reliable evidence of wider community views or experiences. Determining the existence of these phenomena would require quite different research methods such as community surveys or, in the case of the contagion argument, quantitative studies about use of AD.
A second concern is how these results are extended and transformed into substantive potential problems for society. In other words, an issue is raised in the data and unjustifiably elevated to a level beyond which can be safely done from the data. We provide three examples.
Example 1: Fracturing society. The WM article expresses “concern that the legislation for AD in New Zealand has potential to fracture family and community structures” (p21). This statement is followed by the interviewee suggesting that “fractured families” make it more likely, without “support of their family”, to seek AD (p21). The first problem is that there is no evidence in the quote that the interviewee was basing his comment on experience within his family (as opposed to speculation about what might happen). The second problem is that the authors’ claim—that AD legislation could fracture family and community structures—is different from the interviewee’s point, namely that those without family support (ie, in already fractured families) may be more likely to seek AD. Yet, the segue is made from one to the other. The third problem is the shift from families (as stated by the interviewee), to the “potential to fracture family and community structures” (emphasis added) (p21), and then to the even broader claim of “potential of fracturing of our New Zealand communities and broader social settings” (emphasis added) (p21). This progression involves a significant shift from the data (a family perspective) to the wider community level.
Example 2: AD contagion. The article itself noted that the interviewee only “hinted” (p21) at this issue. This was based on the interviewee having known “three extended family members use the [AD] legislation over a short period of time” as well as AD being chosen by two of his father’s friends. However, nothing about contagion can be drawn from his quotes—there is no evidence that the three extended family members or two friends of his father who had AD knew each other or knew that each other had had AD. Furthermore, the interviewee worries that AD may be “infectious” (p21) but does not link this to the experience of his extended family member receiving AD (essential for the case study method). It could just be a pre-existing fear unrelated to the case study.
Even more concerning is that this discussion of contagion, which comprised only three paragraphs in the Results section, was transformed into a substantive concern. The shift during the article is noteworthy: from the interviewee hinting at the issue, to the conclusion calling for further research to investigate “whether a contagion effect of AD is possible”, and then “(or even probable)” (emphasis added) (p22).
Example 3: “Slippery slopes”. The WM article claims the data support slippery-slope arguments. Because of this claim’s significance, two key sentences are extracted in full here: “In conducting this study, it was anticipated that social consequences of AD legislation may be present in terms of the slippery-slope discourse. However, it was unexpected to obtain data that painted a distinct picture of how the slippery-slope effect was unfolding in a country where AD was legal” (p21, endnotes omitted). The interviewee noted a view that there may be an expectation on people to use AD, and we understand this to be the sense in which the term slippery slope is used. (We note, however, that this is not what the “slippery-slope effect” in the “slippery-slope discourse” generally means—even in the literature the authors reference for this claim).
Claims that a “distinct picture” (p21) of this occurring (an expectation that people use AD) are unsustainable. We note the many empirical studies which rebut the common slippery-slope claim that the vulnerable are more likely to seek AD.7–10 These studies include large meta-analyses or population-level studies (and so are at or towards the top of the reliability pyramid). They have been peer reviewed, and many have also been the subject of further external testing by courts,11 expert panels12–14 and non-partisan parliamentary committees.15,16 When this sort of research is placed beside the WM article, it is not reasonable to consider this single interview as reliable evidence of the “slippery-slope effect” (p21).
Appropriate statements of limitations of research
The WM article does not have a sufficiently robust statement of limitations. It acknowledges that it is based on a single interview, and that this interview was conducted in a country other than New Zealand (where AD is legal). However, the implications of this latter point are not identified, namely that findings from this single interview are not generalisable to other countries with different healthcare, social welfare, and legal systems and AD models. To illustrate, it is unclear if AD in the case study would have been available under the EOLCA, as no mention is made of an eligible terminal illness (indeed, the description at p19 makes it highly unlikely). The WM article also fails to note that a person’s pre-existing views about AD may affect their assessment of their experience. But perhaps the most significant limitation omitted was to make clear that views expressed in a single interview cannot support claims about wider societal effects such as community expectations to die and AD contagion. Indeed, instead of noting such limitations, these data from a single interview were explicitly used to ground such claims.
‘Generation’ of evidence through discussion of issues on which no data is reported
Finally, conclusions were drawn where there were no data to support such findings. While consideration of related issues may occur in an article, particularly in Background or Discussion sections, a study’s results and conclusions must be grounded in data.
The WM article at times identifies a concern about AD raised in literature (often without mentioning conflicting literature: see below). It then discusses that issue as a concern of substance, but this occurs without supporting data from the interview. One example is the claim that AD will be shaped by financial drivers. The article notes a potential consequence of AD legislation that it “reduces our future existence to being considered only through the practical lens relating to the cost of care and reduces our life to having a dollar value” (p21). This is revisited in the Conclusion: “… this case study offers insight into some elements associated with slippage [reference to slippery slope in sentence preceding] in terms of family members being expected to die when their care becomes too difficult or expensive” (p22). This significant (and very controversial) issue was not present in the data reported so its inclusion in the article’s Conclusion is not justifiable.
Criterion 3: Reliability evidenced by factual claims being defensible in light of literature
Authors must ensure that factual claims are defensible having regard to the weight of reliable literature and the field’s current state of knowledge. Some claims in the WM article cannot be defended in this way. We are not able to comprehensively catalogue all such concerns here. However, we provide one example of a claim that is not defensible and is also presented in a misleading way. The relevant passage appears in Background: “Despite the potential for those using AD legislation to be judged or stigmatised, there is further concern that AD may produce a contagion effect.13–15 Jones and Paton observed that unlike some studies that perceived AD as providing a suicide-inhibiting effect, their results suggested that any inhibitory mechanisms were counteracted by ‘equal or larger opposite effects’16–18” (p19, WM endnotes in Figure 2).
Figure 2: Selected references as cited by Winnington and MacLeod article.
A preliminary point is why, when the article claims to focus on AD contagion (ie, cases of AD leading to more AD cases), it shifts to engage with literature on the different issue of suicide contagion (ie, legalising AD leading to an increased suicide rate). Further, its suggestion that suicide contagion is a credible concern (later leveraged in potential concerns about AD contagion) is not supported by the literature, nor does the article engage with the current state of knowledge.
For instance, there is a later article by Lowe and Downie17 which critically analyses the primary source relied upon for suicide contagion (Jones and Paton), but this was not considered or even acknowledged in the WM article. Lowe and Downie identify significant errors in the Jones and Paton methodology and concerns about how the results were presented, and urge caution in relation to its findings.17 While not all literature can be cited, it is concerning that an article which has been the subject of a detailed critical analysis is presented without qualification. Also missing was the report from a major review of the state of evidence in relation to various aspects of AD undertaken by the Council of Canadian Academies (comprised of experts both in favour of AD and opposed). It concluded, including after considering the two papers above: “There is no evidence of any association between the legal status of assisted dying in a country and its suicide rate”.18 In short, there is no reliable evidence that suicide contagion will occur if AD is legalised.
A further concern is that the WM article references (Figure 2) are misleading, creating the perception that six references support the suicide contagion proposition in some way. In fact, only one study purports to consider suicide contagion (Jones and Paton: reference 16, although as noted above, Lowe and Downie argue it does not address this concept and should not be relied upon17). References 13 and 14 address potential stigma of AD (presumably a reference for the first half of the sentence), while reference 17 is Posner’s book, which includes a claim that AD may reduce rates of suicide (contrary to suicide contagion). Reference 18 appears to be to Hansard (not a study but rather a statement in parliament) although incomplete citation details mean we cannot locate what the parliamentarian said. Reference 15 refers to suicide contagion (incorrectly according to Lowe and Downie17) but is not an independent study, rather just a commentary on the Jones and Paton article.
There are other concerns about engagement with literature that could be raised. One is using literature which draws on anecdotal evidence (at the bottom of the reliability pyramid): both the Kheriaty (reference 15) and Hendin and Foley (reference 27) articles are relied upon by the WM article but they are in turn merely reporting on single cases they read about in newspaper reports. Another is not engaging with the large body of social science literature (and findings of expert panels, non-partisan parliamentary committees and courts) about “slippery slopes” (discussed above), and also being unclear about what is meant by this concept. However, as mentioned, there is not scope to include these more detailed analyses here.
Discussion
The New Zealand public will shortly decide whether AD should become lawful. Evidence-based law-making, including through a referendum, is critical, especially for significant social policy reform such as AD. This requires critical review of evidence proffered to inform public debates and public decision-making about AD.
The WM article proposed it be considered as evidence in the deliberations about AD in New Zealand. Our analysis has concluded, however, that the article is not reliable evidence and should not form part of these deliberations. It is based on a single interview with a person (the brother-in-law of a person who accessed AD) from an unidentified country where AD is legal (not New Zealand). This methodology is not capable of supporting the article’s significant claims, in particular about potential expectations that people when unwell and facing a life-threatening illness should use AD and the potential of AD contagion. In addition, the WM article presents assertions beyond what its very limited data can sustain and indeed, makes claims for which there is no data in support. Further, the study fails to appropriately limit the scope of its findings; indeed, it makes claims beyond what is justified. Finally, its reliability can also be questioned as not all of its factual claims can be supported by the social science literature.
We conclude by repeating our call for evidence-making law-making on the critical social policy issue of AD.1
Summary
Abstract
Aim
To critically analyse the reliability of an article which claims to be evidence that the End of Life Choice Act 2019 provides a “potential hotspot for family, community and social discord that may not be easily remedied” should the legislation receive public support in New Zealand’s September 2020 referendum.
Method
The subject article was reviewed multiple times by all authors and critiqued against three criteria: a reliability pyramid developed to weigh evidence about assisted dying; principles that guide the conduct of social science research; and the use of reliable and current social science literature to support factual claims.
Results
The study being analysed involved a single interview and so is located at the second bottom row of the reliability pyramid. Its research design is also unable to support the broad findings that are asserted. Other flaws in method included findings being extended beyond the data, and failure to state appropriate limitations in the research method. Further, claims are made that are unsupported by the weight of reliable social science literature.
Conclusion
The subject article is methodologically and factually flawed so is unreliable as evidence. It should not be considered in the assisted dying debates preceding the forthcoming referendum.
Author Information
Acknowledgements
Correspondence
Correspondence Email
Competing Interests
1. White BP, Willmott L. Evidence-based law making on voluntary assisted dying. Aust Health Rev. 2020. http://doi.org/10.1071/AH19201
2. Baicker K, Chandra A. Evidence-based health policy. N Engl J Med. 2017; 377:2413–15.
3. Sackett DL, Rosenberg WM, Muir Gray JA, et al. Evidence based medicine: what it is and what it isn’t. BMJ. 1996; 312:71.
4. New Zealand Government. End of Life Choice referendum. http://www.referendums.govt.nz/endoflifechoice/index.html
5. Winnington R, MacLeod R. Social consequences of assisted dying: a case study. 2020. NZMJ; 133:18–23.
6. Yin, RK. Qualitative Research from Start to Finish. 2nd edition. New York: Guilford Press; 2016.
7. Battin MP, van der Heide A, Ganzini L, et al. Legal Physician-Assisted Dying in Oregon and the Netherlands: Evidence Concerning the Impact on Patients in ‘Vulnerable’ Groups. Journal of Med Ethics. 2007; 33(10):591–7.
8. Rietjens JAC, Deschepper R, Pasman R, Deliens L. Medical End of life Decisions: Does its Use Differ in Vulnerable Patient Groups? A Systematic Review and Meta analysis. Soc Sci and Med. 2012; 74(8):1282–87.
9. Emanuel EJ, Onwuteaka-Philipsen BD, Urwin JW, Cohen J. Attitudes and Practices of Euthanasia and Physician-Assisted Suicide in the United States, Canada and Europe. JAMA. 2016; 316(1):79–90.
10. Blanke C, LeBlanc M, Hershman D, et al. Characterizing 18 years of the Death with Dignity Act in Oregon. JAMA Oncology. 2017; 3(10):1403–6.
11. Carter v Canada (Attorney General) 2012 BCSC 886, paragraphs 359–747.
12. Victorian Government. Ministerial Advisory Panel on Voluntary Assisted Dying: Final Report. 2017, p88.
13. Western Australian Government. Ministerial Expert Panel on Voluntary Assisted Dying: Final Report. 2018, p26.
14. Royal Society of Canada Expert Panel. End-of-Life Decision Making. 2011, pp84–88.
15. Parliament of Victoria, Inquiry into End of Life Choices: Final Report. 2016, pp xxviii, 212–13.
16. Parliament of Western Australia. My Life, My Choice: The Report of the Joint Select Committee on End of Life Choices. 2018, pp175–181 (Finding 42).
17. Lowe MP, Downie J. Does Legalization of Medical Assistance in Dying Affect Rates of Non-assisted Suicide?. Journal of Ethics in Mental Health. 2017; 10:1–9
18. Council of Canadian Academies. The State of Knowledge on Medical Assistance in Dying Where a Mental Disorder Is the Sole Underlying Medical Condition. 2018, p96.
For the PDF of this article,
contact nzmj@nzma.org.nz
Assisted dying and evidence-based law-making: a critical analysis of an article’s role in New Zealand’s referendum | OPEN ACCESS
View Article PDF
Assisted dying (AD) is a significant social policy issue, so reliable evidence to inform law-making is critical.1 Society expects health policy2 and healthcare3 to be evidence-based. This expectation should extend to law-making.1
In September 2020, New Zealanders will participate in a binding referendum to determine whether the End of Life Choice Act 2019 (EOLCA)—legislation permitting AD already passed by Parliament—will come into force.4 A recent article published in the New Zealand Medical Journal by Winnington and MacLeod (WM article)5 raised concerns about the potential impact of this law. Further, the WM article proposed that the “evidence from this study must be factored into the New Zealand debate before the referendum on the [EOLCA]” (p21).
This article responds to the suggestion that the WM article be evidence in this debate. Our goal is to undertake a critical analysis of the evidence it purports to contribute, to determine its reliability and therefore probative value. We conclude that a combination of the nature of the study, flaws in its research design, its use of data to draw conclusions, a failure to outline limitations and inadequate engagement with social science literature make the WM article unreliable. It should not be considered as evidence in deliberations about the EOLCA.
Methods
Criteria to assess reliability
We applied three criteria to assess reliability. The first is a pyramid of evidence developed by one of the authors (JD). This pyramid draws on existing models for assessing evidence and graphically depicts how reliability increases as it ascends the pyramid, but is adapted for the evidence most commonly used when making law about AD and the sort of external review that can occur (Figure 1). Reliability is assessed both by determining the level of the pyramid at which particular evidence sits, and by considering whether it has been externally tested and the nature of that testing—whether by peer review or review by courts, non-partisan parliamentary committees or expert panels.
The second criterion is the principles that guide the conduct of social science research. We are conscious of different approaches in quantitative and qualitative research6 but there is broad acceptance across different research traditions of the following principles: study design (including data collection and analysis) that is appropriate to research aims; rigorous use of selected research methods; and fair presentation of results including only conclusions sustained by data and acknowledging appropriate limitations.
The third criterion for reliability was whether factual claims made in the WM article were defensible in light of available social science literature. We inquired whether factual claims were based on literature that was reliable (informed by some of the above considerations), up to date and fairly represented the field’s state of knowledge.
Review process
Two authors (BW and LW) reviewed the WM article multiple times and compiled a list of possible failures to meet the reliability criteria. These were grouped according to issue type and written up. These critiques were reviewed by other authors with an invitation to add new areas, revise or remove existing ones. All authors endorse the critical analysis below.
Results
Summary of Winnington and MacLeod article
The stated aim of the WM article was to consider “the possibility of consequences … for families left behind, communities and society as a whole” should New Zealand legalise AD (p18). To address this aim, a single semi-structured interview was undertaken with a person who had experience with AD in a country where AD is lawful. The interviewee’s perspective was from being married to a sibling of the person who chose AD. Thematic analysis of this interview identified three key themes: potential expectations that people would seek AD when unwell and possibly facing a life-threatening illness; stigma for individuals using AD and their families; and the potential for AD legislation to produce contagion (not defined in the article but we understand refers to the notion that AD may activate others to seek AD who would not otherwise do so) (p18).
The article calls for further research including “to investigate whether a contagion effect of AD is possible (or even probable)” (p22).
Criterion 1: reliability pyramid
As research based on a single case study, this study falls into the second bottom row of the pyramid (Figure 1). On this basis alone, the reliability of such evidence is limited. In addition, some conclusions or claims are not based on the data or go beyond what the data could support (see examples below), and would fall to the bottom row: anecdotes and opinion.
Figure 1: Adapted reliability pyramid of evidence for assisted dying.
Although the nature of the study means it is of limited reliability, some external testing adds reliability; the article is published in the peer-reviewed New Zealand Medical Journal.
Criterion 2: reliability in terms of principles of social science research
Study design not appropriate for aims
The article’s aim was to consider “the possibility of consequences … for families left behind, communities and society as a whole” should New Zealand legalise AD (p18). In terms of study design, a single interview is not capable of meeting this ambitious aim, even with the qualifier “possibility”. This is particularly so in relation to consequences for the wider communities and society as a whole. Such a method might shed light on family experiences, albeit in a very limited way with only one interview, but it cannot reliably inform about broader community perspectives.
Results extended beyond what data reasonably supports
The WM article’s analysis identified three key themes (abstract, p18):
• “the potential for assisted dying becoming an expectation for others to pursue when unwell and possibly facing a life-threatening illness;”
• “the notion of stigma being associated with the individual using assisted dying legislation and the family left behind;” and
• “that there may be the potential for such legislation to produce a contagion effect.”
We accept the interviewee discussed these three issues. But a threshold concern is whether all themes are supported by the data. It is possible that an interview with a member of an extended family with an experience of AD could, with relevant limitations articulated, produce findings about stigma within a family.
However, it is not justifiable to present the other themes as results from these data. Both relate to wider, societal-level trends rather than individual experience. The experience of one person could not, from a scientific perspective, be reliable evidence of wider community views or experiences. Determining the existence of these phenomena would require quite different research methods such as community surveys or, in the case of the contagion argument, quantitative studies about use of AD.
A second concern is how these results are extended and transformed into substantive potential problems for society. In other words, an issue is raised in the data and unjustifiably elevated to a level beyond which can be safely done from the data. We provide three examples.
Example 1: Fracturing society. The WM article expresses “concern that the legislation for AD in New Zealand has potential to fracture family and community structures” (p21). This statement is followed by the interviewee suggesting that “fractured families” make it more likely, without “support of their family”, to seek AD (p21). The first problem is that there is no evidence in the quote that the interviewee was basing his comment on experience within his family (as opposed to speculation about what might happen). The second problem is that the authors’ claim—that AD legislation could fracture family and community structures—is different from the interviewee’s point, namely that those without family support (ie, in already fractured families) may be more likely to seek AD. Yet, the segue is made from one to the other. The third problem is the shift from families (as stated by the interviewee), to the “potential to fracture family and community structures” (emphasis added) (p21), and then to the even broader claim of “potential of fracturing of our New Zealand communities and broader social settings” (emphasis added) (p21). This progression involves a significant shift from the data (a family perspective) to the wider community level.
Example 2: AD contagion. The article itself noted that the interviewee only “hinted” (p21) at this issue. This was based on the interviewee having known “three extended family members use the [AD] legislation over a short period of time” as well as AD being chosen by two of his father’s friends. However, nothing about contagion can be drawn from his quotes—there is no evidence that the three extended family members or two friends of his father who had AD knew each other or knew that each other had had AD. Furthermore, the interviewee worries that AD may be “infectious” (p21) but does not link this to the experience of his extended family member receiving AD (essential for the case study method). It could just be a pre-existing fear unrelated to the case study.
Even more concerning is that this discussion of contagion, which comprised only three paragraphs in the Results section, was transformed into a substantive concern. The shift during the article is noteworthy: from the interviewee hinting at the issue, to the conclusion calling for further research to investigate “whether a contagion effect of AD is possible”, and then “(or even probable)” (emphasis added) (p22).
Example 3: “Slippery slopes”. The WM article claims the data support slippery-slope arguments. Because of this claim’s significance, two key sentences are extracted in full here: “In conducting this study, it was anticipated that social consequences of AD legislation may be present in terms of the slippery-slope discourse. However, it was unexpected to obtain data that painted a distinct picture of how the slippery-slope effect was unfolding in a country where AD was legal” (p21, endnotes omitted). The interviewee noted a view that there may be an expectation on people to use AD, and we understand this to be the sense in which the term slippery slope is used. (We note, however, that this is not what the “slippery-slope effect” in the “slippery-slope discourse” generally means—even in the literature the authors reference for this claim).
Claims that a “distinct picture” (p21) of this occurring (an expectation that people use AD) are unsustainable. We note the many empirical studies which rebut the common slippery-slope claim that the vulnerable are more likely to seek AD.7–10 These studies include large meta-analyses or population-level studies (and so are at or towards the top of the reliability pyramid). They have been peer reviewed, and many have also been the subject of further external testing by courts,11 expert panels12–14 and non-partisan parliamentary committees.15,16 When this sort of research is placed beside the WM article, it is not reasonable to consider this single interview as reliable evidence of the “slippery-slope effect” (p21).
Appropriate statements of limitations of research
The WM article does not have a sufficiently robust statement of limitations. It acknowledges that it is based on a single interview, and that this interview was conducted in a country other than New Zealand (where AD is legal). However, the implications of this latter point are not identified, namely that findings from this single interview are not generalisable to other countries with different healthcare, social welfare, and legal systems and AD models. To illustrate, it is unclear if AD in the case study would have been available under the EOLCA, as no mention is made of an eligible terminal illness (indeed, the description at p19 makes it highly unlikely). The WM article also fails to note that a person’s pre-existing views about AD may affect their assessment of their experience. But perhaps the most significant limitation omitted was to make clear that views expressed in a single interview cannot support claims about wider societal effects such as community expectations to die and AD contagion. Indeed, instead of noting such limitations, these data from a single interview were explicitly used to ground such claims.
‘Generation’ of evidence through discussion of issues on which no data is reported
Finally, conclusions were drawn where there were no data to support such findings. While consideration of related issues may occur in an article, particularly in Background or Discussion sections, a study’s results and conclusions must be grounded in data.
The WM article at times identifies a concern about AD raised in literature (often without mentioning conflicting literature: see below). It then discusses that issue as a concern of substance, but this occurs without supporting data from the interview. One example is the claim that AD will be shaped by financial drivers. The article notes a potential consequence of AD legislation that it “reduces our future existence to being considered only through the practical lens relating to the cost of care and reduces our life to having a dollar value” (p21). This is revisited in the Conclusion: “… this case study offers insight into some elements associated with slippage [reference to slippery slope in sentence preceding] in terms of family members being expected to die when their care becomes too difficult or expensive” (p22). This significant (and very controversial) issue was not present in the data reported so its inclusion in the article’s Conclusion is not justifiable.
Criterion 3: Reliability evidenced by factual claims being defensible in light of literature
Authors must ensure that factual claims are defensible having regard to the weight of reliable literature and the field’s current state of knowledge. Some claims in the WM article cannot be defended in this way. We are not able to comprehensively catalogue all such concerns here. However, we provide one example of a claim that is not defensible and is also presented in a misleading way. The relevant passage appears in Background: “Despite the potential for those using AD legislation to be judged or stigmatised, there is further concern that AD may produce a contagion effect.13–15 Jones and Paton observed that unlike some studies that perceived AD as providing a suicide-inhibiting effect, their results suggested that any inhibitory mechanisms were counteracted by ‘equal or larger opposite effects’16–18” (p19, WM endnotes in Figure 2).
Figure 2: Selected references as cited by Winnington and MacLeod article.
A preliminary point is why, when the article claims to focus on AD contagion (ie, cases of AD leading to more AD cases), it shifts to engage with literature on the different issue of suicide contagion (ie, legalising AD leading to an increased suicide rate). Further, its suggestion that suicide contagion is a credible concern (later leveraged in potential concerns about AD contagion) is not supported by the literature, nor does the article engage with the current state of knowledge.
For instance, there is a later article by Lowe and Downie17 which critically analyses the primary source relied upon for suicide contagion (Jones and Paton), but this was not considered or even acknowledged in the WM article. Lowe and Downie identify significant errors in the Jones and Paton methodology and concerns about how the results were presented, and urge caution in relation to its findings.17 While not all literature can be cited, it is concerning that an article which has been the subject of a detailed critical analysis is presented without qualification. Also missing was the report from a major review of the state of evidence in relation to various aspects of AD undertaken by the Council of Canadian Academies (comprised of experts both in favour of AD and opposed). It concluded, including after considering the two papers above: “There is no evidence of any association between the legal status of assisted dying in a country and its suicide rate”.18 In short, there is no reliable evidence that suicide contagion will occur if AD is legalised.
A further concern is that the WM article references (Figure 2) are misleading, creating the perception that six references support the suicide contagion proposition in some way. In fact, only one study purports to consider suicide contagion (Jones and Paton: reference 16, although as noted above, Lowe and Downie argue it does not address this concept and should not be relied upon17). References 13 and 14 address potential stigma of AD (presumably a reference for the first half of the sentence), while reference 17 is Posner’s book, which includes a claim that AD may reduce rates of suicide (contrary to suicide contagion). Reference 18 appears to be to Hansard (not a study but rather a statement in parliament) although incomplete citation details mean we cannot locate what the parliamentarian said. Reference 15 refers to suicide contagion (incorrectly according to Lowe and Downie17) but is not an independent study, rather just a commentary on the Jones and Paton article.
There are other concerns about engagement with literature that could be raised. One is using literature which draws on anecdotal evidence (at the bottom of the reliability pyramid): both the Kheriaty (reference 15) and Hendin and Foley (reference 27) articles are relied upon by the WM article but they are in turn merely reporting on single cases they read about in newspaper reports. Another is not engaging with the large body of social science literature (and findings of expert panels, non-partisan parliamentary committees and courts) about “slippery slopes” (discussed above), and also being unclear about what is meant by this concept. However, as mentioned, there is not scope to include these more detailed analyses here.
Discussion
The New Zealand public will shortly decide whether AD should become lawful. Evidence-based law-making, including through a referendum, is critical, especially for significant social policy reform such as AD. This requires critical review of evidence proffered to inform public debates and public decision-making about AD.
The WM article proposed it be considered as evidence in the deliberations about AD in New Zealand. Our analysis has concluded, however, that the article is not reliable evidence and should not form part of these deliberations. It is based on a single interview with a person (the brother-in-law of a person who accessed AD) from an unidentified country where AD is legal (not New Zealand). This methodology is not capable of supporting the article’s significant claims, in particular about potential expectations that people when unwell and facing a life-threatening illness should use AD and the potential of AD contagion. In addition, the WM article presents assertions beyond what its very limited data can sustain and indeed, makes claims for which there is no data in support. Further, the study fails to appropriately limit the scope of its findings; indeed, it makes claims beyond what is justified. Finally, its reliability can also be questioned as not all of its factual claims can be supported by the social science literature.
We conclude by repeating our call for evidence-making law-making on the critical social policy issue of AD.1
Summary
Abstract
Aim
To critically analyse the reliability of an article which claims to be evidence that the End of Life Choice Act 2019 provides a “potential hotspot for family, community and social discord that may not be easily remedied” should the legislation receive public support in New Zealand’s September 2020 referendum.
Method
The subject article was reviewed multiple times by all authors and critiqued against three criteria: a reliability pyramid developed to weigh evidence about assisted dying; principles that guide the conduct of social science research; and the use of reliable and current social science literature to support factual claims.
Results
The study being analysed involved a single interview and so is located at the second bottom row of the reliability pyramid. Its research design is also unable to support the broad findings that are asserted. Other flaws in method included findings being extended beyond the data, and failure to state appropriate limitations in the research method. Further, claims are made that are unsupported by the weight of reliable social science literature.
Conclusion
The subject article is methodologically and factually flawed so is unreliable as evidence. It should not be considered in the assisted dying debates preceding the forthcoming referendum.
Author Information
Acknowledgements
Correspondence
Correspondence Email
Competing Interests
1. White BP, Willmott L. Evidence-based law making on voluntary assisted dying. Aust Health Rev. 2020. http://doi.org/10.1071/AH19201
2. Baicker K, Chandra A. Evidence-based health policy. N Engl J Med. 2017; 377:2413–15.
3. Sackett DL, Rosenberg WM, Muir Gray JA, et al. Evidence based medicine: what it is and what it isn’t. BMJ. 1996; 312:71.
4. New Zealand Government. End of Life Choice referendum. http://www.referendums.govt.nz/endoflifechoice/index.html
5. Winnington R, MacLeod R. Social consequences of assisted dying: a case study. 2020. NZMJ; 133:18–23.
6. Yin, RK. Qualitative Research from Start to Finish. 2nd edition. New York: Guilford Press; 2016.
7. Battin MP, van der Heide A, Ganzini L, et al. Legal Physician-Assisted Dying in Oregon and the Netherlands: Evidence Concerning the Impact on Patients in ‘Vulnerable’ Groups. Journal of Med Ethics. 2007; 33(10):591–7.
8. Rietjens JAC, Deschepper R, Pasman R, Deliens L. Medical End of life Decisions: Does its Use Differ in Vulnerable Patient Groups? A Systematic Review and Meta analysis. Soc Sci and Med. 2012; 74(8):1282–87.
9. Emanuel EJ, Onwuteaka-Philipsen BD, Urwin JW, Cohen J. Attitudes and Practices of Euthanasia and Physician-Assisted Suicide in the United States, Canada and Europe. JAMA. 2016; 316(1):79–90.
10. Blanke C, LeBlanc M, Hershman D, et al. Characterizing 18 years of the Death with Dignity Act in Oregon. JAMA Oncology. 2017; 3(10):1403–6.
11. Carter v Canada (Attorney General) 2012 BCSC 886, paragraphs 359–747.
12. Victorian Government. Ministerial Advisory Panel on Voluntary Assisted Dying: Final Report. 2017, p88.
13. Western Australian Government. Ministerial Expert Panel on Voluntary Assisted Dying: Final Report. 2018, p26.
14. Royal Society of Canada Expert Panel. End-of-Life Decision Making. 2011, pp84–88.
15. Parliament of Victoria, Inquiry into End of Life Choices: Final Report. 2016, pp xxviii, 212–13.
16. Parliament of Western Australia. My Life, My Choice: The Report of the Joint Select Committee on End of Life Choices. 2018, pp175–181 (Finding 42).
17. Lowe MP, Downie J. Does Legalization of Medical Assistance in Dying Affect Rates of Non-assisted Suicide?. Journal of Ethics in Mental Health. 2017; 10:1–9
18. Council of Canadian Academies. The State of Knowledge on Medical Assistance in Dying Where a Mental Disorder Is the Sole Underlying Medical Condition. 2018, p96.
For the PDF of this article,
contact nzmj@nzma.org.nz
Assisted dying and evidence-based law-making: a critical analysis of an article’s role in New Zealand’s referendum | OPEN ACCESS
View Article PDF
Assisted dying (AD) is a significant social policy issue, so reliable evidence to inform law-making is critical.1 Society expects health policy2 and healthcare3 to be evidence-based. This expectation should extend to law-making.1
In September 2020, New Zealanders will participate in a binding referendum to determine whether the End of Life Choice Act 2019 (EOLCA)—legislation permitting AD already passed by Parliament—will come into force.4 A recent article published in the New Zealand Medical Journal by Winnington and MacLeod (WM article)5 raised concerns about the potential impact of this law. Further, the WM article proposed that the “evidence from this study must be factored into the New Zealand debate before the referendum on the [EOLCA]” (p21).
This article responds to the suggestion that the WM article be evidence in this debate. Our goal is to undertake a critical analysis of the evidence it purports to contribute, to determine its reliability and therefore probative value. We conclude that a combination of the nature of the study, flaws in its research design, its use of data to draw conclusions, a failure to outline limitations and inadequate engagement with social science literature make the WM article unreliable. It should not be considered as evidence in deliberations about the EOLCA.
Methods
Criteria to assess reliability
We applied three criteria to assess reliability. The first is a pyramid of evidence developed by one of the authors (JD). This pyramid draws on existing models for assessing evidence and graphically depicts how reliability increases as it ascends the pyramid, but is adapted for the evidence most commonly used when making law about AD and the sort of external review that can occur (Figure 1). Reliability is assessed both by determining the level of the pyramid at which particular evidence sits, and by considering whether it has been externally tested and the nature of that testing—whether by peer review or review by courts, non-partisan parliamentary committees or expert panels.
The second criterion is the principles that guide the conduct of social science research. We are conscious of different approaches in quantitative and qualitative research6 but there is broad acceptance across different research traditions of the following principles: study design (including data collection and analysis) that is appropriate to research aims; rigorous use of selected research methods; and fair presentation of results including only conclusions sustained by data and acknowledging appropriate limitations.
The third criterion for reliability was whether factual claims made in the WM article were defensible in light of available social science literature. We inquired whether factual claims were based on literature that was reliable (informed by some of the above considerations), up to date and fairly represented the field’s state of knowledge.
Review process
Two authors (BW and LW) reviewed the WM article multiple times and compiled a list of possible failures to meet the reliability criteria. These were grouped according to issue type and written up. These critiques were reviewed by other authors with an invitation to add new areas, revise or remove existing ones. All authors endorse the critical analysis below.
Results
Summary of Winnington and MacLeod article
The stated aim of the WM article was to consider “the possibility of consequences … for families left behind, communities and society as a whole” should New Zealand legalise AD (p18). To address this aim, a single semi-structured interview was undertaken with a person who had experience with AD in a country where AD is lawful. The interviewee’s perspective was from being married to a sibling of the person who chose AD. Thematic analysis of this interview identified three key themes: potential expectations that people would seek AD when unwell and possibly facing a life-threatening illness; stigma for individuals using AD and their families; and the potential for AD legislation to produce contagion (not defined in the article but we understand refers to the notion that AD may activate others to seek AD who would not otherwise do so) (p18).
The article calls for further research including “to investigate whether a contagion effect of AD is possible (or even probable)” (p22).
Criterion 1: reliability pyramid
As research based on a single case study, this study falls into the second bottom row of the pyramid (Figure 1). On this basis alone, the reliability of such evidence is limited. In addition, some conclusions or claims are not based on the data or go beyond what the data could support (see examples below), and would fall to the bottom row: anecdotes and opinion.
Figure 1: Adapted reliability pyramid of evidence for assisted dying.
Although the nature of the study means it is of limited reliability, some external testing adds reliability; the article is published in the peer-reviewed New Zealand Medical Journal.
Criterion 2: reliability in terms of principles of social science research
Study design not appropriate for aims
The article’s aim was to consider “the possibility of consequences … for families left behind, communities and society as a whole” should New Zealand legalise AD (p18). In terms of study design, a single interview is not capable of meeting this ambitious aim, even with the qualifier “possibility”. This is particularly so in relation to consequences for the wider communities and society as a whole. Such a method might shed light on family experiences, albeit in a very limited way with only one interview, but it cannot reliably inform about broader community perspectives.
Results extended beyond what data reasonably supports
The WM article’s analysis identified three key themes (abstract, p18):
• “the potential for assisted dying becoming an expectation for others to pursue when unwell and possibly facing a life-threatening illness;”
• “the notion of stigma being associated with the individual using assisted dying legislation and the family left behind;” and
• “that there may be the potential for such legislation to produce a contagion effect.”
We accept the interviewee discussed these three issues. But a threshold concern is whether all themes are supported by the data. It is possible that an interview with a member of an extended family with an experience of AD could, with relevant limitations articulated, produce findings about stigma within a family.
However, it is not justifiable to present the other themes as results from these data. Both relate to wider, societal-level trends rather than individual experience. The experience of one person could not, from a scientific perspective, be reliable evidence of wider community views or experiences. Determining the existence of these phenomena would require quite different research methods such as community surveys or, in the case of the contagion argument, quantitative studies about use of AD.
A second concern is how these results are extended and transformed into substantive potential problems for society. In other words, an issue is raised in the data and unjustifiably elevated to a level beyond which can be safely done from the data. We provide three examples.
Example 1: Fracturing society. The WM article expresses “concern that the legislation for AD in New Zealand has potential to fracture family and community structures” (p21). This statement is followed by the interviewee suggesting that “fractured families” make it more likely, without “support of their family”, to seek AD (p21). The first problem is that there is no evidence in the quote that the interviewee was basing his comment on experience within his family (as opposed to speculation about what might happen). The second problem is that the authors’ claim—that AD legislation could fracture family and community structures—is different from the interviewee’s point, namely that those without family support (ie, in already fractured families) may be more likely to seek AD. Yet, the segue is made from one to the other. The third problem is the shift from families (as stated by the interviewee), to the “potential to fracture family and community structures” (emphasis added) (p21), and then to the even broader claim of “potential of fracturing of our New Zealand communities and broader social settings” (emphasis added) (p21). This progression involves a significant shift from the data (a family perspective) to the wider community level.
Example 2: AD contagion. The article itself noted that the interviewee only “hinted” (p21) at this issue. This was based on the interviewee having known “three extended family members use the [AD] legislation over a short period of time” as well as AD being chosen by two of his father’s friends. However, nothing about contagion can be drawn from his quotes—there is no evidence that the three extended family members or two friends of his father who had AD knew each other or knew that each other had had AD. Furthermore, the interviewee worries that AD may be “infectious” (p21) but does not link this to the experience of his extended family member receiving AD (essential for the case study method). It could just be a pre-existing fear unrelated to the case study.
Even more concerning is that this discussion of contagion, which comprised only three paragraphs in the Results section, was transformed into a substantive concern. The shift during the article is noteworthy: from the interviewee hinting at the issue, to the conclusion calling for further research to investigate “whether a contagion effect of AD is possible”, and then “(or even probable)” (emphasis added) (p22).
Example 3: “Slippery slopes”. The WM article claims the data support slippery-slope arguments. Because of this claim’s significance, two key sentences are extracted in full here: “In conducting this study, it was anticipated that social consequences of AD legislation may be present in terms of the slippery-slope discourse. However, it was unexpected to obtain data that painted a distinct picture of how the slippery-slope effect was unfolding in a country where AD was legal” (p21, endnotes omitted). The interviewee noted a view that there may be an expectation on people to use AD, and we understand this to be the sense in which the term slippery slope is used. (We note, however, that this is not what the “slippery-slope effect” in the “slippery-slope discourse” generally means—even in the literature the authors reference for this claim).
Claims that a “distinct picture” (p21) of this occurring (an expectation that people use AD) are unsustainable. We note the many empirical studies which rebut the common slippery-slope claim that the vulnerable are more likely to seek AD.7–10 These studies include large meta-analyses or population-level studies (and so are at or towards the top of the reliability pyramid). They have been peer reviewed, and many have also been the subject of further external testing by courts,11 expert panels12–14 and non-partisan parliamentary committees.15,16 When this sort of research is placed beside the WM article, it is not reasonable to consider this single interview as reliable evidence of the “slippery-slope effect” (p21).
Appropriate statements of limitations of research
The WM article does not have a sufficiently robust statement of limitations. It acknowledges that it is based on a single interview, and that this interview was conducted in a country other than New Zealand (where AD is legal). However, the implications of this latter point are not identified, namely that findings from this single interview are not generalisable to other countries with different healthcare, social welfare, and legal systems and AD models. To illustrate, it is unclear if AD in the case study would have been available under the EOLCA, as no mention is made of an eligible terminal illness (indeed, the description at p19 makes it highly unlikely). The WM article also fails to note that a person’s pre-existing views about AD may affect their assessment of their experience. But perhaps the most significant limitation omitted was to make clear that views expressed in a single interview cannot support claims about wider societal effects such as community expectations to die and AD contagion. Indeed, instead of noting such limitations, these data from a single interview were explicitly used to ground such claims.
‘Generation’ of evidence through discussion of issues on which no data is reported
Finally, conclusions were drawn where there were no data to support such findings. While consideration of related issues may occur in an article, particularly in Background or Discussion sections, a study’s results and conclusions must be grounded in data.
The WM article at times identifies a concern about AD raised in literature (often without mentioning conflicting literature: see below). It then discusses that issue as a concern of substance, but this occurs without supporting data from the interview. One example is the claim that AD will be shaped by financial drivers. The article notes a potential consequence of AD legislation that it “reduces our future existence to being considered only through the practical lens relating to the cost of care and reduces our life to having a dollar value” (p21). This is revisited in the Conclusion: “… this case study offers insight into some elements associated with slippage [reference to slippery slope in sentence preceding] in terms of family members being expected to die when their care becomes too difficult or expensive” (p22). This significant (and very controversial) issue was not present in the data reported so its inclusion in the article’s Conclusion is not justifiable.
Criterion 3: Reliability evidenced by factual claims being defensible in light of literature
Authors must ensure that factual claims are defensible having regard to the weight of reliable literature and the field’s current state of knowledge. Some claims in the WM article cannot be defended in this way. We are not able to comprehensively catalogue all such concerns here. However, we provide one example of a claim that is not defensible and is also presented in a misleading way. The relevant passage appears in Background: “Despite the potential for those using AD legislation to be judged or stigmatised, there is further concern that AD may produce a contagion effect.13–15 Jones and Paton observed that unlike some studies that perceived AD as providing a suicide-inhibiting effect, their results suggested that any inhibitory mechanisms were counteracted by ‘equal or larger opposite effects’16–18” (p19, WM endnotes in Figure 2).
Figure 2: Selected references as cited by Winnington and MacLeod article.
A preliminary point is why, when the article claims to focus on AD contagion (ie, cases of AD leading to more AD cases), it shifts to engage with literature on the different issue of suicide contagion (ie, legalising AD leading to an increased suicide rate). Further, its suggestion that suicide contagion is a credible concern (later leveraged in potential concerns about AD contagion) is not supported by the literature, nor does the article engage with the current state of knowledge.
For instance, there is a later article by Lowe and Downie17 which critically analyses the primary source relied upon for suicide contagion (Jones and Paton), but this was not considered or even acknowledged in the WM article. Lowe and Downie identify significant errors in the Jones and Paton methodology and concerns about how the results were presented, and urge caution in relation to its findings.17 While not all literature can be cited, it is concerning that an article which has been the subject of a detailed critical analysis is presented without qualification. Also missing was the report from a major review of the state of evidence in relation to various aspects of AD undertaken by the Council of Canadian Academies (comprised of experts both in favour of AD and opposed). It concluded, including after considering the two papers above: “There is no evidence of any association between the legal status of assisted dying in a country and its suicide rate”.18 In short, there is no reliable evidence that suicide contagion will occur if AD is legalised.
A further concern is that the WM article references (Figure 2) are misleading, creating the perception that six references support the suicide contagion proposition in some way. In fact, only one study purports to consider suicide contagion (Jones and Paton: reference 16, although as noted above, Lowe and Downie argue it does not address this concept and should not be relied upon17). References 13 and 14 address potential stigma of AD (presumably a reference for the first half of the sentence), while reference 17 is Posner’s book, which includes a claim that AD may reduce rates of suicide (contrary to suicide contagion). Reference 18 appears to be to Hansard (not a study but rather a statement in parliament) although incomplete citation details mean we cannot locate what the parliamentarian said. Reference 15 refers to suicide contagion (incorrectly according to Lowe and Downie17) but is not an independent study, rather just a commentary on the Jones and Paton article.
There are other concerns about engagement with literature that could be raised. One is using literature which draws on anecdotal evidence (at the bottom of the reliability pyramid): both the Kheriaty (reference 15) and Hendin and Foley (reference 27) articles are relied upon by the WM article but they are in turn merely reporting on single cases they read about in newspaper reports. Another is not engaging with the large body of social science literature (and findings of expert panels, non-partisan parliamentary committees and courts) about “slippery slopes” (discussed above), and also being unclear about what is meant by this concept. However, as mentioned, there is not scope to include these more detailed analyses here.
Discussion
The New Zealand public will shortly decide whether AD should become lawful. Evidence-based law-making, including through a referendum, is critical, especially for significant social policy reform such as AD. This requires critical review of evidence proffered to inform public debates and public decision-making about AD.
The WM article proposed it be considered as evidence in the deliberations about AD in New Zealand. Our analysis has concluded, however, that the article is not reliable evidence and should not form part of these deliberations. It is based on a single interview with a person (the brother-in-law of a person who accessed AD) from an unidentified country where AD is legal (not New Zealand). This methodology is not capable of supporting the article’s significant claims, in particular about potential expectations that people when unwell and facing a life-threatening illness should use AD and the potential of AD contagion. In addition, the WM article presents assertions beyond what its very limited data can sustain and indeed, makes claims for which there is no data in support. Further, the study fails to appropriately limit the scope of its findings; indeed, it makes claims beyond what is justified. Finally, its reliability can also be questioned as not all of its factual claims can be supported by the social science literature.
We conclude by repeating our call for evidence-making law-making on the critical social policy issue of AD.1
Summary
Abstract
Aim
To critically analyse the reliability of an article which claims to be evidence that the End of Life Choice Act 2019 provides a “potential hotspot for family, community and social discord that may not be easily remedied” should the legislation receive public support in New Zealand’s September 2020 referendum.
Method
The subject article was reviewed multiple times by all authors and critiqued against three criteria: a reliability pyramid developed to weigh evidence about assisted dying; principles that guide the conduct of social science research; and the use of reliable and current social science literature to support factual claims.
Results
The study being analysed involved a single interview and so is located at the second bottom row of the reliability pyramid. Its research design is also unable to support the broad findings that are asserted. Other flaws in method included findings being extended beyond the data, and failure to state appropriate limitations in the research method. Further, claims are made that are unsupported by the weight of reliable social science literature.
Conclusion
The subject article is methodologically and factually flawed so is unreliable as evidence. It should not be considered in the assisted dying debates preceding the forthcoming referendum.
Author Information
Acknowledgements
Correspondence
Correspondence Email
Competing Interests
1. White BP, Willmott L. Evidence-based law making on voluntary assisted dying. Aust Health Rev. 2020. http://doi.org/10.1071/AH19201
2. Baicker K, Chandra A. Evidence-based health policy. N Engl J Med. 2017; 377:2413–15.
3. Sackett DL, Rosenberg WM, Muir Gray JA, et al. Evidence based medicine: what it is and what it isn’t. BMJ. 1996; 312:71.
4. New Zealand Government. End of Life Choice referendum. http://www.referendums.govt.nz/endoflifechoice/index.html
5. Winnington R, MacLeod R. Social consequences of assisted dying: a case study. 2020. NZMJ; 133:18–23.
6. Yin, RK. Qualitative Research from Start to Finish. 2nd edition. New York: Guilford Press; 2016.
7. Battin MP, van der Heide A, Ganzini L, et al. Legal Physician-Assisted Dying in Oregon and the Netherlands: Evidence Concerning the Impact on Patients in ‘Vulnerable’ Groups. Journal of Med Ethics. 2007; 33(10):591–7.
8. Rietjens JAC, Deschepper R, Pasman R, Deliens L. Medical End of life Decisions: Does its Use Differ in Vulnerable Patient Groups? A Systematic Review and Meta analysis. Soc Sci and Med. 2012; 74(8):1282–87.
9. Emanuel EJ, Onwuteaka-Philipsen BD, Urwin JW, Cohen J. Attitudes and Practices of Euthanasia and Physician-Assisted Suicide in the United States, Canada and Europe. JAMA. 2016; 316(1):79–90.
10. Blanke C, LeBlanc M, Hershman D, et al. Characterizing 18 years of the Death with Dignity Act in Oregon. JAMA Oncology. 2017; 3(10):1403–6.
11. Carter v Canada (Attorney General) 2012 BCSC 886, paragraphs 359–747.
12. Victorian Government. Ministerial Advisory Panel on Voluntary Assisted Dying: Final Report. 2017, p88.
13. Western Australian Government. Ministerial Expert Panel on Voluntary Assisted Dying: Final Report. 2018, p26.
14. Royal Society of Canada Expert Panel. End-of-Life Decision Making. 2011, pp84–88.
15. Parliament of Victoria, Inquiry into End of Life Choices: Final Report. 2016, pp xxviii, 212–13.
16. Parliament of Western Australia. My Life, My Choice: The Report of the Joint Select Committee on End of Life Choices. 2018, pp175–181 (Finding 42).
17. Lowe MP, Downie J. Does Legalization of Medical Assistance in Dying Affect Rates of Non-assisted Suicide?. Journal of Ethics in Mental Health. 2017; 10:1–9
18. Council of Canadian Academies. The State of Knowledge on Medical Assistance in Dying Where a Mental Disorder Is the Sole Underlying Medical Condition. 2018, p96.
Contact diana@nzma.org.nz
for the PDF of this article
Assisted dying and evidence-based law-making: a critical analysis of an article’s role in New Zealand’s referendum | OPEN ACCESS
View Article PDF
Assisted dying (AD) is a significant social policy issue, so reliable evidence to inform law-making is critical.1 Society expects health policy2 and healthcare3 to be evidence-based. This expectation should extend to law-making.1
In September 2020, New Zealanders will participate in a binding referendum to determine whether the End of Life Choice Act 2019 (EOLCA)—legislation permitting AD already passed by Parliament—will come into force.4 A recent article published in the New Zealand Medical Journal by Winnington and MacLeod (WM article)5 raised concerns about the potential impact of this law. Further, the WM article proposed that the “evidence from this study must be factored into the New Zealand debate before the referendum on the [EOLCA]” (p21).
This article responds to the suggestion that the WM article be evidence in this debate. Our goal is to undertake a critical analysis of the evidence it purports to contribute, to determine its reliability and therefore probative value. We conclude that a combination of the nature of the study, flaws in its research design, its use of data to draw conclusions, a failure to outline limitations and inadequate engagement with social science literature make the WM article unreliable. It should not be considered as evidence in deliberations about the EOLCA.
Methods
Criteria to assess reliability
We applied three criteria to assess reliability. The first is a pyramid of evidence developed by one of the authors (JD). This pyramid draws on existing models for assessing evidence and graphically depicts how reliability increases as it ascends the pyramid, but is adapted for the evidence most commonly used when making law about AD and the sort of external review that can occur (Figure 1). Reliability is assessed both by determining the level of the pyramid at which particular evidence sits, and by considering whether it has been externally tested and the nature of that testing—whether by peer review or review by courts, non-partisan parliamentary committees or expert panels.
The second criterion is the principles that guide the conduct of social science research. We are conscious of different approaches in quantitative and qualitative research6 but there is broad acceptance across different research traditions of the following principles: study design (including data collection and analysis) that is appropriate to research aims; rigorous use of selected research methods; and fair presentation of results including only conclusions sustained by data and acknowledging appropriate limitations.
The third criterion for reliability was whether factual claims made in the WM article were defensible in light of available social science literature. We inquired whether factual claims were based on literature that was reliable (informed by some of the above considerations), up to date and fairly represented the field’s state of knowledge.
Review process
Two authors (BW and LW) reviewed the WM article multiple times and compiled a list of possible failures to meet the reliability criteria. These were grouped according to issue type and written up. These critiques were reviewed by other authors with an invitation to add new areas, revise or remove existing ones. All authors endorse the critical analysis below.
Results
Summary of Winnington and MacLeod article
The stated aim of the WM article was to consider “the possibility of consequences … for families left behind, communities and society as a whole” should New Zealand legalise AD (p18). To address this aim, a single semi-structured interview was undertaken with a person who had experience with AD in a country where AD is lawful. The interviewee’s perspective was from being married to a sibling of the person who chose AD. Thematic analysis of this interview identified three key themes: potential expectations that people would seek AD when unwell and possibly facing a life-threatening illness; stigma for individuals using AD and their families; and the potential for AD legislation to produce contagion (not defined in the article but we understand refers to the notion that AD may activate others to seek AD who would not otherwise do so) (p18).
The article calls for further research including “to investigate whether a contagion effect of AD is possible (or even probable)” (p22).
Criterion 1: reliability pyramid
As research based on a single case study, this study falls into the second bottom row of the pyramid (Figure 1). On this basis alone, the reliability of such evidence is limited. In addition, some conclusions or claims are not based on the data or go beyond what the data could support (see examples below), and would fall to the bottom row: anecdotes and opinion.
Figure 1: Adapted reliability pyramid of evidence for assisted dying.
Although the nature of the study means it is of limited reliability, some external testing adds reliability; the article is published in the peer-reviewed New Zealand Medical Journal.
Criterion 2: reliability in terms of principles of social science research
Study design not appropriate for aims
The article’s aim was to consider “the possibility of consequences … for families left behind, communities and society as a whole” should New Zealand legalise AD (p18). In terms of study design, a single interview is not capable of meeting this ambitious aim, even with the qualifier “possibility”. This is particularly so in relation to consequences for the wider communities and society as a whole. Such a method might shed light on family experiences, albeit in a very limited way with only one interview, but it cannot reliably inform about broader community perspectives.
Results extended beyond what data reasonably supports
The WM article’s analysis identified three key themes (abstract, p18):
• “the potential for assisted dying becoming an expectation for others to pursue when unwell and possibly facing a life-threatening illness;”
• “the notion of stigma being associated with the individual using assisted dying legislation and the family left behind;” and
• “that there may be the potential for such legislation to produce a contagion effect.”
We accept the interviewee discussed these three issues. But a threshold concern is whether all themes are supported by the data. It is possible that an interview with a member of an extended family with an experience of AD could, with relevant limitations articulated, produce findings about stigma within a family.
However, it is not justifiable to present the other themes as results from these data. Both relate to wider, societal-level trends rather than individual experience. The experience of one person could not, from a scientific perspective, be reliable evidence of wider community views or experiences. Determining the existence of these phenomena would require quite different research methods such as community surveys or, in the case of the contagion argument, quantitative studies about use of AD.
A second concern is how these results are extended and transformed into substantive potential problems for society. In other words, an issue is raised in the data and unjustifiably elevated to a level beyond which can be safely done from the data. We provide three examples.
Example 1: Fracturing society. The WM article expresses “concern that the legislation for AD in New Zealand has potential to fracture family and community structures” (p21). This statement is followed by the interviewee suggesting that “fractured families” make it more likely, without “support of their family”, to seek AD (p21). The first problem is that there is no evidence in the quote that the interviewee was basing his comment on experience within his family (as opposed to speculation about what might happen). The second problem is that the authors’ claim—that AD legislation could fracture family and community structures—is different from the interviewee’s point, namely that those without family support (ie, in already fractured families) may be more likely to seek AD. Yet, the segue is made from one to the other. The third problem is the shift from families (as stated by the interviewee), to the “potential to fracture family and community structures” (emphasis added) (p21), and then to the even broader claim of “potential of fracturing of our New Zealand communities and broader social settings” (emphasis added) (p21). This progression involves a significant shift from the data (a family perspective) to the wider community level.
Example 2: AD contagion. The article itself noted that the interviewee only “hinted” (p21) at this issue. This was based on the interviewee having known “three extended family members use the [AD] legislation over a short period of time” as well as AD being chosen by two of his father’s friends. However, nothing about contagion can be drawn from his quotes—there is no evidence that the three extended family members or two friends of his father who had AD knew each other or knew that each other had had AD. Furthermore, the interviewee worries that AD may be “infectious” (p21) but does not link this to the experience of his extended family member receiving AD (essential for the case study method). It could just be a pre-existing fear unrelated to the case study.
Even more concerning is that this discussion of contagion, which comprised only three paragraphs in the Results section, was transformed into a substantive concern. The shift during the article is noteworthy: from the interviewee hinting at the issue, to the conclusion calling for further research to investigate “whether a contagion effect of AD is possible”, and then “(or even probable)” (emphasis added) (p22).
Example 3: “Slippery slopes”. The WM article claims the data support slippery-slope arguments. Because of this claim’s significance, two key sentences are extracted in full here: “In conducting this study, it was anticipated that social consequences of AD legislation may be present in terms of the slippery-slope discourse. However, it was unexpected to obtain data that painted a distinct picture of how the slippery-slope effect was unfolding in a country where AD was legal” (p21, endnotes omitted). The interviewee noted a view that there may be an expectation on people to use AD, and we understand this to be the sense in which the term slippery slope is used. (We note, however, that this is not what the “slippery-slope effect” in the “slippery-slope discourse” generally means—even in the literature the authors reference for this claim).
Claims that a “distinct picture” (p21) of this occurring (an expectation that people use AD) are unsustainable. We note the many empirical studies which rebut the common slippery-slope claim that the vulnerable are more likely to seek AD.7–10 These studies include large meta-analyses or population-level studies (and so are at or towards the top of the reliability pyramid). They have been peer reviewed, and many have also been the subject of further external testing by courts,11 expert panels12–14 and non-partisan parliamentary committees.15,16 When this sort of research is placed beside the WM article, it is not reasonable to consider this single interview as reliable evidence of the “slippery-slope effect” (p21).
Appropriate statements of limitations of research
The WM article does not have a sufficiently robust statement of limitations. It acknowledges that it is based on a single interview, and that this interview was conducted in a country other than New Zealand (where AD is legal). However, the implications of this latter point are not identified, namely that findings from this single interview are not generalisable to other countries with different healthcare, social welfare, and legal systems and AD models. To illustrate, it is unclear if AD in the case study would have been available under the EOLCA, as no mention is made of an eligible terminal illness (indeed, the description at p19 makes it highly unlikely). The WM article also fails to note that a person’s pre-existing views about AD may affect their assessment of their experience. But perhaps the most significant limitation omitted was to make clear that views expressed in a single interview cannot support claims about wider societal effects such as community expectations to die and AD contagion. Indeed, instead of noting such limitations, these data from a single interview were explicitly used to ground such claims.
‘Generation’ of evidence through discussion of issues on which no data is reported
Finally, conclusions were drawn where there were no data to support such findings. While consideration of related issues may occur in an article, particularly in Background or Discussion sections, a study’s results and conclusions must be grounded in data.
The WM article at times identifies a concern about AD raised in literature (often without mentioning conflicting literature: see below). It then discusses that issue as a concern of substance, but this occurs without supporting data from the interview. One example is the claim that AD will be shaped by financial drivers. The article notes a potential consequence of AD legislation that it “reduces our future existence to being considered only through the practical lens relating to the cost of care and reduces our life to having a dollar value” (p21). This is revisited in the Conclusion: “… this case study offers insight into some elements associated with slippage [reference to slippery slope in sentence preceding] in terms of family members being expected to die when their care becomes too difficult or expensive” (p22). This significant (and very controversial) issue was not present in the data reported so its inclusion in the article’s Conclusion is not justifiable.
Criterion 3: Reliability evidenced by factual claims being defensible in light of literature
Authors must ensure that factual claims are defensible having regard to the weight of reliable literature and the field’s current state of knowledge. Some claims in the WM article cannot be defended in this way. We are not able to comprehensively catalogue all such concerns here. However, we provide one example of a claim that is not defensible and is also presented in a misleading way. The relevant passage appears in Background: “Despite the potential for those using AD legislation to be judged or stigmatised, there is further concern that AD may produce a contagion effect.13–15 Jones and Paton observed that unlike some studies that perceived AD as providing a suicide-inhibiting effect, their results suggested that any inhibitory mechanisms were counteracted by ‘equal or larger opposite effects’16–18” (p19, WM endnotes in Figure 2).
Figure 2: Selected references as cited by Winnington and MacLeod article.
A preliminary point is why, when the article claims to focus on AD contagion (ie, cases of AD leading to more AD cases), it shifts to engage with literature on the different issue of suicide contagion (ie, legalising AD leading to an increased suicide rate). Further, its suggestion that suicide contagion is a credible concern (later leveraged in potential concerns about AD contagion) is not supported by the literature, nor does the article engage with the current state of knowledge.
For instance, there is a later article by Lowe and Downie17 which critically analyses the primary source relied upon for suicide contagion (Jones and Paton), but this was not considered or even acknowledged in the WM article. Lowe and Downie identify significant errors in the Jones and Paton methodology and concerns about how the results were presented, and urge caution in relation to its findings.17 While not all literature can be cited, it is concerning that an article which has been the subject of a detailed critical analysis is presented without qualification. Also missing was the report from a major review of the state of evidence in relation to various aspects of AD undertaken by the Council of Canadian Academies (comprised of experts both in favour of AD and opposed). It concluded, including after considering the two papers above: “There is no evidence of any association between the legal status of assisted dying in a country and its suicide rate”.18 In short, there is no reliable evidence that suicide contagion will occur if AD is legalised.
A further concern is that the WM article references (Figure 2) are misleading, creating the perception that six references support the suicide contagion proposition in some way. In fact, only one study purports to consider suicide contagion (Jones and Paton: reference 16, although as noted above, Lowe and Downie argue it does not address this concept and should not be relied upon17). References 13 and 14 address potential stigma of AD (presumably a reference for the first half of the sentence), while reference 17 is Posner’s book, which includes a claim that AD may reduce rates of suicide (contrary to suicide contagion). Reference 18 appears to be to Hansard (not a study but rather a statement in parliament) although incomplete citation details mean we cannot locate what the parliamentarian said. Reference 15 refers to suicide contagion (incorrectly according to Lowe and Downie17) but is not an independent study, rather just a commentary on the Jones and Paton article.
There are other concerns about engagement with literature that could be raised. One is using literature which draws on anecdotal evidence (at the bottom of the reliability pyramid): both the Kheriaty (reference 15) and Hendin and Foley (reference 27) articles are relied upon by the WM article but they are in turn merely reporting on single cases they read about in newspaper reports. Another is not engaging with the large body of social science literature (and findings of expert panels, non-partisan parliamentary committees and courts) about “slippery slopes” (discussed above), and also being unclear about what is meant by this concept. However, as mentioned, there is not scope to include these more detailed analyses here.
Discussion
The New Zealand public will shortly decide whether AD should become lawful. Evidence-based law-making, including through a referendum, is critical, especially for significant social policy reform such as AD. This requires critical review of evidence proffered to inform public debates and public decision-making about AD.
The WM article proposed it be considered as evidence in the deliberations about AD in New Zealand. Our analysis has concluded, however, that the article is not reliable evidence and should not form part of these deliberations. It is based on a single interview with a person (the brother-in-law of a person who accessed AD) from an unidentified country where AD is legal (not New Zealand). This methodology is not capable of supporting the article’s significant claims, in particular about potential expectations that people when unwell and facing a life-threatening illness should use AD and the potential of AD contagion. In addition, the WM article presents assertions beyond what its very limited data can sustain and indeed, makes claims for which there is no data in support. Further, the study fails to appropriately limit the scope of its findings; indeed, it makes claims beyond what is justified. Finally, its reliability can also be questioned as not all of its factual claims can be supported by the social science literature.
We conclude by repeating our call for evidence-making law-making on the critical social policy issue of AD.1
Summary
Abstract
Aim
To critically analyse the reliability of an article which claims to be evidence that the End of Life Choice Act 2019 provides a “potential hotspot for family, community and social discord that may not be easily remedied” should the legislation receive public support in New Zealand’s September 2020 referendum.
Method
The subject article was reviewed multiple times by all authors and critiqued against three criteria: a reliability pyramid developed to weigh evidence about assisted dying; principles that guide the conduct of social science research; and the use of reliable and current social science literature to support factual claims.
Results
The study being analysed involved a single interview and so is located at the second bottom row of the reliability pyramid. Its research design is also unable to support the broad findings that are asserted. Other flaws in method included findings being extended beyond the data, and failure to state appropriate limitations in the research method. Further, claims are made that are unsupported by the weight of reliable social science literature.
Conclusion
The subject article is methodologically and factually flawed so is unreliable as evidence. It should not be considered in the assisted dying debates preceding the forthcoming referendum.
Author Information
Acknowledgements
Correspondence
Correspondence Email
Competing Interests
1. White BP, Willmott L. Evidence-based law making on voluntary assisted dying. Aust Health Rev. 2020. http://doi.org/10.1071/AH19201
2. Baicker K, Chandra A. Evidence-based health policy. N Engl J Med. 2017; 377:2413–15.
3. Sackett DL, Rosenberg WM, Muir Gray JA, et al. Evidence based medicine: what it is and what it isn’t. BMJ. 1996; 312:71.
4. New Zealand Government. End of Life Choice referendum. http://www.referendums.govt.nz/endoflifechoice/index.html
5. Winnington R, MacLeod R. Social consequences of assisted dying: a case study. 2020. NZMJ; 133:18–23.
6. Yin, RK. Qualitative Research from Start to Finish. 2nd edition. New York: Guilford Press; 2016.
7. Battin MP, van der Heide A, Ganzini L, et al. Legal Physician-Assisted Dying in Oregon and the Netherlands: Evidence Concerning the Impact on Patients in ‘Vulnerable’ Groups. Journal of Med Ethics. 2007; 33(10):591–7.
8. Rietjens JAC, Deschepper R, Pasman R, Deliens L. Medical End of life Decisions: Does its Use Differ in Vulnerable Patient Groups? A Systematic Review and Meta analysis. Soc Sci and Med. 2012; 74(8):1282–87.
9. Emanuel EJ, Onwuteaka-Philipsen BD, Urwin JW, Cohen J. Attitudes and Practices of Euthanasia and Physician-Assisted Suicide in the United States, Canada and Europe. JAMA. 2016; 316(1):79–90.
10. Blanke C, LeBlanc M, Hershman D, et al. Characterizing 18 years of the Death with Dignity Act in Oregon. JAMA Oncology. 2017; 3(10):1403–6.
11. Carter v Canada (Attorney General) 2012 BCSC 886, paragraphs 359–747.
12. Victorian Government. Ministerial Advisory Panel on Voluntary Assisted Dying: Final Report. 2017, p88.
13. Western Australian Government. Ministerial Expert Panel on Voluntary Assisted Dying: Final Report. 2018, p26.
14. Royal Society of Canada Expert Panel. End-of-Life Decision Making. 2011, pp84–88.
15. Parliament of Victoria, Inquiry into End of Life Choices: Final Report. 2016, pp xxviii, 212–13.
16. Parliament of Western Australia. My Life, My Choice: The Report of the Joint Select Committee on End of Life Choices. 2018, pp175–181 (Finding 42).
17. Lowe MP, Downie J. Does Legalization of Medical Assistance in Dying Affect Rates of Non-assisted Suicide?. Journal of Ethics in Mental Health. 2017; 10:1–9
18. Council of Canadian Academies. The State of Knowledge on Medical Assistance in Dying Where a Mental Disorder Is the Sole Underlying Medical Condition. 2018, p96.
Contact diana@nzma.org.nz
for the PDF of this article
Assisted dying and evidence-based law-making: a critical analysis of an article’s role in New Zealand’s referendum | OPEN ACCESS
Assisted dying (AD) is a significant social policy issue, so reliable evidence to inform law-making is critical. Society expects health policy and healthcare to be evidence-based. This expectation should extend to law-making.
Subscriber Content
The full contents of this pages only available to subscribers.
Login, subscribe or email nzmj@nzma.org.nz to purchase this article.
