[Note: in an earlier version of this post I scrutinized the term “significant estimates” at the bottom of the risk tables in the original report for public consultation. The reference to “significant” estimates” was missing in the final report and I thought this was odd. However, I have learned that the term “significant estimate” means that the data are based upon estimates that are statistically significant, not that they are really big reaches of logic. So I have deleted it since it seemed unfair to include it as a critique when it appears the CCSA was trying to be transparent. Why this is missing from the final report, however, is up for speculation]
You know that when you are about to sign a contract, standard advice is “read the fine print.” It’s to your own benefit. You’re about to sign something (*a job contract, lease, rental agreement, etc) that affects your behaviour or life, and without understanding what you’re getting yourself into, you could be in trouble.
The problem with this advice is that often it is written in the language of a discipline that you may not understand. This includes “Legalese”, Jargon, acronyms, complex terminology that you may not quite comprehend.
Scholars also have coded language all their own, and different disciplines have standards and terms that may have been designed as short forms that everyone within the field understands, but can baffle people from outside. Whenever I’m having conversations with my colleagues in Health Sciences (who come from a range of disciplines) I am finding myself stopping them and asking what they mean. (This is how I learned what “knock out mice” are and what a “wicked problem” is.)
Despite the jargon, reading the fine print is important. In research, often the fine print comes in two forms: the notes and the limitations.
In history, for example, we often use endnotes or footnotes to keep the citations from interrupting the flow of an argument. (This is not exclusively a history thing; in fields where narrative and rhetoric is an important part of explaining research–philosophy, literature, etc–notes may be preferred). Notes are also useful for adding additional information that does not fit in the paragraph, but may add context or support to the information provided. These are called “discursive” notes, because they do more than cite.
In scientific papers, good researchers recognize “limitations” in their work. They recognize that a study they ran missed certain demographics of the population, or was done under unique conditions, etc. It is a standard practice–and important–for people to frame the conclusions within their proper context. It also is a way of calling on other researchers to undertake research that fills in the gaps noted in those limitations. If a study ended up looking only at white men, for example, it would be good to do research that looked at women, racialized people, etc.
(It is also as an aside, clever because you can use those limitations to justify new funding requests to continue the research to address those limitations. This is not devious, it is the nature of the scholarly funding ecosystem: if there is no demonstrated need for a project to be funded, it won’t get funded.
When the Canadian Centre on Substance Use and Addiction released its low risk guidelines for public consultation (consider it a draft of the final report) it did not mention the limitations of the research. Although the research upon which the CCSA based its guidelines did include limitations, the consultation report did not. The only time limitations are mentioned was when they explain the limitations to their 2011 guidelines, and then the limitations to some rapid reviews they undertook to answer questions such as about the connections between mental health and alcohol (which they found to be inconclusive).
Discussing limitations in this way is not surprising, unusual, or necessarily problematic. Just as I noted above, research reports often conclude with limitations so that other researchers can pick up the baton, so to speak, and run with it.
However, another way you can discuss limitations is to soften or dismiss the evidence in research that does not show what you are looking for (this is not a good way to do research, since the only thing you should be looking for is clarity: testing a hypothesis, examining a research question etc.). This is why, if you run a study that does not have good outcomes, shows wishy-washy conclusions, and could be taken in different ways, you can say “well, there were a lot of limitations to this research” as a way of saying “nothing to see here, move along.”
So that is one thing to consider. The other thing is the appearance, finally, of limitations to the report.
The limitations to the report should be read carefully. They run a page and a half, and really should give anyone pause before enthusiastically adapting the guidance as “fact.” What is interesting to me is how the CCSA presents limitations then uses some softening language to try to get out of a bind.
For example (p46):
This is the key limitation: there are confounding variables that are not considered. Whereas they think of variables like age and sex (standard demographic considerations and important), when considering something that is often done in a social context for relaxation, there are also variables that are more difficult to calculate, like social connectedness, happiness, relaxation, stress relief, etc. But the important thing about this limitation is that it is a really big deal, and just mentioned as an aside.
(The “softening language” follows the “however” at the bottom: it says don’t worry about this [very important] limitation, because it’s probably not a problem. Trust me.”)
Second example, (p47):
This is a HUGE qualifier to put into a limitations statement. I’m glad they did.
The results do not “account for risk and protective factors at the community, familial or individual levels” (emph mine)?
Two things: first, I applaud them for recognizing there may be protective factors; up to this point protective factors of drinking were argued away.
Second: “Community, familial, or individual levels”? What other levels are there? Well, the population level. So the outcome of drinking is expressed as an average: If I drink 5 drinks a week and someone else drinks 50 drinks a week, we average to 27.5 drinks a week. We both become, on average, heavy drinkers. Mathematics made me into a drunk.
Another limitation, that needs some attention: (p47)
So much going on here. First “health conditions that result in death” suggests that death is avoidable. All states of being result in death. I think. Correct me if I’m wrong here. (I think they mean premature death).
Second, this is another great illustration of how some researchers look at drinking and see only a problem. They say “Future alcohol guidelines should aim to incorporate other social issues” and then gives a list of negative social issues. Yet again, the researchers are dismissing protective factors and the potential positive value drink can provide in life. If you assume drinking is just bad, then this paragraph makes sense. I do not, and it does not.
I’m sure that the CCSA would disagree with this assessment. My point here is to indicate notable areas of weakness that should cause people – including governments and public health agencies – to question the guidance.. As so much of what I have said already has indicated, taking drinking out of its social context, running analyses that do not recognize social factors that cannot be broken into statistical bits (as you can do with age, sex, etc), and then making big conclusions that is based on averages, is deeply problematic. The resulting distortions serves a (say it with me) neo-temperance agenda, and can cause more harm than good.
(c) 2023 Dan Malleck
Drug History Canada 