You know that when you are about to sign a contract, standard advice is “read the fine print.” It’s to your own benefit. You’re about to sign something (*a job contract, lease, rental agreement, etc) that affects your behaviour or life, and without understanding what you’re getting yourself into, you could be in trouble.
The problem with this advice is that often it is written in the language of a discipline that you may not understand. This includes “Legalese”, Jargon, acronyms, complex terminology that you may not quite comprehend.
Scholars also have coded language all their own, and different disciplines have standards and terms that may have been designed as short forms that everyone within the field understands, but can baffle people from outside. Whenever I’m having conversations with my colleagues in Health Sciences (who come from a range of disciplines) I am finding myself stopping them and asking what they mean. (This is how I learned what “knock out mice” are and what a “wicked problem” is.)
Despite the jargon, reading the fine print is important. In research, often the fine print comes in two forms: the notes and the limitations.
In history, for example, we often use endnotes or footnotes to keep the citations from interrupting the flow of an argument. (This is not exclusively a history thing; in fields where narrative and rhetoric is an important part of explaining research–philosophy, literature, etc–notes may be preferred). Notes are also useful for adding additional information that does not fit in the paragraph, but may add context or support to the information provided. These are called “discursive” notes, because they do more than cite.
In scientific papers, good researchers recognize “limitations” in their work. They recognize that a study they ran missed certain demographics of the population, or was done under unique conditions, etc. It is a standard practice–and important–for people to frame the conclusions within their proper context. It also is a way of calling on other researchers to undertake research that fills in the gaps noted in those limitations. If a study ended up looking only at white men, for example, it would be good to do research that looked at women, racialized people, etc.
(It is also as an aside, clever because you can use those limitations to justify new funding requests to continue the research to address those limitations. This is not devious, it is the nature of the scholarly funding ecosystem: if there is no demonstrated need for a project to be funded, it won’t get funded.
When the Canadian Centre on Substance Use and Addiction released its low risk guidelines for public consultation (consider it a draft of the final report) it did not mention the limitations of the research. Although the research upon which the CCSA based its guidelines did include limitations, the consultation report did not. The only time limitations are mentioned was when they explain the limitations to their 2011 guidelines, and then the limitations to some rapid reviews they undertook to answer questions such as about the connections between mental health and alcohol (which they found to be inconclusive).
Discussing limitations in this way is not surprising, unusual, or necessarily problematic. Just as I noted above, research reports often conclude with limitations so that other researchers can pick up the baton, so to speak, and run with it.
However, another way you can discuss limitations is to soften or dismiss the evidence in research that does not show what you are looking for (this is not a good way to do research, since the only thing you should be looking for is clarity: testing a hypothesis, examining a research question etc.). This is why, if you run a study that does not have good outcomes, shows wishy-washy conclusions, and could be taken in different ways, you can say “well, there were a lot of limitations to this research” as a way of saying “nothing to see here, move along.”
We will come back to limitations in a second, but the other thing to look at is “fine print” often in the form of notes to explain or contextualize information
On its vividly coloured risk tables in the consultation document, (reproduced below) there is an interesting, though not entirely clear, note. Check it out.
Do you see the note? “Bolded percentages indicate significant estimates.”
Given the stark information that the CCSA is trying to provide and the seriously restrictive guidelines they are recommending, I think it is something when you admit that most of the data are “significant estimates.”
However, the language is unclear. Does this mean the estimates are “statistically significant” (which means mathematical tests were done to show they are genuinely powerful and convincing) or that it is significant that these are estimates, not, indeed, “hard” evidence. To me “significant estimate” means a big estimate and a bit of a reach (and given the limitations about population-level data we will discuss below, i think this is an accurate interpretation)
It is difficult to emphasize what a big deal this note is. It’s as if I wrote an article about something Sir John A Macdonald said, then in a note said “I’m not sure if it was Sir John, or just another John Macdonald who said it.” It could completely upend the meaning and strength of my argument.
As you might imagine, that little note should be the cause of a lot of caution. If most of your data is an estimate, and yet you are asking us to significantly alter our lives and our approach to drinking, I’d like something more convincing than an estimate, especially one you call “significant.”
Interestingly, this note was not included in the final report (January 2023); the bolded data disappeared and the tables were recalculated to show drinks per week rather than grams of alcohol (that was wise because most people can’t calculate grams of alcohol in their heads). It also expanded the number of drinks in the columns to 1-7, 14, 21 and 35 (man, that’s a lot of booze) which could be a distortion but I’ll ignore it for now.
I don’t know if this means that this no longer represents “significant estimates” but I doubt it. I’m guessing the note caused confusion and was removed.
In the final report they also got rid of the death ratios and rearranged the illnesses. Rather than placing at the top something for which alcohol was protective (ischemic heart disease) and which is responsible for more deaths than any other single condition in the list, they put it in the middle and removed the information that shows how big a deal it is. Again: more people die of heart disease than any other single condition on that table, and alcohol by the CCSA’s own data, is protective at moderate levels.
So that is one thing to consider. The other thing is the appearance, finally, of limitations to the report.
The limitations to the report should be read carefully. They run a page and a half, and really should give anyone pause before enthusiastically adapting the guidance as “fact.” What is interesting to me is how the CCSA presents limitations then uses some softening language to try to get out of a bind.
For example (p46):
This is the key limitation: there are confounding variables that are not considered. Whereas they think of variables like age and sex (standard demographic considerations and important), when considering something that is often done in a social context for relaxation, there are also variables that are more difficult to calculate, like social connectedness, happiness, relaxation, stress relief, etc. But the important thing about this limitation is that it is a really big deal, and just mentioned as an aside.
(The “softening language” follows the “however” at the bottom: it says don’t worry about this [very important] limitation, because it’s probably not a problem. Trust me.”)
Second example, (p47):
This is a HUGE qualifier to put into a limitations statement. I’m glad they did.
The results do not “account for risk and protective factors at the community, familial or individual levels” (emph mine)?
Two things: first, I applaud them for recognizing there may be protective factors; up to this point protective factors of drinking were argued away.
Second: “Community, familial, or individual levels”? What other levels are there? Well, the population level. So the outcome of drinking is expressed as an average: If I drink 5 drinks a week and someone else drinks 50 drinks a week, we average to 27.5 drinks a week. We both become, on average, heavy drinkers. Mathematics made me into a drunk.
Another limitation, that needs some attention: (p47)
So much going on here. First “health conditions that result in death” suggests that death is avoidable. All states of being result in death. I think. Correct me if I’m wrong here. (I think they mean premature death).
Second, this is another great illustration of how some researchers look at drinking and see only a problem. They say “Future alcohol guidelines should aim to incorporate other social issues” and then gives a list of negative social issues. Yet again, the researchers are dismissing protective factors and the potential positive value drink can provide in life. If you assume drinking is just bad, then this paragraph makes sense. I do not, and it does not.
I’m sure that the CCSA would disagree with this assessment. My point here is to indicate notable areas of weakness that should cause people – including governments and public health agencies – to question the guidance.. As so much of what I have said already has indicated, taking drinking out of its social context, running analyses that do not recognize social factors that cannot be broken into statistical bits (as you can do with age, sex, etc), and then making big conclusions that is based on averages, is deeply problematic. The resulting distortions serves a (say it with me) neo-temperance agenda, and can cause more harm than good.
(c) 2023 Dan Malleck
Drug History Canada 