John Cologne, Statistics Dept., Radiation Effects Research Foundation
19 November 2010
This excellent article by Turner, Dobson, and Pocock is highly welcomed and long overdue. A couple of points made in the article will be obvious to most statisticians, but might benefit from some clarification for persons with less sophisticated statistical training or experience.
First of all, one example is given where a null hypothesis of no effect could be rejected with two groups but not four. This is counter-intuitive to the statement elsewhere in the article that "dichotomies should be avoided". The reason is because it depends on the underlying, unobserved true effect. How many groups and where to place cutpoints depends heavily on the shape of the effect. I have encountered situations where it was possible for data with a strong linear relationship to demonstrate a significant two-group difference but not global heterogeneity with a large number of groups. Hence the conclusion by the authors that trend tests can sometimes be more powerful. An important question is when trend tests are more powerful: since this depends on the unknown underlying effect, who knows?!
In regards to that point, it is said that a trend test will be substantially more powerful when "the relationship between the risk factor and outcome appears to be monotonic". Naturally, if we test what we see appearing in the data, it will be a powerful test; indeed it will inflate the significance of the test. Unless purely for descriptive purposes, trend tests should be based on subject matter consideration, not on review of the data.
The fact that some case-control studies based quantiles on controls only may be due to the desire to base cutpoints on the population distribution of the risk factor, as the controls are more representative of the general population for a rare outcome and strong risk factor given the over-sampling of cases. However, there are instances, such as a counter-matched nested case control study, where basing cutpoints on the distribution of risk factor in the cases can be beneficial in terms of efficiency.
Finally, although the authors mention it elsewhere in their article, they do not include among their recommendations the need to report all categorizations assessed and analyzed, not just those whose results are reported. Thus, I would add recommendation #14: "Report and describe all categorizations attempted or analyzed, even if the results were rejected and not presented in the paper, and state why the chosen method was preferred". This will reveal dredging expeditions and allow readers to better assess the inferential merits of the reported results.
a few clarifications
19 November 2010
This excellent article by Turner, Dobson, and Pocock is highly welcomed and long overdue. A couple of points made in the article will be obvious to most statisticians, but might benefit from some clarification for persons with less sophisticated statistical training or experience.
First of all, one example is given where a null hypothesis of no effect could be rejected with two groups but not four. This is counter-intuitive to the statement elsewhere in the article that "dichotomies should be avoided". The reason is because it depends on the underlying, unobserved true effect. How many groups and where to place cutpoints depends heavily on the shape of the effect. I have encountered situations where it was possible for data with a strong linear relationship to demonstrate a significant two-group difference but not global heterogeneity with a large number of groups. Hence the conclusion by the authors that trend tests can sometimes be more powerful. An important question is when trend tests are more powerful: since this depends on the unknown underlying effect, who knows?!
In regards to that point, it is said that a trend test will be substantially more powerful when "the relationship between the risk factor and outcome appears to be monotonic". Naturally, if we test what we see appearing in the data, it will be a powerful test; indeed it will inflate the significance of the test. Unless purely for descriptive purposes, trend tests should be based on subject matter consideration, not on review of the data.
The fact that some case-control studies based quantiles on controls only may be due to the desire to base cutpoints on the population distribution of the risk factor, as the controls are more representative of the general population for a rare outcome and strong risk factor given the over-sampling of cases. However, there are instances, such as a counter-matched nested case control study, where basing cutpoints on the distribution of risk factor in the cases can be beneficial in terms of efficiency.
Finally, although the authors mention it elsewhere in their article, they do not include among their recommendations the need to report all categorizations assessed and analyzed, not just those whose results are reported. Thus, I would add recommendation #14: "Report and describe all categorizations attempted or analyzed, even if the results were rejected and not presented in the paper, and state why the chosen method was preferred". This will reveal dredging expeditions and allow readers to better assess the inferential merits of the reported results.
Competing interests
No competing interests.