Skip to main content

Archived Comments for: Estimating uncertainty in observational studies of associations between continuous variables: example of methylmercury and neuropsychological testing in children

Back to article

  1. Extreme assumptions are unnecessary

    Esben Budtz-Jørgensen, DEPT. OF BIOSTAT, UNIVERSITY OF COPENHAGEN

    14 November 2007

    Authors: Esben Budtz-Jørgensen and Niels Keiding (Department of Biostatistics, University of Copenhagen, Denmark)

    Philippe Grandjean (University of Southern Denmark, Odense, Denmark; and Harvard School of Public Health, Boston, MA, USA)

    The paper by Goodman et al. (1) aims at modeling the impact of extreme uncertainties on the results of an observational study, in this case the risk of developmental neurotoxicity associated with maternal dietary exposure to methylmercury during pregnancy. Thus, the paper rests on an apparent disagreement between the findings of epidemiological studies carried out in the Seychelles and the Faroe Islands. However, we have previously shown that the results of the Seychelles study are sufficiently variable to be in statistical agreement with a mercury-associated deficit on the Boston Naming Test reported from the Faroe Islands (2). Goodman et al. could have reached the same conclusion, had they converted the cord blood regression coefficient from the Faroes study to the hair concentration scale (multiply by 5) for comparison with the regression coefficient of the Seychelles study. A combination of the two estimates would generate a statistically significant effect, as was also reported by Axelrad et al. (3) based on a meta-analysis of the studies in New Zealand, Seychelles, and Faroes.

    The analyses of the Seychelles and the Faroe Island data have included a large number of covariates. The mercury effect estimate originally reported from the Faroes (4) changed only negligibly by including a large number of additional covariates (5,6) including many of those Goodman et al. consider important. The analyses carried out by Goodman et al. fail to take into account the effects of the covariates already adjusted for, and the calculations erroneously indicate that an unbiased effect estimation can be achieved without knowing the relationship between the confounders present and the unmeasured confounder. The authors further assume that the unmeasured confounder has an unrealistically high correlation with both exposure and outcome variable. It is no surprise that the result of an epidemiological study is sensitive to assumptions about such a very strong confounder. Assumptions on information bias and selection bias are also extreme.

    Recent evidence suggests that the previously published analyses of the data may have underestimated the effect of mercury exposure. We have examined the confounding effect from beneficial influences of fish nutrients and found that adjustment for maternal fish intake during pregnancy leads to an increase in the estimated mercury toxicity (7). Preliminary data from the Seychelles suggest that adjustment for beneficial effects of fish intake in that population will also result in mercury-associated deficits becoming statistically significant (8), in accordance with the findings from the Faroes. We have previously argued that the apparent difference between population studies in New Zealand, Seychelles, and Faroes, is likely due to uncertainties, with little reason for controversy (9). This was confirmed by the meta-analysis of Axelrad et al. (3).

    Adjustment for imprecision in covariates and the confounding effects of fish intake would likely make the studies even more similar and result in stronger effect estimates.

    The calculations presented by Goodman et al. (1) are therefore of little relevance to our current understanding of developmental neurotoxicity caused by maternal methylmercury exposure.

    Email: Esben Budtz-Jørgensen* - E.Budtz-Joergensen@biostat.ku.dk ; Niels Keiding - nk@biostat.ku.dk; Philippe Grandjean - PGrandjean@health.sdu.dk.

    * Corresponding author

    References

    1.Goodman M, Barraj LM, Mink PJ, Britton NL, Yager JW, Flanders WD, Kelsh MA: Estimating uncertainty in observational studies of associations between continuous variables: example of methylmercury and neuropsychological testing in children. Epidemiol Perspect Innov 2007, 4:9

    2.Keiding N, Budtz-Jørgensen E, Grandjean P: Prenatal methylmercury exposure in the Seychelles (letter). Lancet 2003, 362: 664-5.

    3.Axelrad DA, Bellinger DC, Ryan LM, Woodruff TJ: Dose-response relationship of prenatal mercury exposure and IQ: An integrative analysis of epidemiological data. Environ Health Perspect 2007, 115: 609-15.

    4.Grandjean P, Weihe P, White RF, Debes F, Araki S, Yokoyama K, Murata K, Sørensen N, Dahl R, Jørgensen PJ: Cognitive deficit in 7-year-old children with prenatal exposure to methylmercury. Neurotoxicol Teratol 1997, 19: 417-28.

    5.Budtz-Jørgensen E, Keiding N, Grandjean P, Weihe P: Confounder selection in environmental epidemiology: Assessment of health effects of prenatal mercury exposure. Ann Epidemiol 2007, 17: 27-35.

    6.Choi AL, Budtz-Jørgensen E, Jørgensen PJ, Steuerwald U, Debes F, Weihe P, Grandjean P: Selenium as a potential protective factor against mercury developmental neurotoxicity. Environ Res 2007 (epub).

    7.Budtz-Jørgensen E, Grandjean P, Weihe P: Separation of risks and benefits of seafood intake. Environ Health Perspect 2007, 115: 323-7.

    8.Strain JJ, Bonham MP, Davidson PW, Myers GJ, Thurston SW, Clarkson TW, Stokes-Riner A, Janciuras J, Sloane-Reeves J, Cernichiari E, Shamlaye CF, Duffy EM, Robson PJ, Wallace JMW. Long-chain polyunsaturated fatty acids and mercury (abstract 39). Presented at International Conference on Fetal Programming and Developmental Toxicity, Faroe Islands, 20-24 May, 2007 [http://www.pptox.dk]

    9.Grandjean P. Mercury Risks: Controversy or just uncertainty? Publ Health Rep 1999, 114: 512-5.

    Competing interests

    No competing interests

  2. rejoinder

    Esben Budtz-Jørgensen, Dept. of Biostat., University of Copenhagen

    6 December 2007

    Aurthors: Esben Budtz-Jørgensen and Niels Keiding (Department of Biostatistics, University of Copenhagen, Denmark)

    Philippe Grandjean (University of Southern Denmark, Odense, Denmark; and Harvard School of Public Health, Boston, MA, USA)

    As a rejoinder to Goodman et al.'s response, we would agree that sensitivity analyses can be useful when evaluating evidence from observational studies. However, such calculations are informative only if based on realistic assumptions about the unknown parameters. Thus, an observed exposure-response relationship can always be removed by postulating the presence of a strong unmeasured confounder. Correlations between the confounder and the exposure and response parameters should therefore be realistic and based on available evidence. In their analysis of the Faroese data, Goodman et al. assume a correlation of 0.8 between an unmeasured confounder and the BNT response. Accordingly, the confounder would explain 64% of the variation of this outcome. This assumption is extreme. In our recent publication (1), we included an extended set of 20 potential confounders, but, together with mercury exposure, they accounted for less than 24% of the variation. In their response, Goodman et al. fail to comment on their misleading assumption. Rather than raising questions on the validity of the Faroes results, their study suggests that unrealistic assumptions are required to explain away the significant impacts of developmental methylmercury exposure.

    References

    1.Budtz-Jørgensen E, Keiding N, Grandjean P, Weihe P: Confounder selection in environmental epidemiology: Assessment of health effects of prenatal mercury exposure. Ann Epidemiol 2007, 17: 27-35.

    Competing interests

    None declared

  3. Second response by M. Goodman, L. Barraj, P. Mink and D. Flanders

    Michael Goodman, Emory University School of Public Health

    6 December 2007

    It is not correct that we “assumed a correlation of 0.8 between an unmeasured confounder and the BNT response.” Rather we assumed a range of correlations with a minimum of 0.2 and a maximum of 0.8. Budtz-Jorgensen and colleagues suggest that unaccounted confounding did not affect their study results, thereby offering an assumption of their own. We accepted this assumption and re-ran the sensitivity analyses to correct only for information and selection bias. The results were as follows.

    For the Faroe Islands Study (FIS) after adjusting for information and selection bias the slope (SE) changed from -0.0190 (0.0063) as originally reported by the FIS group to -0.0022 (0.0392). As a reminder, adjustment for selection, information and confounding bias in our paper resulted in a corrected FIS slope of -0.0024 (0.0439). For the Seychelles Child Development Study (SCDS) the observed slope was -0.0120 (0.0460), adjustment for information and selection bias changed the result to -0.0122 (0.2000). In our paper, further correction of SCDS results for confounding produced a slope of -0.0132 with a SE of 0.2240.

    In summary, even if one were to argue that the effect of unaccounted confounding should be ignored, it is important to remember that the overall uncertainty associated with systematic error may be a result of bias from multiple sources. Thus, even in the absence of any unaccounted confounding, our conclusions remain essentially the same.

    Competing interests

    This research was funded by the Electric Power Research Institute (EPRI), a private, independent, non-profit center for public interest energy and environmental research

  4. Re: Estimating uncertainty in observational studies of associations between continuous variables: Example of methylmercury and neuropsychological testing in children. Reply by M. Goodman, P. Mink and L. Barraj

    Michael Goodman, Emory University

    6 December 2007

    We thank Drs. Budtz-Jørgensen, Keiding and Grandjean for their comments. The following is our response to the specific concerns raised in their letter.

    1. … the paper rests on an apparent disagreement between the findings of epidemiological studies carried out in the Seychelles and the Faroe Islands. However, we have previously shown that the results of the Seychelles study are sufficiently variable to be in statistical agreement with a mercury-associated deficit on the Boston Naming Test reported from the Faroe Island.

    RESPONSE: Systematic errors are unavoidable in observational epidemiology, and it is reasonable to assume that neither the Seychelles Child Development Study (SCDS) nor the Faroe Islands Study (FIS) are immune to imperfections of study design and implementation. For this reason it is only fair that our sensitivity analyses addressed uncertainty in both studies.

    2. Goodman et al. could have reached the same conclusion, had they converted the cord blood regression coefficient from the Faroes study to the hair concentration scale (multiply by 5) for comparison with the regression coefficient of the Seychelles study. A combination of the two estimates would generate a statistically significant effect, as was also reported by Axelrad et al. based on a meta-analysis of the studies in New Zealand, Seychelles, and Faroes.

    RESPONSE: Rescaling in the Axelrad (2007) paper did not change the interpretation of the Boston Naming Test result for the SCDS study [1]. Using the original scale, the β estimate was -0.012 with a standard error of 0.046; after rescaling, the corresponding estimate changed to -0.038 (SE 0.144). Performing a meta-analysis on the two studies does not eliminate, but merely hides, the disagreement between FIS and SCDS results. The New Zealand study did not use BNT.

    3. The analyses of the Seychelles and the Faroe Island data have included a large number of covariates. The mercury effect estimate originally reported from the Faroes changed only negligibly by including a large number of additional covariates including many of those Goodman et al. consider important.

    RESPONSE: We agree that the impact of unaccounted confounding alone may be modest; this was confirmed by our sensitivity analyses. However, as shown in Figures 1 and 2 in our paper, when combined with other modest sources of bias the overall error may be substantial.

    4. The analyses carried out by Goodman et al. fail to take into account the effects of the covariates already adjusted for, and the calculations erroneously indicate that unbiased effect estimation can be achieved without knowing the relationship between the confounders present and the unmeasured confounder.

    RESPONSE: We used adjusted effect estimates in our sensitivity analyses. Our paper does not indicate that that unbiased effect estimation can be achieved without knowing the relationship between the confounders present and the unmeasured confounder. Unfortunately, we did not have information on relationship between the confounders present and the unmeasured confounder. This particular caveat of our sensitivity analyses is discussed in the article along with other limitations.

    5. The authors further assume that the unmeasured confounder has an unrealistically high correlation with both exposure and outcome variable. Assumptions on information bias and selection bias are also extreme.

    RESPONSE: The differences in exposure levels between participants and non-participants in the FIS have been reported [2, 3] and, in fact, exceed the differences assumed in our selection bias simulation. The reported participation rate in the SCDS also falls within the proposed scenarios [4]. We demonstrated the potential effect of confounding by home environment and the need for a comprehensive parental IQ evaluation in an earlier publication [5]. The correlation coefficients between potential confounders and exposure are similar to those reported in the FIS. The potential misclassification due to fatigue, timing and sequencing of testing and lack of adequate blinding also finds support in the literature [6, 7].

    6. Preliminary data from the Seychelles suggest that adjustment for beneficial effects of fish intake in that population will also result in mercury-associated deficits becoming statistically significant, in accordance with the findings from the Faroes.

    RESPONSE: The relevance of the abstract by Strain et al (2007) is unclear as it does not appear to examine the BNT results [8]. We are pleased, however, that Strain and colleagues felt it was necessary to examine uncertainty associated with previously unaccounted confounding.

    In conclusion, we would like to re-emphasize that our sensitivity analyses aimed to estimate the potential systematic error for each study and in either direction. We are well aware that sensitivity analyses that are based on assumptions may be less informative than those based on real data. Nevertheless, in the absence of sensitivity analyses, one implicitly assumes that systematic error had no effect on the study results, an assumption that is even more difficult to defend. We hope that the methods of our sensitivity analyses will prove informative and helpful to researchers who wish to examine uncertainty in their data or in published studies by other researchers. Whether or not our findings add to the “current understanding of developmental neurotoxicity caused by maternal methylmercury exposure” is, of course, a matter of opinion.

    References

    1. Axelrad DA, Bellinger DC, Ryan LM, Woodruff TJ: Dose-response relationship of prenatal mercury exposure and IQ: an integrative analysis of epidemiologic data. Environ Health Perspect 2007, 115:609-615.

    2. Grandjean P, Weihe P, White RF, Debes F, Araki S, Yokoyama K, Murata K, Sorensen N, Dahl R, and Jorgensen PJ.: Cognitive deficit in 7-year-old children with prenatal exposure to methylmercury. Neurotoxicol Teratol 1997, 19:417-428.

    3. Grandjean P, Weihe P: Neurobehavioral effects of intrauterine mercury exposure: potential sources of bias. Environ Res 1993, 61:176-183.

    4. Marsh D, Clarkson TW, Myers GJ, Davidson PW, Cox C, Cernichiari E, Tanner MA, Lednar W, Shamlaye C, Choisy O, Hoareau C, Berlin M: The Seychelles study of fetal methylmercury exposure and child development: Introduction. Neurotoxicology 1995, 16:583-596.

    5. Mink PJ, Goodman M, Barraj LM, Imrey H, Kelsh MA, Yager J: Evaluation of uncontrolled confounding in studies of environmental exposures and neurobehavioral testing in children. Epidemiology 2004, 15:385-393.

    6. Baron I: Neuropsychological Evaluation of the Child. New York: Oxford University Press; 2004.

    7. Sattler J: Assessment of Children: Cognitive Applications. 4th edn. San Diego: Jerome M. Sattler, Publisher, Inc.; 2001.

    8. Strain J, Bonham M, Davidson P, Myers G, Thurston S, Clarkson T, Stokes-Riner A, Janciuras J, Sloane-Reeves J, Cernichiari E, et al: Long-chain polyunsaturated fatty acids and mercury. Presented at the International Conference on Fetal Programming and Developmental Toxicity; Faroe Islands. 2007

    Competing interests

    This research was funded by the Electric Power Research Institute (EPRI), a private, independent, non-profit center for public interest energy and environmental research

Advertisement