Type M errors via simulation
Continuing with the above two-sample \(t\)-test structure, suppose we’re interested in
minimizing Type M errors to ensure that significant results weren’t
due to higher sampling variability, ultimately resulting in significance being raised only when
unreasonably large effect sizes estimates are observed. For the purpose of
this power demonstration, suppose “unreasonably large” is defined such
that the ratio of the absolute (standardized) mean difference in a two-sample \(t\)-test,
which was flagged as significant, was larger than three times the value of
the true (standardized) mean difference (hence, M.ratio = 3).
Note that while focusing on an M.ratio = 3 provides the sufficient means to study
Type M errors using a cut-off logic to define power, it is also possible to store
the observed \(M\) values for further inspection too, which can be done
in Spower if a list or
data.frame is returned from the supplied experiment. For this to behave correctly,
however, output information relevant to the power computations
(probability values/logicals) must be explicitly specified using
Spower(..., select) so that other values returned from the simulation are
stored but not summarised.
As before, the first step is to define the experiment using the conditional
\(p\)-value logic nested within a while() loop, followed by the power (and extra) criteria of interest.
l_two.t_typeM <- function(n, mean, mu = 0,
alpha = .05, M.ratio = 3, ...){
while(TRUE){
# return_analysis argument used to return model object
out <- p_t.test(n=n, d=mean, mu=mu, return_analysis=TRUE, ...)
if(out$p.value < alpha) break
}
diff <- unname(out$estimate[1] - out$estimate[2])
M <- abs(diff)/mean
# return data.frame, where "retain" indicates the (logical) power information
data.frame(retain=M < M.ratio, M=M)
}With \(N=50\) per group and a “small” standardized effect size of .2 gives the following.
# only use the "retain" information to compute power, though store the rest
l_two.t_typeM(n=50, mean=.2) |> Spower(select='retain') -> typeM
typeM##
## Execution time (H:M:S): 00:00:13
## Design conditions:
##
## # A tibble: 1 × 4
## n mean sig.level power
## <dbl> <dbl> <dbl> <lgl>
## 1 50 0.2 0.05 NA
##
## Estimate of power: 0.861
## 95% Confidence Interval: [0.854, 0.868]In this case, power represents the probability that, given a
significant result was observed, the resulting \(M\) ratio was less than the
cutoff of 3 (hence, was less than three times the true effect size). The
compliment, which reflects the Type M error, is then 0.139,
which is indeed quite high as approximately
14% of the samples flagged as
significant would have needed rather large observed effects to indicate that the
sample was “unusual” in the statistical significance sense.
With respect to the distribution of the observed \(M\) values themselves, these can
be further extracted using SimResults(). Notice that when plotted, the rejection magnitudes are not normally distributed,
which is to be expected given the nature of the conditional simulation experiment.
## # A tibble: 10,000 × 5
## n mean sig.level retain M
## <dbl> <dbl> <dbl> <lgl> <dbl>
## 1 50 0.2 0.05 TRUE 2.47
## 2 50 0.2 0.05 TRUE 2.44
## 3 50 0.2 0.05 TRUE 2.57
## 4 50 0.2 0.05 TRUE 2.52
## 5 50 0.2 0.05 TRUE 2.16
## 6 50 0.2 0.05 FALSE 3.59
## 7 50 0.2 0.05 TRUE 1.78
## 8 50 0.2 0.05 TRUE 2.21
## 9 50 0.2 0.05 TRUE 2.79
## 10 50 0.2 0.05 FALSE 3.38
## # ℹ 9,990 more rows## mean SD min max
## 2.4863313 0.4696919 1.6198006 5.3105530
Finally, increasing the sample size greatly helps with the Type M issues, as seen below by doubling the sample size below.
# double the total sample size
l_two.t_typeM(n=100, mean=.2) |> Spower(select='retain') -> typeM2
typeM2##
## Execution time (H:M:S): 00:00:09
## Design conditions:
##
## # A tibble: 1 × 4
## n mean sig.level power
## <dbl> <dbl> <dbl> <lgl>
## 1 100 0.2 0.05 NA
##
## Estimate of power: 0.991
## 95% Confidence Interval: [0.989, 0.993]where the Type M error for the \(M.ratio = 3\) cutoff is now
## [1] 0.0088so approximately 1% of the significant results would have been due to an overly large effect size estimate. Again, the distributional properties of the observed \(M\) values can be extracted and further analyzed should the need arise.
## # A tibble: 10,000 × 5
## n mean sig.level retain M
## <dbl> <dbl> <dbl> <lgl> <dbl>
## 1 100 0.2 0.05 TRUE 1.43
## 2 100 0.2 0.05 TRUE 1.45
## 3 100 0.2 0.05 TRUE 1.83
## 4 100 0.2 0.05 TRUE 1.72
## 5 100 0.2 0.05 TRUE 1.98
## 6 100 0.2 0.05 TRUE 1.61
## 7 100 0.2 0.05 TRUE 2.01
## 8 100 0.2 0.05 TRUE 2.52
## 9 100 0.2 0.05 TRUE 1.48
## 10 100 0.2 0.05 TRUE 1.49
## # ℹ 9,990 more rows## mean SD min max
## 1.8333941 0.3667748 1.2174419 4.4615467
Type M errors are of course intimately related to precision criteria in power analyses, and in that sense are an alternative way of looking at power planning to help from relying on “extreme” observations. Personally, the notion of thinking in terms of “ratios relative to the true population effect size” feels somewhat unnatural, though the point is obviously important as seeing significance flagged only when unreasonably large effect sizes occur is indeed troubling (e.g., replication issues).
In contrast then, I would recommend using precision-based power planning over focusing explicitly on Type M errors when power planning as this seems more natural, where obtaining greater precision in the target estimates will necessarily decrease the number of Type M error in both the “extreme” and “underwhelming” sense of the effect size estimates (the ladder of which Type M errors do not address as they do not focus on non-significant results by definition). More importantly, precision criteria are framed in the metric of the parameters relevant to the data analyst rather than on a metricless effect size ratio, which in general should be more natural to specify.