Introduction

In the SimTOST R package, which is specifically designed for sample size estimation for bioequivalence studies, hypothesis testing is based on the Two One-Sided Tests (TOST) procedure. (Sozu et al. 2015) In TOST, the equivalence test is framed as a comparison between the the null hypothesis of ‘new product is worse by a clinically relevant quantity’ and the alternative hypothesis of ‘difference between products is too small to be clinically relevant’.

Hypotheses

The null and alternative hypotheses for the equivalence test are presented below for two different approaches:

Difference of Means (DOM)

Individual equivalence between a test treatment ($T$) and a reference treatment ($R$) is assessed by comparing the difference between their means, $\mu_T - \mu_R$, with lower and upper equivalence limits $L$ and $U$. The TOST procedure evaluates two one-sided hypotheses:

$$H_{01}: \mu_T - \mu_R \leq L \qquad\text{versus}\qquad H_{11}: \mu_T - \mu_R > L,$$

and

$$H_{02}: \mu_T - \mu_R \geq U \qquad\text{versus}\qquad H_{12}: \mu_T - \mu_R < U.$$ The TOST formulation follows the standard equivalence-testing framework (Sozu et al. 2015; Schuirmann 1987).

Equivalence is established if and only if both $H_{01}$ and $H_{02}$ are rejected at the chosen significance level $\alpha$. Equivalently, the alternative equivalence hypothesis is

$$H_1: L < \mu_T - \mu_R < U,$$

against the composite null hypothesis

$$H_0: \mu_T - \mu_R \leq L \qquad\text{or}\qquad \mu_T - \mu_R \geq U.$$

For multiple endpoints, these hypotheses are evaluated for each endpoint, and the trial-level decision requires the specified number of endpoint hypotheses to be rejected.

The DOM test can be implemented in sampleSize() by setting ctype = "DOM" and distribution = "norm".

For pharmacokinetic (PK) outcomes, such as the area under the curve (AUC) and maximum concentration (Cmax), log-transformation is commonly applied to achieve normality. To perform this transformation, the logarithm of the geometric mean should be provided to mu_list, while the logarithmic variance can be derived from the coefficient of variation (CV) using the formula:

$$\text{Logarithmic Variance} = \log\left(1 + {\text{CV}^2}\right)$$

The use of the logarithmic scale for bioequivalence analysis is described in standard sample-size and regulatory guidance (Sozu et al. 2015; Committee for Medicinal Products for Human Use (CHMP) 2010). The equation itself follows from the definition of a log-normal distribution.

Equivalence limits must also be specified on the log scale to align with the transformed data.

Ratio of Means (ROM)

The same two one-sided structure can be expressed for the ratio of means, $\mu_T / \mu_R$, using lower and upper ratio limits $L$ and $U$:

$$H_{01}: \frac{\mu_T}{\mu_R} \leq L \qquad\text{versus}\qquad H_{11}: \frac{\mu_T}{\mu_R} > L,$$

and

$$H_{02}: \frac{\mu_T}{\mu_R} \geq U \qquad\text{versus}\qquad H_{12}: \frac{\mu_T}{\mu_R} < U.$$ The ratio formulation is the corresponding TOST on the ratio-of-means scale (Sozu et al. 2015; Schuirmann 1987).

Thus, equivalence requires rejection of both $H_{01}$ and $H_{02}$, or equivalently:

$$H_1: L < \frac{\mu_T}{\mu_R} < U,$$

against

$$H_0: \frac{\mu_T}{\mu_R} \leq L \qquad\text{or}\qquad \frac{\mu_T}{\mu_R} \geq U.$$

Here, $\mu_T$ and $\mu_R$ represent the arithmetic mean endpoints for the test and reference products, respectively.

The ROM test can be implemented in sampleSize() by setting ctype = "ROM" and distribution = "lnorm". Note that the mu_list argument should contain the arithmetic means of the endpoints, while sigma_list should contain their corresponding variances.

The ROM test is converted to a Difference of Means (DOM) tests by log-transforming the data and equivalence limits. The variance on the log scale is calculated using the normalized variance formula:

$$\text{Logarithmic Variance} = \log\left(1 + \frac{\text{Arithmetic Variance}}{\text{Arithmetic Mean}^2}\right)$$

The logarithmic mean is then calculated as:

$$\text{Logarithmic Mean} = \log(\text{Arithmetic Mean}) - \frac{1}{2}(\text{Logarithmic Variance})$$ These transformations use the standard log-normal parameterization.

Count outcomes: event-rate ratio (RR)

For count outcomes, such as the number of events observed during a specified exposure period, SimTOST uses the event-rate ratio rather than the continuous outcome ratio of means. The event-rate ratio is

$$RR = \frac{\lambda_T}{\lambda_R},$$ The event-rate ratio is the estimand used for count-outcome equivalence comparisons (Chang et al. 2017; Zhu 2017), where $\lambda_T$ and $\lambda_R$ are the event rates in the test and reference groups. For lower and upper rate-ratio limits $L$ and $U$, the hypotheses are:

$$H_{01}: \frac{\lambda_T}{\lambda_R} \leq L \qquad\text{versus}\qquad H_{11}: \frac{\lambda_T}{\lambda_R} > L,$$

and

$$H_{02}: \frac{\lambda_T}{\lambda_R} \geq U \qquad\text{versus}\qquad H_{12}: \frac{\lambda_T}{\lambda_R} < U.$$

Equivalence is concluded only when both one-sided null hypotheses are rejected. Equivalently:

$$H_1: L < \frac{\lambda_T}{\lambda_R} < U,$$

against the composite null

$$H_0: \frac{\lambda_T}{\lambda_R} \leq L \qquad\text{or}\qquad \frac{\lambda_T}{\lambda_R} \geq U.$$

The count test is implemented on the log-rate-ratio scale, using $\log(L)$ and $\log(U)$ as the boundaries. It is distinct from the continuous ROM estimand. In sampleSize() and simPower(), use distribution = "pois" or distribution = "nbinom" and provide a named rate_list. Count outcomes also require exposure; negative-binomial outcomes additionally require a positive dispersion value. The ctype argument is not used for count outcomes because the estimand is automatically the event-rate ratio.

For multiple endpoints, list_comparator identifies the treatment pairs and list_lequi.tol and list_uequi.tol provide the rate-ratio limits for each selected endpoint. A simulated trial is successful when the required number of endpoints passes for every required comparator. Endpoint dependence can be specified with cor_mat. For count outcomes, cor_mat specifies the dependence between latent normal variables used to generate correlated counts. The resulting Pearson correlation between the observed discrete counts is generally different and depends on the marginal count distributions. For normally distributed continuous outcomes, the latent correlation and Pearson correlation coincide because the transformation is linear. This follows directly from the definition of Pearson correlation: a common linear transformation preserves the correlation, apart from a possible change in sign if the transformation reverses orientation.

Regulatory requirements

When evaluating bioequivalence, certain statistical and methodological requirements must be adhered to, as outlined in the European Medicines Agency’s bioequivalence guidelines (Committee for Medicinal Products for Human Use (CHMP) 2010). These requirements ensure that the test and reference products meet predefined criteria for equivalence in terms of PK parameters. The key considerations are summarized below:

• Hypothesis testing should be based on the ratio of the population geometric means
• The 90% confidence interval for the ratio of the test and reference products should be contained within the acceptance interval of 80 to 125%.
• A margin of clinical equivalence ($\Delta$) is chosen by defining the largest difference that is clinically acceptable, so that a difference larger than this would matter in practice.
• The data should be transformed prior to analysis using a logarithmic transformation and subsequently be analyzed using ANOVA

When conducting a DOM test, the FDA recommends that the equivalence acceptance criterion (EAC) be defined as $\delta = EAC = 1.5 \sigma_R$, where $\sigma_R$ represents the variability of the log-transformed endpoint for the reference product.

Testing of multiple endpoints

Assessment of equivalence is often required for more than one primary variable. (Sozu et al. 2015) For example, the EMA recommends showing equivalence both for AUC and Cmax

A decision must be made as to whether it is desirable to

• Demonstrate equivalence for all primary endpoints
  • This is the most common setting and is often referred to as having multiple co-primary endpoints.
  • Equivalence must be demonstrated for all endpoints to conclude overall equivalence.
• Demonstrate equivalence for at least one of the primary endpoints
  • Known as having multiple primary endpoints.
  • Equivalence is required for at least one endpoint to meet the study’s objectives.

Testing multiple co-primary endpoints

When a trial defines multiple co-primary endpoints, equivalence must be demonstrated for all of them to claim overall treatment equivalence. In this setting, each endpoint is tested separately at the usual significance level ($\alpha$), and equivalence is established only if all individual tests are statistically significant. Because conclusions require rejecting all null hypotheses, the decision is an intersection–union test and a formal endpoint-wise multiplicity adjustment is not needed to control the Type I error rate (Berger and Hsu 1996; Hamasaki et al. 2018). However, as the number of co-primary endpoints ($K$) increases, the likelihood of failing to meet equivalence on at least one endpoint also rises, resulting in a higher Type II error rate (i.e., a greater risk of incorrectly concluding non-equivalence) (Mielke et al. 2018)

This has several important implications:

• Reduced Power in Rare Diseases. Previous studies have shown that the sample size required to maintain a given power level increases as the number of endpoints increases. (Mielke et al. 2018) This effect is particularly pronounced when the test statistics are uncorrelated or when a large number of tests are performed. In common conditions, this loss of power can often be compensated by increasing the sample size. In rare diseases, however, patient recruitment is often limited, making it more challenging to achieve equivalence across all endpoints and increasing the risk of an inconclusive result.
• Alternative Statistical Approaches. To address power loss from requiring equivalence across multiple endpoints, alternative methods have been proposed. For example, one option is to power the study so that equivalence can be demonstrated for at least $k$ tests, rather than requiring all endpoints to meet the equivalence criterion. Another approach is hierarchical testing, where endpoints are tested sequentially based on predefined rules, allowing for partial conclusions when equivalence is demonstrated in a subset of endpoints. See Testing multiple primary endpoints for more details.
• Regulatory Considerations. Regulatory agencies often require a pre-specified statistical strategy to handle multiple endpoints in equivalence trials. Without proper planning, the risk of failing to establish equivalence in all endpoints may lead to inconclusive results, even if the treatments are meaningfully similar.

Testing multiple primary endpoints

When a trial aims to establish equivalence for at least $k$ of $m$ primary endpoints, the decision is a partial-conjunction test. When $k < m$, an endpoint-wise adjustment is generally needed to control the Type I error across the possible successful endpoint subsets (Benjamini and Heller 2008; Mielke et al. 2018). The appropriate adjustment depends on the prespecified $k$-out-of-$m$ rule.

For example, if a study includes $m = 3$ independent primary endpoints and uses a significance level of $\alpha = 5$ for each test, the overall probability of falsely concluding equivalence for at least one endpoint is:

$$1 – (1-\alpha)^m = 1 - (1-0.05)^3 = 0.1426.$$ This is the probability of at least one false rejection under independent tests (Dunn 1961; Sidak 1967). This means that the overall probability of making any false positive error, also known as the family-wise error rate (FWER), increases to approximately 14%.

To address this issue, adjustments to the significance level are necessary for multiple endpoint comparisons for which various methods have been proposed. In SimTOST, the following approaches are included:

Bonferroni correction

The most common and easiest procedure for multiplicity adjustment to control the FWER is the Bonferroni method (Bonferroni 1936). Each hypothesis is tested at level

$$\alpha_{bon}= \alpha/m$$

where $m$ is the total number of tests. Although simple, this method is highly conservative, particularly when tests are correlated, as it assumes all tests are independent. This conservativeness remains pronounced even for $k=1$, where only one of the $m$ hypotheses needs to be rejected. (Mielke et al. 2018)

In the sampleSize() function, the Bonferroni correction can be applied by setting adjust = "bon".

Sidak correction

The Sidak correction (Sidak 1967) is an alternative method for controlling the FWER. Like the Bonferroni correction, it assumes that tests are independent. However, the Sidak correction accounts for the joint probability of all tests being non-significant, making it mathematically less conservative than the Bonferroni method. The adjusted significance level is calculated as:

$$\alpha_{sid}= 1-(1-\alpha)^ {1/m}$$

The Sidak correction can be implemented by specifying adjust = "sid" in the sampleSize() function.

K adjustment: Mielke’s weak k-out-of-m correction

The K adjustment (Mielke et al. 2018) explicitly accounts for the scenario where equivalence is required for only $k$ out of $m$ endpoints. Unlike the Bonferroni and Sidak corrections, which assume that all $m$ tests contribute equally to the overall Type I error rate, the k-adjustment directly incorporates the number of endpoints ($k$) required for equivalence into the adjustment. The adjusted significance level is calculated as:

$$\alpha_k= \frac{k*\alpha}{m}$$

where $k$ is the number of endpoints required for equivalence, and $m$ is the total number of endpoints evaluated.

T adjustment: Mielke’s strong k-out-of-m correction

The t adjustment (Mielke et al. 2018) described below is the strong k-out-of-m procedure for partial-conjunction decisions and is distinct from the weak k adjustment.

When the decision rule requires equivalence for at least k of m endpoints, the null hypothesis is a partial-conjunction hypothesis. In this setting, adjust = "t" applies Mielke et al.’s strong k-out-of-m calibration. The significance level for each relevant boundary endpoint is:

$$\alpha_{t}= \frac{\alpha}{m-k+1}$$

where m is the number of selected endpoints and k is the minimum number that must demonstrate equivalence. This adjustment controls the Type I error rate for the partial-conjunction decision, including cases in which only some endpoints satisfy the null hypothesis (Mielke et al. 2018).

Hierarchical testing of multiple endpoints

Hierarchical testing is an approach to multiple endpoint testing where endpoints are tested in a predefined order, typically based on their clinical or regulatory importance. A fallback testing strategy is applied, allowing sequential hypothesis testing. If a hypothesis earlier in the sequence fails to be rejected, testing stops, and subsequent hypotheses are not evaluated. (Chowdhry et al. 2024)

To implement hierarchical testing in simTOST, the user specifies adjust = "seq" in the sampleSize() function and defines primary and secondary endpoints using the type_y vector argument. The significance level ($\alpha$) is adjusted separately for each group of endpoints, ensuring strong control of the Family-Wise Error Rate (FWER) while maintaining interpretability.

1. Evaluate (co-)primary endpoints
   • Testing begins with the pre-specified (co-)primary endpoints at the nominal significance level.
   • Equivalence must be demonstrated for all (co-)primary endpoints.
   • If any primary endpoint fails to meet the equivalence criteria, testing stops, and secondary endpoints are not evaluated.
2. Proceed to secondary endpoints (if applicable)
   • If all primary endpoints meet the equivalence criteria, testing proceeds to the secondary endpoints.
   • Equivalence must be demonstrated for at least k secondary endpoints.
   • The significance level for secondary endpoints is adjusted using k-adjustment.
3. Final Decision
   • The test is considered successful if all (co-)primary endpoints meet the equivalence criteria and at least k secondary endpoints demonstrate equivalence.

An example of hierarchical testing can be found in this vignette.

Testing of multiple treatments

In certain cases, it may be necessary to compare multiple treatments simultaneously. This can be achieved by specifying multiple comparators in the mu_list and sigma_list parameters. The sampleSize() function can accommodate multiple treatments, allowing for the evaluation of equivalence across different products or formulations.

Although trials with multiple arms are common, there is no clear consensus in the literature as to whether statistical corrections should be applied for testing multiple primary hypotheses in such analyses. In SimTOST, no adjustments are made for trials involving more than two treatment arms.

References

Benjamini, Yoav, and Ruth Heller. 2008. “Screening for Partial Conjunction Hypotheses.” Biometrics 64 (4): 1215–22. https://doi.org/10.1111/j.1541-0420.2007.00984.x.

Berger, Roger L., and Jason C. Hsu. 1996. “Bioequivalence Trials, Intersection-Union Tests and Equivalence Confidence Sets.” Statistical Science 11 (4): 283–319. https://doi.org/10.1214/ss/1032280304.

Bonferroni, Carlo Emilio. 1936. Teoria Statistica Delle Classi e Calcolo Delle Probabilità. Seeber.

Chang, Yu-Wei, Yi Tsong, and Zhigen Zhao. 2017. “Sample Size Determination for a Three-Arm Equivalence Trial of Poisson and Negative Binomial Responses.” Journal of Biopharmaceutical Statistics 27 (2): 239–56. https://doi.org/10.1080/10543406.2016.1269787.

Chowdhry, Amit K., John Park, John Kang, Gukan Sakthivel, and Stephanie Pugh. 2024. “Finding Multiple Signals in the Noise: Handling Multiplicity in Clinical Trials.” International Journal of Radiation Oncology*Biology*Physics 119 (3): 750–55. https://doi.org/10.1016/j.ijrobp.2023.12.007.

Committee for Medicinal Products for Human Use (CHMP). 2010. Guideline on the Investigation of Bioequivalence. CPMP/EWP/QWP/1401/98. European Medicines Agency. https://www.ema.europa.eu/en/documents/scientific-guideline/guideline-investigation-bioequivalence-rev1_en.pdf.

Dunn, Olive Jean. 1961. “Multiple Comparisons Among Means.” Journal of the American Statistical Association 56 (293): 52–64.

Hamasaki, Toshimitsu, Scott R. Evans, and Koko Asakura. 2018. “Design, Data Monitoring, and Analysis of Clinical Trials with Co-Primary Endpoints: A Review.” Journal of Biopharmaceutical Statistics 28 (1): 28–51. https://doi.org/10.1080/10543406.2017.1378668.

Mielke, Johanna, Byron Jones, Bernd Jilma, and Franz König. 2018. “Sample Size for Multiple Hypothesis Testing in Biosimilar Development.” Statistics in Biopharmaceutical Research 10 (1): 39–49. https://doi.org/10.1080/19466315.2017.1371071.

Schuirmann, Donald J. 1987. “A Comparison of the Two One-Sided Tests Procedure and the Power Approach for Assessing the Equivalence of Average Bioavailability.” Journal of Pharmacokinetics and Biopharmaceutics 15 (6): 657–80. https://doi.org/10.1007/BF01068419.

Sidak, Zbyněk. 1967. “Rectangular Confidence Regions for the Means of Multivariate Normal Distributions.” Journal of the American Statistical Association 62 (318): 626–33.

Sozu, Takashi, Tomoyuki Sugimoto, Toshimitsu Hamasaki, and Scott R. Evans. 2015. Sample Size Determination in Clinical Trials with Multiple Endpoints. SpringerBriefs in Statistics. Springer International Publishing. https://doi.org/10.1007/978-3-319-22005-5.

Zhu, Haiyuan. 2017. “Sample Size Calculation for Comparing Two Poisson or Negative Binomial Rates in Non-Inferiority or Equivalence Trials.” Statistics in Biopharmaceutical Research 9 (1): 107–15. https://doi.org/10.1080/19466315.2016.1225594.