Sample size for hypothesis testing on the difference between two means

The BIS.Net Team BIS.Net Team

There are many situations where the analyst needs to decide on the difference between two means. Several researchers have reported that the difference in two means can be more important than information on the means separately.

Examples of hypothesis testing for the difference in two means include

  • Differences between gender in areas such as weight, height and salary
  • Comparing the mean output of two different machines
  • Comparing mean blood pressure prior and after medication

Although confidence intervals and hypothesis test are related there are sometimes advantages in conducting a hypothesis test, instead of using confidence intervals to determine if here has been a change in standard deviation. The advantage of hypothesis testing for these applications is that sample size can be tailored to the detection of a difference considered to be important to be detected. Not every difference, even if statistically significant is of importance.

To determine the sample size the analyst will need to specify a reference value for the null hypothesis, which can be considered the status quo. Usually this is equal to zero i.e. no difference in means. An amount of concern, i.e. the minimum change from the reference value that is important to detect must be specified. Additionally, the analyst must specify two types of risk. The first is the risk or probability of falsely concluding there is a change at least equal to the specified amount of concern when there is none.. This is called the alpha risk. The second is the risk or probability of concluding there has been no change when in fact there was a change. This is called the beta risk. Both risks are due to ‘the way the numbers fall’ when sampling. By chance sampling may have selected samples with measurements larger or smaller than expected. Finally, the type of change must be specified, which can be > or < or <>

Once this information is available the following expression is used to obtain the sample size

Sample Size = Int(((Z_Value(alpha) + Z_Value(beta)) ^ 2 * (sd1 ^ 2 + sd2 ^ 2) / amount-of-concern ^ 2) + 1)

For a two tailed test (<>) alpha=alpha/2

There are some limitations for estimating the sample size however. These are:

  • The underlying distribution must be normally distributed.
  • The standard deviations of both populations must be known or at least be based on a large sample size.
  • The samples must be independent

Once the sample size is determined it is possible to calculate critical values for the differences in two means. An amount beyond the critical value implies that there is reasonable evidence that there has been a change beyond the amount of concern. For a > than hypothesis an upper critical value is calculated. For a < than hypothesis a lower critical value is calculated. For a <> equal hypothesis both a lower and upper critical value is calculated.

A preferred and recommended option to using critical values is to perform a hypothesis test after sampling with the calculated sample size and make decisions based on the p value.

The BIS.Net Sample Size app also displays an OC and Power curve such as shown in the image below.

Hypothesis testing on the difference between two means

Power refers to the power of detecting the specified change. The power curve shows the power of concluding there has been a change greater or equal to various hypothesized values of the population difference in two means. The red lines correspond to the specified alternative difference calculated from the amount of concern. In this instance the null hypothesis difference is 0, the amount of concern 1 and hence the upper alternative mean is 1 and lower -1, each of which have a power of 90% of being detected and 10% probability of not being detected (obtained from the OC Curve). Using this curve, the analyst can determine the power at different alternative differences in two means.

Download the Inferences APP, comprised of mainstream and machine-powered analytics for statistical analysis

Analytics as a Service (AaaS) for Quality

Drive quality improvement through actionable insights using analytics you can trust! Use up to 200 analytics tools downloadable through a suite of Apps!

FREE usage of the analytics Apps for quality improvement
  • Augmented with machine-powered smarts
  • Always updated with the latest tools and features
  • No licencing or fixed subscriptions - Pay ONLY for the analysis you run from 20 USD cents per analysis, billed monthly! Set a budget so you don't exceed!