In the first part of our series examining USP <1033> in depth, we discussed the goals of bioassay validation – specifically, to demonstrate that an assay is suitable for its intended use. Part of this demonstration is a detailed quantification of the accuracy and precision of the assay. We saw how accuracy and precision can be thought of conceptually, linked this to how such concepts appear visually in probability distributions, and, finally, described quantifications of accuracy and precision in terms of relative bias and the geometric coefficient of variation.
In this part, we’ll explore how accuracy and precision combine to affect the probability of an assay result being out-of-specification and how this can be used to inform validation acceptance limits. We will then discuss the concepts of assay linearity and range, and how these are to be included in a validation study.
Key Takeaways
-
Accuracy and precision combine to determine the probability of an out-of-specification result (Prob(OOS)). These metrics must therefore be considered jointly rather than independently.
-
Validation limits for relative bias and intermediate precision can be justified by calculating their impact on Prob(OOS), either the historical Cpm index or by directly modelling the process distribution.
-
Linearity and range ensure reliability across conditions. Demonstrating acceptable dilutional linearity and confirming the assay range covers (and, where appropriate, exceeds) specification limits ensures consistent performance across the potency levels encountered in practice.
Specification Limits and Probability of Out of Spec
We’ve focussed so far on the validation of assays which are used for batch release testing. A key part of batch release testing is the choice of specification limits. These are the bounds which determine whether a batch of product is suitable for release to patients. Specification limits are typically set for one or more Critical Quality Attributes (CQAs): key properties which determine the safety and efficacy of the final product.
Relative potency is a common CQA for which specification limits must be set. These can be informed by preclinical and clinical information about the safety and efficacy of the product – example limits might be 70% – 143%. Note that it is important that the limits are set to be symmetric on the log scale and not the raw scale. Our example limits of 70% – 143% may appear asymmetric on the raw scale, but have multiplicative symmetry on the log scale about a value of one.
For simplicity, let’s take a simple scenario where relative potency is the only CQA which we’re concerned with. That means the release decision is fully determined by whether the relative potency measured by the assay falls within the specification limits. Based on this, an assay which is fit for purpose for such testing is one which is highly likely to provide in-spec results when the product is truly in-spec, and highly likely to provide out-of-spec results when the product is truly out-of-spec.
The probability of an assay returning an out-of-spec result, or Prob(OOS), is, therefore, a key component of assessing whether an assay is fit for purpose. A good target is for when the true relative potency of a sample is 100%. That means acceptable performance for an assay is to incorrectly fail no more than 1 in 100 assays where the true relative potency is 100.
Combining Accuracy and Precision
There are several factors which affect the Prob(OOS) associated with an assay. As one might expect, the accuracy and precision of the assay are key among these, alongside the specification limits themselves. Importantly, Prob(OOS) depends on the combination of accuracy and precision. We can visualise this by examining the distribution associated with an assay, as shown in Figure 1.
Recall that the probability of an observation falling within a given range is related to the area under the distribution over that range. That means that the probability of an observation being out-of-spec is shown by the area under the distribution which is outside the specification limits – this is the shaded area in Figure 1. It is clear then why the choice of specification limits affects the Prob(OOS): if the specification limits are narrowed then, all else being equal, there will be more area outside them, indicating that Prob(OOS) has increased.
The precision of an assay is encoded in the width of its distribution – its standard deviation. A wider distribution indicates a less precise assay and a greater standard deviation. Meanwhile, the bias of an assay is represented by the horizontal distance between the peak of its distribution and the target relative potency, which is 100% in Figure 1. An assay with lower precision will have a wider distribution, meaning that, assuming constant specification limits, more of the distribution will fall outside the limits. Similarly, a more biased assay will have a greater horizontal shift, again moving more of the distribution outside the limits. In both cases, the Prob(OOS) will increase as a result.
These two factors – the width and shift of the distribution – combine to give the overall Prob(OOS). A more precise assay can afford to be less accurate to maintain a desirable Prob(OOS) than a less precise assay. That’s because the narrower distribution means that the curve can be shifted further before a problematic amount of the distribution is outside the specification limits. Similarly, if the assay is highly accurate – if its distribution peak is close to the centre of the specification limits – then it can afford to be less precise and have a wider distribution before its Prob(OOS) increases too far.
Figure 2 visualises how the accuracy and precision of an assay combine to give a Prob(OOS). In the plot, the shade of grey of the contour darkens as the Prob(OOS) decreases. The black region indicates combinations of relative bias and intermediate precision which produce a Prob(OOS) of less than the target of 1%. For values of the relative bias further from zero, the precision must be smaller to maintain a Prob(OOS) < 1%.
Note here that there are contributors to the precision of the assay in which we are interested:
- Production Variability (
). This is the variability in the relative potency inherent to the product itself. This mainly originates from the biological process used to manufacture the product. - Measurement Variability (
– where RV stands for Reportable Value). The variability in the relative potency introduced by the testing process. The reported relative potency from an assay is commonly calculated over several runs consisting of several plates. The testing variability accounts for the variability introduced by this combination. The intermediate precision of the assay is the testing variability when the reportable value is calculated from one run of one plate. - Process Variability (
). The overall variability of the full manufacturing and testing process of the product: 
.
Setting Acceptance Criteria
So, the probability of an assay providing an out-of-spec result is dependent on its accuracy and precision, and the choice of specification limits. How does this feed into the process of assay validation? Prob(OOS) usually directly informs the choice of acceptance criteria: the performance the assay must demonstrate in the validation study in order to pass. USP <1033> states that “When there is an existing product specification [i.e. pre-defined specification limits], acceptance criteria can be justified on the basis of the risk that measurements may fall outside of the product specification”.
Generally, the acceptance criteria for the accuracy of the assay will be a range in which the 90% confidence interval on the relative bias (%RB) must fall. For the precision, the 95% confidence interval on the intermediate precision (IP) need only fall below a critical value since we aren’t concerned about the precision being too low.
There are two recognised methods of assessing the Prob(OOS) based on the chosen acceptance criteria. The first, known as the Probability Capability Index (Cpm), provides an estimate of the Prob(OOS) according to the below formula:
![\[C_{pm} = \frac{ USL - LSL}{ 6 \sqrt{\sigma^2_{Product}+RB^2+\sigma^2_{RV}}}\]](https://www.quantics.co.uk/wp-content/ql-cache/quicklatex.com-53244e30ad5de57ae0ddc1d49f32bd49_l3.png "Rendered by QuickLaTeX.com")
Where:
are the upper and lower specification limits, respectively, measured on the log scale
is the production variability
is maximum allowable relative bias for a given acceptance criterion
is the measurement variability for the maximal intermediate precision for a given acceptance criterion
The greater the Cpm, the lower the Prob(OOS). The below table breaks down the relationship between Cpm and the approximate Prob(OOS):
| Cpm | Approximate Prob(OOS) when true RP = 100% | |
|---|---|---|
| 0.33 | 32% | 1 in 3 |
| 0.67 | 5% | 1 in 20 |
| 1.00 | 0.3% | 1 in 300 |
| 1.33 | 0.01% | 1 in 10,000 |
In the latest draft version USP <1033>, the Cpm method has been superseded by a method of directly calculating the Prob(OOS). This is beneficial because Cpm provides an approximation to the Prob(OOS). We are specifically interested in calculating Prob(OOS) from the process distribution. This accounts for the bias as well as the total variability of the assay. The standard deviation of the process distribution will be 
, and its mean will be 
. The difference between and the true underlying mean of the product will be the bias of the assay.
Figure 3 visualises Prob(OOS) for a process distribution. The pale grey curve represents the distribution associated with the product itself, while the black distribution shows the manufacturing distribution. Notice how the process distribution is wider than the product distribution: this reflects that 
. Similarly, the process distribution is shifted to the right relative to the product distribution. Definitionally, the mean of the product distribution is the “true” relative potency, so this shift between 
, which we assume to be 100% in this case, and reflects the bias of the assay. Also marked are the Upper and Lower Specification Limits (USL and LSL respectively). The orange shaded area represents the probability of an observation falling out-of-spec above the USL, while the green shaded area represents the probability of an observation falling out-of-spec below the LSL. Prob(OOS), therefore, is the total of the green and orange shaded areas.
To calculate the Prob(OOS), therefore, we must evaluate
![\[Prob(OOS) =Prob(x<LSL)+Prob(x>USL)\]](https://www.quantics.co.uk/wp-content/ql-cache/quicklatex.com-d3f9f9a851392b7be0dc3ce460b78d19_l3.png "Rendered by QuickLaTeX.com")
For an observation of the relative potency of a new batch of product . We use the Cumulative Density Function for the normal distribution 
to give us
![\[Prob(OOS)=\Phi\left(\frac{LSL-\mu_{Process}}{\sqrt{\sigma^2_{Process}}}\right)+\left(1-\frac{USL-\mu_{Process}}{\sqrt{\sigma^2_{Process}}}\right)\]](https://www.quantics.co.uk/wp-content/ql-cache/quicklatex.com-3aa1f887b87b5eacca88efef2033d2d7_l3.png "Rendered by QuickLaTeX.com")
To evaluate a set of chosen acceptance criteria, we set to be equal to the process distribution mean associated with the %RB acceptance criterion, then calculate combining the , and product variability, and input both into the formula for Prob(OOS). We take this approach as the criteria tell us the “worst” possible scenario where the Prob(OOS) is maximised, meaning we know that the Prob(OOS) will be lower than that calculated if the measured %RB and IP fall within their respective acceptance criteria.
Dilutional Linearity and Assay Range
Alongside a demonstration of the overall accuracy and precision of the assay, a further key component of a validation study is an assessment of the assay linearity and range. Dilutional linearity is the relationship between the expected and measured relative potency over a series of potency levels. To test the linearity of the assay, the measured relative potency is trended against the expected relative potency over the chosen levels (50% to 200% expected potency is a common choice). An example of this is shown in Figure 4.
We expect the trend to be linear, so we apply a simple linear regression model:
![\[\ln(MRP)=a+b\ln(ERP)\]](https://www.quantics.co.uk/wp-content/ql-cache/quicklatex.com-d5f6d554377d92a4a0e3baa8a86654ec_l3.png "Rendered by QuickLaTeX.com")
Where:
is the Measured Relative Potency
is the Expected Relative Potency
is the Intercept
is the slope
An assay which is perfectly unbiased across all included levels will return a trend with an intercept of zero and a slope of one: the measured relative potency will exactly match the expected relative potency at all levels. Of course, this is unrealistic, so an acceptance range is typically set on the confidence interval for the fitted slope: 0.80 to 1.20 being a standard choice.
The range of the assay is the potency levels over which the assay has been demonstrated to exhibit acceptable accuracy, precision, and linearity. The assay range should minimally cover the product specification range, and should be broader for stability testing or if the relative potency of the product is expected to be very high or very low.
A Basis for Decision Making
By framing assay performance in terms of the probability of an out-of-specification result, USP <1033> provides a clear bridge between statistical characterisation of an assay and its real-world role in batch release. Accuracy and precision are not independent boxes to be ticked: they act together to shape the process distribution. It is this combined effect that ultimately determines whether an assay is fit for purpose.
This perspective also provides justification to validation acceptance criteria. Limits on relative bias and intermediate precision can be explicitly justified by the level of risk they . Whether through the historical Cpm index or via direct calculation of Prob(OOS), acceptance criteria are chosen so that an assay meeting them delivers an acceptably low chance of incorrect release decisions. We’ve also seen how these global performance considerations sit alongside assessments of assay range and dilutional linearity. Demonstrating acceptable accuracy and precision across the relevant potency range ensures that the assay remains reliable not just at nominal conditions, but wherever it may be applied in practice.
In the final part of this series, we’ll run through the worked example of a validation study provided in USP <1033>, highlighting key decision points and pitfalls.
Comments are closed.