Quantifying a biossay: confidence interval of the accuracy

A bioassay validation is a set of procedures that ensure the precision and accuracy of the test results. Precision refers to the proximity of results to each other, and accuracy is the proximity of measurements results to the true value. Here we want to focus on the accuracy and the building of a confidence interval around it.

 

Let's design the validation experiments with three possible sources of variability, day, analyst, and plate having nd levels, na levels, and np levels, respectively. Each analyst performs nm measurements per plate per day on each sample. The samples will be serially diluted by a factor of two, resulting in dl dilution levels from 1/1 to 1/(2^(dl-1)).

So, the total number of measurements, N, for each dilution level will N=nd*na*np*nm.

 

To calculate the accuracy, A, we can follow the FDA recommendation [1] of comparing the mean of the measured values with the "true" value, and define accuracy as that is the ratio between the observed geometric mean of the measured titers (GMTobs) and the expected geometric mean of the titers (GMTexp), that is: 

A = GMTobs/GMTexp

The GMTexp associated at the i-th dilution level equals the GMTobs calculated over the titers measured at the i-th dilution level divided by the i-th dilution level.

Of course, the accuracy of the neat (i.e., undiluted) sample is equal to 1, because at the first dilution level GMTexp = GMTobs.

To enhance the quality of a validation process, it is recommended to include not only the point estimates of the accuracy but also their confidence intervals in the reports. This will provide more information about the precision and variability of methods. In the section below, we can see the accuracy estimates and their confidence intervals calculated using the custom R function conf_int_Accuracy.

Calculating the confidence interval of the accuracy is beneficial in bioassay validation because it provides a measure of the precision of the assay results. The confidence interval is a range of values that is likely to contain the true value of the parameter being estimated, with a certain degree of confidence. In other words, it quantifies the uncertainty associated with the assay results. By comparing the confidence interval against target validation acceptance criteria, one can determine whether the assay is fit for its intended purpose. The USP <1033> recommends comparing confidence intervals against target validation acceptance criteria in a bioassay validation exercise [2]. The European Medicines Agency (EMA) guideline for bioanalytical method validation also recommends calculating the confidence interval as part of the validation process [3]. 

Example 

Let's consider a design-of-experiment consisting of eleven dilution levels in a two-fold serial dilution (from 1:1 to 1:1024) with N measurements of titers for each dilution level. 

In the vector GMTobs, we collect the results of the eleven row-wise geometric means calculated across N titers:

GMTobs<- c(11494, 6089, 2560, 1208, 640, 293, 135, 78, 39, 17, 10,5) 

Let s be the vector of the standard deviations calculated on the log2-transformed titers:

s<- c(0.114600161,0.133153354,0.125538186,0.084989754,0,0.101697554,0.133153354,

0.061447491,0.061447491,0.133153354,0,0)

After running the conf_int_Accuracy function

conf_int_Accuracy(GMTobs,s)

we get these results:

Dilution	Accuracy	CI95%.lower	CI95%.upper
1.0000	1.0000000	0.9355777	1.0688583
0.5000	1.0595093	0.9857258	1.1388157
0.2500	0.8907446	0.8306566	0.9551793
0.1250	0.8406402	0.7927762	0.8913941
0.0625	0.8913649	0.8503661	0.9343404
0.0312	0.8161560	0.7663595	0.8691881
0.0156	0.7500000	0.6977705	0.8061390
0.0078	0.8666667	0.8215771	0.9142309
0.0039	0.8666667	0.8215771	0.9142309
0.0020	0.7727273	0.7189150	0.8305675
0.0010	0.9090909	0.8672767	0.9529211
0.0005	0.8333333	0.7950037	0.8735110

The graph can be completed by entering the limits of acceptability. In doing so, we could have an immediate representation of any out-of-bounds points.  

References

[1] Accuracy and Precision in Bioanalysis: Review of Case Studies. https://www.fda.gov/media/135131/download.

[2] Statistical Assessments of Bioassay Validation Acceptance Criteria - BioProcess InternationalBioProcess International (bioprocessintl.com)

[3] Advanced Topics in Calculating and Using Confidence Intervals for Model Validation (uah.edu)