This tutorial will help you set up and interpret a Gage R&R (Reproducibility & Repeatability) analysis for attributes in Excel using the XLSTAT software.
Dataset for running a Gage R&R Attributes Analysis in Excel using XLSTAT
The Data correspond to the assessment of the skin health state of 15 patients by 5 appraisers in two repetitions. Skin health state is evaluated on a 5-point ordinal scale: Excellent, Good, Medium, Bad, Very Bad. Besides, skin conditions were also assessed biologically in order to have reference values.
Data are stored in an observations/variables table format that consists in one column with the appraisers’ names, one column with the patients’ identifications, one column with the measurements made by the appraisers and one column with the reference values. For information only, there is a fifth column with the repetitions.
An Excel sheet containing both the data and the results can be downloaded by clicking here.
Goal of this tutorialThe aim of this tutorial is to conduct a Gage R&R study for Attributes to control and judge a measurement process. In our example, we seek to assess the ability of 5 appraisers to evaluate the skin condition of 15 patients based on visual appraisal. Results are compared to reference values of skin condition obtained by biological analysis.
Setting up a Gage R&R Attribute Analysis
To conduct Gage R&R study for Attributes, select the XLSTAT / SPC / Gage R&R Attributes command.
The SPC Gage R&R for Attributes dialog box appears.
In the General tab, select the data format observations/variables table (one column for all measurements). Then, select the column Estimated State in the Measurements field and select the Ordinal data type. Select the column Appraiser in the Operators field and the column Patient as Parts. Activate the option Reference and select the column Effective State in the Reference field.
Activate the option Column labels as the first row of the selections contains column names.
In the Optionstab, activate all the statistics and enter 95 for the confidence interval.
In the Outputs tab, activate Agreement to display the assessment agreement.
In the Charts tab, activate Agreement charts and the two sub-options Within operator and Operator versus reference.
The computations begin once you have clicked on OK. You are asked to confirm the number of rows and columns (this message can be bypassed by un-selecting the Ask for selections confirmation in the XLSTAT options panel).
Interpreting the results of a Gage R&R Attribute Analysis in XLSTAT
The results are displayed in four sections: Within operator, Between operators, Operator versus reference and All versus reference.
In all sections, the first displayed table is the assessment agreement table. In the Within operators assessment agreement table, we see that 15 patients (inspected) were evaluated and Lea agreed with herself over the repetitions on the rating of the 15 patients (matched), that is 100.00% of patients were matched with a 95% confidence interval ranging from 81.90% to 100.00%; while Luis agreed with himself for only 8 patients, that is 53.33% with a 95% confidence interval ranging from 26.59% to 78.73%.
The assessment agreement table is illustrated by the assessment agreement chart showing the percent agreement for each appraiser with their confidence interval.
The second table contains the Fleiss’ kappa for each operator and each response in the Within operator and the Operator versus reference sections, and for each response in the Between operator and the All versus reference sections.
In the Within operators section, we see that Lea shows Fleiss’ kappa of 1.00. In fact, she is in total agreement with herself. Luis shows an overall Fleiss’ kappa of 0.3878, significantly different from 0 (p-value < 0.05). However such a low value demonstrate poor agreement. A general rule indicates that values of kappa greater than 0.75 indicate good to excellent agreement.
The third table contains Cohen’s kappa for each operator and each response in the Within operator and the Operator versus reference sections, and for each response in the Between operator and the All versus reference sections.
The Within Operator Cohen’s kappa can only be computed for each operator if there are exactly two repetitions on each part and the Between Operators Cohen’s kappa can only be computed for two operators with single repetition.
The Cohen’s kappa’s are in agreement with the Fleiss’ kappa’s.
The fourth table contains the Kendall’s coefficient of concordance in the Within operator and the Between operator sections, and the Kendall’s correlation coefficients in the Operator versus reference and the All versus reference sections.
Lea, Tim and Zoe have Kendall’s coefficients of concordance of 1.00, showing strong association between the two repetitions. Peter shows a Kendall’s coefficient of concordance of 0.9869, also demonstrating strong association. These 4 coefficient are associated to p-value < 0.05, the null hypothesis (coefficient is equal to 0) is rejected. Luis has a Kendall’s coefficient of concordance of 0.5865 with a p-value of 0.2882, the null hypothesis cannot be rejected.
Let’s now have a look at the Operator versus reference Kendall’s correlation coefficient. Results indicate that there is a strong association between the evaluations of Lea, Tim and Zoe and the reference (coefficient close to 1). The association between Luis’ evaluations and the reference is lower (0.6074) but still significantly different from zero. Only Peter shows a very low coefficient (0.1078) with a p-value of 0.4491. In his case, the null hypothesis cannot be rejected.
AIAG. (2010). Measurement Systems Analysis (MSA) Reference Manual. 4th Edition, Chrysler, Ford, GM.