Thyroid nodules are ubiquitous in modern medical practice, either discovered through palpation or as an incidental finding on radiological examinations. While thyroid nodules themselves are very common,¹ the rate of thyroid cancer in New Zealand is relatively low (356 registered cases in 2016, rate 6.4/100,000 people)² but has steadily increased in recent decades. For unknown reasons, rates are three times higher in Pacific women and two times higher in Maori women compared to European women.³ Guidelines for investigation of thyroid nodules recommend thyroid ultrasound as the first radiological investigation in the presence of normal or low thyroid function tests.⁴ While characteristics of a nodule on ultrasound can be used to predict the risk of malignancy, studies, often from tertiary centres, suggest variable sensitivity and specificity for individual ultrasound characteristics (ranging from <0.5–0.93).^5–7

At Counties Manukau District Health Board (CMDHB) in Auckland, New Zealand, over half of thyroid ultrasounds are performed in private community practices with the remainder performed in hospital radiology departments. We sought to assess the quality of the thyroid reporting from this diverse range of practices and radiologists, and to compare the estimates of malignancy risk from ultrasound reports with the results of cytological and histological assessments of the nodules. We also compared the results of malignancy risk assessment by sub-specialist radiologists with a radiology registrar and how those assessments compared to the original radiology reports.

Methods

We collated all patients who had a thyroid fine needle aspirate result at CMDHB over an 18-month period spanning 2012–2014, and had a diagnostic ultrasound available. There were no other specific inclusion or exclusion criteria. A total of 91 nodules from 84 patients were included.

Basic demographic details were obtained from the electronic medical record, and all original sonographic imaging was retrieved and viewed on the CMDHB Radiology Department PACS system. A senior radiology registrar and one of two radiology consultants with a Head and Neck sub-specialty interest viewed and interpreted the images blinded to patient details, except for the imprinted age and gender on the images. The sub-specialty radiologists used the current CMDHB thyroid nodule template and their own experience for assessment. The template assessed the maximum size of the nodule and the presence of extra-thyroid extension, micro-calcifications, solid composition, echogenicity, taller than wide dimensions, central vascularity and suspicious lymph nodes. The radiology registrar used the American Thyroid Association (ATA) 2015 guidelines,⁴ which were the most up-to-date guidelines at the time. Based on the imaging features, each nodule was categorised as being either low, intermediate or high risk of malignancy. The original radiology reports were also reviewed by the radiology registrar and categorisation of malignancy risk was attempted using only the information and conclusions in the report.

We considered this an audit of current practices regarding thyroid nodule reporting as defined by the New Zealand National Ethics Advisory Committee guidelines,8 and therefore that it was a low-risk project that did not require ethical approval.

We assessed the inter-rater agreement using Kappa values. Kappa >0.8 was considered almost perfect agreement, 0.61–0.8 to be substantial, 0.41–0.6 moderate, 0.21–0.4 fair and 0–0.2 slight agreement.⁹ All analyses were performed using Stata IC 13 or Microsoft Excel 2013. For the comparison of nodule risk assessment and histology, we considered that all patients who did not have a histological sample did not have a thyroid cancer, since there was at least two years of follow-up data for each such patient with no clinical evidence of thyroid cancer.

Results

Demographics and laboratory results

Of the 84 patients with 91 nodules, 76 (90%) were female; the average age was 51 years (range 20–86 years); 40% were New Zealand European, 22% Maori and 16% Polynesian. 55% of the ultrasounds were done in private providers and 45% at CMDHB.

Table 1 shows the Bethesda cytology classification for the 91 nodules. Forty-eight of the 84 patients (54 nodules) had thyroid surgery for which histology was available. Nineteen nodules from 17 patients had thyroid cancers, of which six cancers in five patients were incidentally found at surgery (ie, the thyroid cancer was not in the pre-operative nodule(s) of interest). One patient with two benign nodules also had parathyroid cancer which was not considered in the analyses. The remaining 33 nodules had benign histological findings.

Table 1: Bethesda categorisation of the cytology of the 91 nodules.

Assessment of original ultrasound reports

Table 2 shows that the majority of reports did not report on nodule echogenicity (69%), margins (78%), taller-than-wide shape (98%) or extra-thyroid extension (75%), but generally mentioned nodule consistency (77%), vascularity (70%), the presence or absence of micro-calcifications (63%) or cervical lymphadenopathy (87%). Based on the information provided in the ultrasound report, a detailed independent assessment of malignancy risk using all the information recommended in the ATA guidelines was only possible for 1/91 nodules.

Table 2: Thyroid nodule characteristics as described in original ultrasound report.

Sixty-six of the 91 reports had no definitive indication of the malignancy risk of the relevant nodule. Table 3 shows that in the 25 reports with a definitive risk assessment, 17 (68%) categorised the nodule as intermediate or high risk. Of these 17, 10 (59%) had abnormal cytology (Bethesda 3–6) on FNA. Of the eight nodules categorised as low risk, only one had benign cytology (Bethesda 2) but five had a non-diagnostic FNA (Bethesda 1), and two (25%) had abnormal cytology.

Table 3: Malignancy nodule risk assessment in original report versus cytology results.

Assessment of nodules by sub-specialty radiologist and radiology registrar

Table 4 shows the characteristics of the 91 nodules assessed. The majority of characteristics could be assessed for each nodule, although irregular margins were only a component of the ATA guidelines not the CMDHB template. There was insufficient information in the original radiology reports to enable a meaningful inter-rater analysis between those reports and the assessment by the sub-specialty radiologist or registrar. Inter-rater agreement between the sub-specialty radiologist and the registrar was fair to moderate.

Table 4: Assessment of thyroid nodule characteristics by sub-specialty radiologist and radiology registrar.

Of the 91 nodules, 13 were categorised as high risk of malignancy, 25 as intermediate risk, and 53 low risk by the sub-specialty radiologist. The comparable numbers for the radiology registrar were 10 high, 33 intermediate and 48 low risk. The inter-rater Kappa value for malignancy risk assessment between the radiology registrar and sub-specialty radiologist was 0.22, indicating only fair agreement.

Table 5 shows the cytology and histology results compared with the malignancy risk assessments by the sub-specialty radiologist and registrar. Of the 33 nodules in 30 patients with benign (Bethesda 2) cytology, 12 and 15 respectively were categorised by the sub-speciality radiologist and registrar as high/intermediate malignancy risk. At surgery, one patient had two Bethesda 2 nodules and a Bethesda 5 nodule, and a thyroid cancer was diagnosed in the Bethesda 5 nodule. None of the other 17 patients (19 nodules) undergoing surgery had a cancer identified. Of the 36 nodules in 35 patients with abnormal cytology (Bethesda 3–6), 18 and 17 respectively were categorised as intermediate/high risk by the subspecialist and registrar. 26/35 patients (27 nodules) had surgery, and 11 nodules in 10 patients had a cancer identified. Of these 18 and 17 nodules respectively categorised as intermediate/high risk by the sub-specialists or the registrar, seven nodules in 11 patients (39%) and seven nodules in 11 patients (41%), respectively, undergoing surgery had a cancer identified. Of the 18 nodules with abnormal cytology but a low risk ultrasound assessment by the sub-speciality radiologist, 15 patients with 16 nodules had surgery and four had a thyroid cancer identified, and one patient had a parathyroid cancer. Of the 19 nodules with abnormal cytology but a low-risk ultrasound assessment by the registrar, 16 patients with 16 nodules had surgery and four had a thyroid cancer identified, one had parathyroid cancer and one patient had an incidental thyroid cancer.

Table 5: Cytology and histology results versus sub-specialist radiologist and registrar nodule malignancy risk assessment.

Table 5 also shows the comparison of ultrasound-based risk assessment with final histological diagnosis in the 84 patients with 91 nodules, assuming that patients who did not undergo surgery did not have a thyroid cancer. The sensitivity and specificity for the sub-speciality radiologist intermediate/high risk assessment and presence of thyroid cancer in the nodule of interest (ie, not an incidental cancer) was 0.62 (0.32–0.86, 95% CI) and 0.62 (0.50–0.72, 95% CI) respectively, and for the radiology registrar was 0.69 (0.39–0.91, 95% CI) and 0.56 (0.45–0.68, 95% CI) respectively.

Discussion

In this mixture of community-based, private-practice and DHB thyroid ultrasound reports, the majority did not explicitly estimate the malignancy risk of an individual nodule, nor contain sufficient information to allow an independent malignancy risk assessment. However, when the original images were reviewed, the images were almost always sufficient to permit assessment of individual nodule characteristics and an overall malignancy risk. Agreement between the senior sub-speciality radiologists and the registrar for individual nodule characteristics nodules was variable, and for overall malignancy risk was fair. Of nodules with benign (Bethesda 2) cytology, 12/33 (36%) were categorised as intermediate/high risk of malignancy by the sub-speciality radiologist, but no cancer was identified in histology or during clinical follow-up. Of nodules with abnormal cytology (Bethesda 3–6), 18/36 (50%) were categorised as intermediate/high risk of malignancy by the sub-specialty radiologist, and seven of these 18 (39%) were subsequently found to have a thyroid cancer. The sensitivity and specificity of an intermediate/high risk categorisation by the sub-speciality radiologist for thyroid cancer was 62% and 62% respectively.

These findings have a number of clinical implications. From a clinical perspective, the majority of the original ultrasound reports were inadequate to allow an assessment of the malignancy risk either by reading the report and its conclusions or by an independent assessment of nodule characteristics. Thus, the reports were often inadequate to guide management of thyroid nodules. It is likely that following the thyroid ultrasound, many patients were referred for a specialist review, which may have required a review of the thyroid ultrasound images by the specialist or in a radiology conference to allow an assessment of malignancy risk to be made. In such cases, the second radiology review is a waste of a limited resource, and creates unnecessary delay. Our findings of sub-optimal thyroid ultrasound reports are not unique, with others recently reporting similar results.^10,11 For example, Karkada and colleagues reported that almost half of nodules were not classified for malignancy risk, and 32–91% of reports did not mention key ultrasound characteristics.¹⁰

A second clinical issue arises when a patient has had a thyroid FNA with abnormal (Bethesda 3–6) cytology. The case may then be discussed at a multidisciplinary meeting with review of both the cytology and radiology findings. However, for the intermediate or high-risk malignancy estimate by a head and neck radiologist, the sensitivity and specificity for malignancy was only 62%, 8/38 (21%) had a cancer diagnosed, and 7/18 (39%) who also had abnormal cytology had a cancer identified. Furthermore, only half of nodules with abnormal cytology were considered intermediate or high risk of malignancy by ultrasound characteristics, but conversely, 22% of those with abnormal cytology who were considered at low risk by ultrasound characteristics had a cancer identified. Taken together, these results suggest that reviewing the thyroid ultrasound after a cytological assessment might not be helpful in increasing the likelihood of a thyroid cancer being diagnosed. This proposition could be tested formally in a clinical study.

Disagreement among radiologists in characterising thyroid nodules is common in clinical practice. However, some previous studies have reported strong agreement between radiologists when assessing individual nodule features.^12,13 The differences between the strong agreement in those studies and the much weaker agreement in the current study might be explained by the study designs: some studies gave specific training prior to the research;¹² and some compare assessments between highly experienced sub-specialists working at single, tertiary-level institutions with high volume throughput.^12,13 In contrast, our study was an audit of current practice and no specific training was provided, with reporting being undertaken by a mixture of radiologists in private and public practice. The senior radiology registrar with specific training and learning for this study might be considered similar in level to a general radiologist working between private and public institutions who may report only a few thyroid ultrasound studies for assessment of nodules each year. The comparisons between the subspecialist and the registrar are therefore clinically relevant.

Since we initiated our study, the American College of Radiologists has released a TI-RADs system for estimating malignancy risk based on ultrasound characteristics of thyroid nodules.¹⁴ It is hoped that this system might overcome some of the problems we identified by creating standardised reports in which each relevant thyroid nodule is assigned a TI-RADS score, with explicit recommendations made based on those scores. However, a likely limiting factor in its widespread use and clinical application is the agreement between radiologists regarding the TI-RADS score. To date, we are not aware of validation studies of TI-RADs as used in a clinical practice similar to CMDHB. It is not clear whether validation studies in highly trained, highly experienced sub-specialists, who are very familiar with the use of TI-RADS and report large volumes of ultrasound reports would be replicated in, or are relevant to, clinical practices in which the majority of thyroid ultrasounds are reported by general radiologists who report only small volumes of thyroid ultrasound. At a practical level, if only a TI-RADs score is reported, without the information upon which the score is based, an independent assessment of malignancy risk will not be possible, which might require review of the ultrasound images, again creating delays and wasting resources. On the other hand, if all the information is reported for multiple nodules, a report can quickly become long and unwieldy.

Limitations

This study is a retrospective review and therefore there is potential for confounding. The assessment of the thyroid nodules was blinded, however we were unable to remove the imprinted information on the ultrasound pictures of age and gender, which may have influenced decisions. Furthermore, reviewing ultrasound static images retrospectively removes the opportunity for real-time evaluation and discussion with the original sonographer. Another issue is that the sub-specialists used their local template for assessing malignancy risk whereas the registrar used the ATA guidelines. This might have accounted for some differences in results, although the underlying nodule characteristics to be assessed do not differ between the systems, except that irregular margins was not included in the local template.

Conclusion

The majority of thyroid ultrasound reports were suboptimal because they did not explicitly estimate the malignancy risk of an individual nodule, or contain sufficient information to allow an independent assessment of malignancy risk. When assessing nodule characteristics, agreement between the sub-speciality radiologists and the registrar was variable for individual characteristics and low for the overall malignancy risk. The level of agreement between cytological and histological findings and the estimated malignancy risk based on ultrasound findings was not high. This raises the question of whether there is any value in reviewing ultrasound findings once cytology is known. This approach, which is commonly followed in our and other institutions, should be assessed formally.