Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 May 20.
Published in final edited form as: Proc SPIE Int Soc Opt Eng. 2018 Mar 7;10577:105770C. doi: 10.1117/12.2293871

Evaluation of Search Strategies for Microcalcifications and Masses in 3D Images

Miguel P Eckstein 1, Miguel A Lago 1, Craig K Abbey 1
PMCID: PMC7237824  NIHMSID: NIHMS1588197  PMID: 32435079

Abstract

Medical imaging is quickly evolving towards 3D image modalities such as computed tomography (CT), magnetic resonance imaging (MRI) and digital breast tomosynthesis (DBT). These 3D image modalities add volumetric information but further increase the need for radiologists to search through the image data set. Although much is known about search strategies in 2D images less is known about the functional consequences of different 3D search strategies. We instructed readers to use two different search strategies: drillers had their eye movements restricted to a few regions while they quickly scrolled through the image stack, scanners explored through eye movements the 2D slices. We used real-time eye position monitoring to ensure observers followed the drilling or the scanning strategy while approximately preserving the percentage of the volumetric data covered by the useful field of view. We investigated search for two signals: a simulated microcalcification and a larger simulated mass. Results show an interaction between the search strategy and lesion type. In particular, scanning provided significantly better detectability for microcalcifications at the cost of 5 times more time to search while there was little change in the detectability for the larger simulated masses. Analyses of eye movements support the hypothesis that the effectiveness of a search strategy in 3D imaging arises from the interaction of the fixational sampling of visual information and the signals’ visibility in the visual periphery.

Keywords: 3D search, search strategies, driller, scanner, peripheral processing, useful field of view

INTRODUCTION

New 3D imaging modalities from CT to DBT are fundamentally different from traditional planar 2D radiological imaging. The 3D imaging techniques have increased the volume of data that needs to be scrutinized by radiologists by at least 50 times. Although there is a long history of studies investigating radiological errors during interpretation of 2D mammograms and chest x-rays (Krupinski, 2010, 2011; Kundel et al., 1978), there is little understanding of the errors and search strategies with the newer 3D imaging technologies. Clinical reading times with 3D images (e.g. DBT) do not allow radiologists to exhaustively move their eyes and fixate all regions of each slice in a volume. For example, a study investigated lung CTs showing that radiologists scan about 26 % of the lung parenchyma volume (with a definition of 5 degrees as a useful field of view around the fovea; Rubin et al., 2015). Thus, radiologists must rely on their peripheral vision (retinal processing away from the high resolution fovea) and guided eye movements to sample the visual information in the 3D data. The increased role of peripheral processing in 3D search introduces potential increases in search errors (Krupinski, 1996; Kundel, 1975a, 2015; Kundel et al., 1989) and dissociations of performance between signals that are difficult to see in the visual periphery and those that are more visible away from the fovea (Diaz et al., 2012; Eckstein et al., 2017; Lago et al., 2017).

The eye movement strategies a radiologist might adopt to maximize the probability of detecting disease in 3D images are not known. A study has examined the eye movement strategies of radiologists looking for nodules in lung CTs. Drew et al., (2013) categorized eye movement plans into scanner or driller strategies. Drillers tend to restrict eye movements to a small region of the image while quickly scrolling through depth. In contrast scanners move more slowly through depth and search an entire level of the lung before moving on to the next slice. They found that drillers’ performance was superior to the scanners in terms of lung nodule detection rate, percentage of the lung covered, and the percentage of search errors (nodules that were never fixated). However, such research is correlational leaving unanswered the question of whether the higher performance of radiologists engaging in drill-type search (drillers) is a result of their search strategy or whether the search strategy is a byproduct of the radiologists’ experience or variation in visual abilities (e.g., higher ability to detect signals in the visual periphery). Not known is whether these eye movement strategies have a causal functional impact on perceptual decision accuracy. One might hypothesize that drilling with fast succession of slices might increase observer efficiency to temporally integrate information (Eckstein, 1996; Eckstein et al., 1996) about the signal across slices leading to better detection accuracies. In contrast, scanning strategies might be more exhaustive in terms of foveating a larger number of possible signal locations in a subset of slices and increase the probability of detection of the signal. In this paper, we evaluate the functional consequences of various search strategies by evaluating the same individuals in 3D search while instructing them to follow implementations of the driller and scanner strategy. Utilizing real time eye position data we ensured that observers did not depart from the specified strategy and that both strategies covered similar % of the volumetric data. In particular we hypothesize that the influence of search strategy might interact with the visibility of the signal in the visual periphery. Thus, we investigated how search strategies impacted detection rates for a larger mass-like signal more visible in the visual periphery and a smaller micrcocalcification-like signal which detectability steeply declines with retinal eccentricity.

METHODS

Image generation

We generated correlated 3D Gaussian noise fields with a 3D 1/f2.8 power spectrum and 1024 × 820 × 100 voxels in size. The noise fields were displayed as 2D slices of the volume with the option to scroll up and down the volume. There were two possible signals: a small and solid sphere (6 voxels diameter; 0.13 deg.) mimicking a microcalcification or a 3D Gaussian blob (standard deviation = 10 pixels; 0.22 deg.) approximating a mass. Figure 1 shows an example of both signals embedded in the background. Prior to the initiation of a trial, observers were indicated with a cue which signal they needed to searched for (small or large signal). This signal was only present in 50% of the trials and observers were informed of this probability.

Figure 1:

Figure 1:

Open in a new tab

Example of several consecutive slices of the 3D simulated microcalcification (top) and the simulated mass (bottom) embedded in filtered noise samples.

Experimentally controlled search strategies

In order to study the performance of the two specific search strategies, observers were trained to either follow a driller or scanner strategy. For the scanner strategy, observers were instructed to freely search through each slice presented. We utilized real time eye position data to determine when over 80% of the current slice were explored by the observer through eye movement search. For each fixation we defined the useful field of view as the classic 5 deg. (Kundel, 1975b). We combined the area of 5 deg. circular regions around each fixation. When the total area explored exceeded 80 % of the slice, the computer automatically replaced the image by a new slice. In order to match the % of the entire volume explored across scanner and driller strategies, every third slice in the volume was presented. If it was a signal present trial, the slice that contained the central slice of the signal was always presented during the scanner strategy condition. For drillers, four quadrants were defined (top and bottom left, top and bottom right). The observers had to fixate on one of the quadrants to scroll automatically through the volume at a speed of 13 frames/s (based on radiologists’ typical scrolling rates). After reaching the last slice, the reader had to re-fixate at the next quadrant to start the scrolling process again in the other direction. For both search strategies, after reaching the final slice (scanner) or the last slice of the final quadrant (driller), the stimuli was removed from the screen. With an assumption of a diameter of 5 degrees of visual angle as the useful field of view (UFV, Kundel, 1975b), the percentage of volume explored with the UFV was 27% for scanner and driller strategies. An infrared video based eye tracker (Eye tribe, Kobenhaven, Denmark) was utilized to ensure that the participants followed a given search strategy. The eye tracker allowed us to calculate the % of explored area for the scanner trials in real time.

Procedure

Figure 2 shows the outline of a single trial. First, a word appears indicating the type of lesion the observer should search for (microcalcification or mass). After ensuring with the tracker that observers are fixating at the initial central cross, the first slice is presented. For scanners, the slices were presented until the % of explored area reached the threshold. Then, the display presented the subsequent third slice. For drillers, a circle was shown in the corresponding quadrant and only if the observer was fixating at that quadrant, the scrolling was activated. If the participant fixated away from that quadrant, the image was removed from the screen and the fixation was re-displayed. Observers then re-fixated the cross to resume the scrolling. Five observers (UCSB undergraduate students) participated in 40 trials per condition (scanner microcalcification, scanner mass, driller microcalcification, driller micrcocalcification). We measured the hit rate, false alarm rate, and response times for each trial. For each trial, we also measured the minimum distance of the observers’ fovea from the signal.

Figure 2.

Figure 2.

Open in a new tab

Timeline of the trial of the experiment

RESULTS

Figure 3 shows the hit rate (averaged across the five subjects) for scanner and driller strategies for both signal types. Hit rate for both, microcalcifications (p = 0.0013) and masses (p = 0.035), were significantly higher when observers scanned the 3D volumes rather than drilled. The effect was larger for microcalcifications (Cohen’s d = 3.09 for microcalcification vs. 0.96 for masses). Figure 3 also shows the false positive rate for scanners vs. driller strategies. False positive rate did not change for microcalcifications across search strategies. For the masses, the scanner strategy seemed to increase the false positive rate but did not reach significance (p= 0.2).

Figure 3.

Figure 3.

Open in a new tab

Hit rate and false alarm rate for driller and scanner strategies for each type of lesion (microcalcification: MCALC; masses: MASS).

Figure 4 shows that the detectability index d’ for microcalcifications (MCALC) is significantly (p = 0.01) higher for observers when using the scanner strategy vs. driller strategy. The differences in d’ across search strategies did not reach statistical significance for masses (p=0.3).

Figure 4.

Figure 4.

Open in a new tab

Detectability index d’ for drillers and scanner strategies for each type of lesion.

In terms of time to complete the search, the scanner strategy involved a five-fold increase (Figure 5). The result is partly determined by the experimental protocol that matches the % area explored across both strategies.

Figure 5:

Figure 5:

Open in a new tab

Time per trial for each search strategy averaged across observers and signal type.

To assess the role of differences in foveal vs. peripheral processing across the two search strategies, we quantified the minimum distance within a trial between the fovea and the signal (for signal present trials). Figure 6 shows a histogram of the minimum distances (in degrees visual angle) for driller and scanner strategies for the two signal types. Results show that the scanner strategy results in a larger % of trials in which the fovea gets closer to the signal for both microcalcifications and masses. These results suggest that the higher d’ for scanner strategies relates to the larger percentage of trials in which the signals are foveated.

Figure 6.

Figure 6.

Open in a new tab

Histogram of minimum trial distances of observer’s fovea to signal for driller and scanner strategies for microcalcification signals (left) and mass signals (right)

What remains unexplained is the dissociation across signal types. Larger percentage of trials foveating the signal for the scanner strategy has a substantial effect on microcalcification d’, but less of an effect on the detectability of masses (Figure 4). Such dissociation can be understood if one considers how the detectability of these two simulated signals degrades with retinal eccentricity. Figure 7 shows d’ vs. retinal eccentricity in a yes/no signal known exactly task measured with a different group of observers (Eckstein et al., 2017; Lago et al., 2017). Detectability degrades very abruptly with retinal eccentricity for the microcalcification signal and more modestly for masses. Thus, foveating a signal in larger percentage of trials during a scanner strategy will result in greater d’ benefits for microcalcifications.

Figure 7.

Figure 7.

Open in a new tab

Index of detectability vs. retinal eccentricity (degrees visual angle; dva) for the two simulated signals collected for a yes/no signal known exactly task. Non-radiologist observers were instructed to maintain gaze and the signals were presented at different points away from the fovea

DISCUSSION AND CONCLUSIONS

The introduction of 3D imaging techniques into the clinic has motivated new research to attempt to understand the relationships between 2D and 3D search, and to evaluate the effectiveness of different types of search strategies. Two distinct strategies with 3D images previously investigated are the driller and scanner strategies (Drew et al., 2013). Such study was correlational and could not conclusively evaluate the functional consequences of the strategies.

The goal of the current paper was to experimentally manipulate search strategies with the same individuals to evaluate the causal effects of search strategies on detection performance. In this first study, we utilized very simple signals and backgrounds. We found that for the synthetic 3D noise, the scanner strategy improved detection performance at the cost of much longer reading times. Importantly, there was an interaction between the influence of the search strategy and the signal type. In particular, the benefits of the scanner strategy over the driller strategy were only observed for smaller microcalcifications but not masses. This dissociation can be explained in terms of an interaction between the search strategy and the detectability of the signals in the visual periphery. Scanner strategies allowed observers to sample the signals closer to the fovea which has a larger performance effect on signals that are not visible in the visual periphery (e.g., microcalcifications).

We evaluated two extreme search strategies that are not a realistic representation of more natural radiologist strategies that show a more subtle mixtures of scanner and driller strategies. The current results should not be generalized to different image types nor considered to be contradictory with the results in Drew et al., 2013. There are some important differences. One difference is that Drew et al. equated time for all readings. If we had equated time across the two strategies it is likely that we would have a very different result with an additional benefit for drilling. Performance with the scanner strategy would likely degrade by shortening the time to a 1/5 of the current reading time. In addition, clinical CT images contain 3D structures that can be confused with the signal in a single slice. Drilling across depths in CT images can serve to untangle normal anatomy from the searched signals.

To summarize, the current experiment offers an experimental protocol to evaluate search strategies with more complex images. The current findings highlight the importance of the interactions between the properties of the human visual periphery and searched signals in determining how search strategies influence detection rates. In addition, future work should evaluate and develop model observers (Abbey and Barrett, 2001; Barrett et al., 1993; Eckstein, 2001; Gifford, 2013; Gifford et al., 2016; Zhang et al., 2004a, 2004b) to accurately predict human performance for various search strategies in 3D search. Some of the search models that take into account foveal vs. peripheral processing (Ackermann and Landy, 2010; Eckstein et al., 2015; Najemnik and Geisler, 2005, 2009; Renninger et al., 2007; Zhang and Eckstein, 2010) from the field of visual science might be good starting points for such endeavor.

ACKNOWLEDGEMENTS

The research was funded by the National Institute of Health grant R01 EB018958.

REFERENCES

  1. Abbey CK, and Barrett HH (2001). Human- and model-observer performance in ramp-spectrum noise: effects of regularization and object variability. J. Opt. Soc. Am. A Opt. Image Sci. Vis 18, 473–488. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Ackermann JF, and Landy MS (2010). Suboptimal Choice of Saccade Endpoint in Search with Unequal Payoffs. J. Vis 10, 530–530. [Google Scholar]
  3. Barrett HH, Yao J, Rolland JP, and Myers KJ (1993). Model observers for assessment of image quality. Proc. Natl. Acad. Sci. U. S. A 90, 9758–9765. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Diaz I, Eckstein MP, Luyet A, Bize P, and Bochud FO (2012). Measurements of the detectability of hepatic hypovascular metastases as a function of retinal eccentricity in CT images. In SPIE Medical Imaging, (International Society for Optics and Photonics), p. 83180J–83180J. [Google Scholar]
  5. Drew T, Vo ML-H, Olwal A, Jacobson F, Seltzer SE, and Wolfe JM (2013). Scanners and drillers: characterizing expert visual search through volumetric images. J. Vis 13. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Eckstein MP (1996). Detection and contrast discrimination of moving signals in uncorrelated Gaussian noise. In Proceedings of SPIE, (Newport Beach, CA, USA), pp. 9–25. [Google Scholar]
  7. Eckstein MP (2001). Model observers for signal-known-statistically tasks (SKS). In Proceedings of SPIE, (San Diego, CA, USA), pp. 91–102. [Google Scholar]
  8. Eckstein MP, Whiting JS, and Thomas JP (1996). Role of knowledge in human visual temporal integration in spatiotemporal noise. J. Opt. Soc. Am. A Opt. Image Sci. Vis 13, 1960–1968. [DOI] [PubMed] [Google Scholar]
  9. Eckstein MP, Schoonveld W, Zhang S, Mack SC, and Akbas E (2015). Optimal and human eye movements to clustered low value cues to increase decision rewards during search. Vision Res. 113, 137–154. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Eckstein MP, Lago MA, and Abbey CK (2017). The role of extra-foveal processing in 3D imaging. In SPIE Medical Imaging, (International Society for Optics and Photonics), p. 101360E–101360E. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Gifford HC (2013). A visual-search model observer for multislice-multiview SPECT images. Med. Phys 40, 092505. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Gifford HC, Liang Z, and Das M (2016). Visual-search observers for assessing tomographic x-ray image quality. Med. Phys 43, 1563–1575. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Krupinski EA (1996). Visual scanning patterns of radiologists searching mammograms. Acad. Radiol 3, 137–144. [DOI] [PubMed] [Google Scholar]
  14. Krupinski EA (2010). Current perspectives in medical image perception. Atten. Percept. Psychophys 72, 1205–1217. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Krupinski EA (2011). The role of perception in imaging: past and future. Semin. Nucl. Med 41, 392–400. [DOI] [PubMed] [Google Scholar]
  16. Kundel HL (1975a). Peripheral vision, structured noise and film reader error. Radiology 114, 269–273. [DOI] [PubMed] [Google Scholar]
  17. Kundel HL (1975b). Peripheral vision, structured noise and film reader error. Radiology 114, 269–273. [DOI] [PubMed] [Google Scholar]
  18. Kundel HL (2015). Visual search and lung nodule detection on CT scans. Radiology 274, 14–16. [DOI] [PubMed] [Google Scholar]
  19. Kundel HL, Nodine CF, and Carmody D (1978). Visual scanning, pattern recognition and decision-making in pulmonary nodule detection. Invest. Radiol 13, 175–181. [DOI] [PubMed] [Google Scholar]
  20. Kundel HL, Nodine CF, and Krupinski EA (1989). Searching for lung nodules. Visual dwell indicates locations of false-positive and false-negative decisions. Invest. Radiol 24, 472–478. [PubMed] [Google Scholar]
  21. Lago MA, Abbey CK, and Eckstein MP (2017). Foveated Model Observers to predict human performance in 3D images. Proc. SPIE-- Int. Soc. Opt. Eng 10136. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Najemnik J, and Geisler WS (2005). Optimal eye movement strategies in visual search. Nature 434, 387–391. [DOI] [PubMed] [Google Scholar]
  23. Najemnik J, and Geisler WS (2009). Simple summation rule for optimal fixation selection in visual search. Vision Res. 49, 1286–1294. [DOI] [PubMed] [Google Scholar]
  24. Renninger LW, Verghese P, and Coughlan J (2007). Where to look next? Eye movements reduce local uncertainty. J. Vis 7, 6. [DOI] [PubMed] [Google Scholar]
  25. Rubin GD, Roos JE, Tall M, Harrawood B, Bag S, Ly DL, Seaman DM, Hurwitz LM, Napel S, and Roy Choudhury K (2015). Characterizing search, recognition, and decision in the detection of lung nodules on CT scans: elucidation with eye tracking. Radiology 274, 276–286. [DOI] [PubMed] [Google Scholar]
  26. Zhang S, and Eckstein MP (2010). Evolution and optimality of similar neural mechanisms for perception and action during search. PLoS Comput. Biol 6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Zhang Y, Pham BT, and Eckstein MP (2004a). Automated optimization of JPEG 2000 encoder options based on model observer performance for detecting variable signals in X-ray coronary angiograms. IEEE Trans. Med. Imaging 23, 459–474. [DOI] [PubMed] [Google Scholar]
  28. Zhang Y, Pham B, and Eckstein MP (2004b). Evaluation of JPEG 2000 encoder options: human and model observer detection of variable signals in X-ray coronary angiograms. IEEE Trans. Med. Imaging 23, 613–632. [DOI] [PubMed] [Google Scholar]

RESOURCES