U.S. patent application number 15/878463 was filed with the patent office on 2020-08-20 for interactive 3d cursor for use in medical imaging.
The applicant listed for this patent is Kathleen M Douglas Douglas. Invention is credited to David Byron Douglas, Kathleen M Douglas, Robert E Douglas.
Application Number | 20200264709 15/878463 |
Document ID | 20200264709 / US20200264709 |
Family ID | 1000005001824 |
Filed Date | 2020-08-20 |
Patent Application | download [pdf] |
![](/patent/app/20200264709/US20200264709A9-20200820-D00000.png)
![](/patent/app/20200264709/US20200264709A9-20200820-D00001.png)
![](/patent/app/20200264709/US20200264709A9-20200820-D00002.png)
![](/patent/app/20200264709/US20200264709A9-20200820-D00003.png)
![](/patent/app/20200264709/US20200264709A9-20200820-D00004.png)
![](/patent/app/20200264709/US20200264709A9-20200820-D00005.png)
![](/patent/app/20200264709/US20200264709A9-20200820-D00006.png)
![](/patent/app/20200264709/US20200264709A9-20200820-D00007.png)
![](/patent/app/20200264709/US20200264709A9-20200820-D00008.png)
![](/patent/app/20200264709/US20200264709A9-20200820-D00009.png)
![](/patent/app/20200264709/US20200264709A9-20200820-D00010.png)
View All Diagrams
United States Patent
Application |
20200264709 |
Kind Code |
A9 |
Douglas; Kathleen M ; et
al. |
August 20, 2020 |
INTERACTIVE 3D CURSOR FOR USE IN MEDICAL IMAGING
Abstract
An interactive 3D cursor facilitates selection and manipulation
of a three-dimensional volume from a three-dimensional image. The
selected volume image may be transparency-adjusted and filtered to
remove selected tissues from view. Qualitative and quantitative
analysis of tissues in a selected volume may be performed. Location
indicators, annotations, and registration markers may be overlaid
on selected volume images.
Inventors: |
Douglas; Kathleen M; (Winter
Park, FL) ; Douglas; Robert E; (Winter Park, FL)
; Douglas; David Byron; (Winter Park, FL) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Douglas; Kathleen M
Douglas; Robert E
Douglas; David Byron |
Winter Park
Winter Park
Winter Park |
FL
FL
FL |
US
US
US |
|
|
Prior
Publication: |
|
Document Identifier |
Publication Date |
|
US 20190227641 A1 |
July 25, 2019 |
|
|
Family ID: |
1000005001824 |
Appl. No.: |
15/878463 |
Filed: |
January 24, 2018 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
14877442 |
Oct 7, 2015 |
9980691 |
|
|
15878463 |
|
|
|
|
12176569 |
Jul 21, 2008 |
9349183 |
|
|
14877442 |
|
|
|
|
11941578 |
Nov 16, 2007 |
8384771 |
|
|
12176569 |
|
|
|
|
60877931 |
Dec 28, 2006 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06T 2207/10081
20130101; A61B 6/466 20130101; G06T 7/62 20170101; G06F 3/03543
20130101; G06T 19/006 20130101; G06T 2207/30204 20130101; G06F
3/0346 20130101; G06T 2207/10104 20130101; G06F 3/04812 20130101;
G06F 3/04842 20130101 |
International
Class: |
G06F 3/0346 20060101
G06F003/0346; G06F 3/0481 20060101 G06F003/0481; G06F 3/0484
20060101 G06F003/0484; G06F 3/0354 20060101 G06F003/0354; G06T 7/62
20060101 G06T007/62; G06T 19/00 20060101 G06T019/00; A61B 6/00
20060101 A61B006/00 |
Claims
1. A method comprising generating a three-dimensional cursor that
has a non-zero volume; responsive to a first input, moving the
three-dimensional cursor within a three-dimensional image;
responsive to a second input, selecting a volume of the
three-dimensional image designated by the three-dimensional cursor;
and responsive to a third input, presenting a modified version of
the selected volume of the three-dimensional image.
2. The method of claim 1 wherein presenting the modified version of
the selected volume of the three-dimensional image comprises
removing an un-selected volume of the three-dimensional image from
view.
3. The method of claim 1 wherein presenting the modified version of
the selected volume of the three-dimensional image comprises
changing transparency of presented tissues within or external to
the selected volume.
4. The method of claim 1 wherein presenting the modified version of
the selected volume of the three-dimensional image comprises
filtering a selected tissue to remove the selected tissue from
view.
5. The method of claim 1 comprising presenting the
three-dimensional cursor with measurement markings on at least one
edge, surface or side.
6. The method of claim 1 wherein presenting the modified version of
the selected volume of the three-dimensional image comprises
presenting inputted location indicators.
7. The method of claim 1 wherein presenting the modified version of
the selected volume of the three-dimensional image comprises
presenting inputted annotations.
8. The method of claim 1 comprising changing a size dimension of
the three-dimensional cursor responsive to a fourth input.
9. The method of claim 8 comprising changing a geometric shape of
the three-dimensional cursor responsive to a fifth input.
10. The method of claim 1 comprising automatically generating a
statistical representation of the selected volume of the
three-dimensional image.
11. The method of claim 1 wherein presenting the modified version
of the selected volume of the three-dimensional image comprises
presenting at least one tissue type with false color.
12. The method of claim 1 wherein presenting the modified version
of the selected volume of the three-dimensional image comprises
presenting volumetric changes over time with false color.
13. The method of claim 1 comprising presenting multiple computed
tomography images associated with the selected volume using
reference lines.
14. The method of claim 1 comprising presenting multiple axial
computed tomography images associated with the selected volume
using reference lines.
15. The method of claim 1 comprising presenting a maximum intensity
projection (MIP) image of a positron emission tomography (PET) scan
with the three-dimensional cursor overlaid thereon to indicate
orientation and location of the selected volume.
16. The method of claim 1 comprising presenting a radiology report
enhanced with information obtained using the three-dimensional
cursor.
17. The method of claim 16 comprising automatically calculating and
presenting a quantitative analysis and a qualitative analysis
associated with multiple time points.
18. The method of claim 1 wherein presenting the modified version
of the selected volume of the three-dimensional image comprises
presenting inputted registration markers.
19. The method of claim 18 comprising automatically calculating
volumetric change based on the registration markers.
20. The method of claim 18 comprising automatically re-orienting
the selected volume of the three-dimensional image based on the
registration markers.
21. The method of claim 1 comprising using multiple volumes
selected with the three-dimensional cursor to designate a
pre-operative planning pathway for guiding surgical
intervention.
22. The method of claim 1 comprising presenting the selected volume
with an augmented reality, virtual reality or mixed reality
headset.
23. An apparatus comprising a computing device; and a human-machine
interface comprising a three-dimensional cursor that has a non-zero
volume; the human-machine interface moving the three-dimensional
cursor within a three-dimensional image responsive to a first
input; the human-machine interface selecting a volume of the
three-dimensional image designated by the three-dimensional cursor
responsive to a second input; and the human-machine interface
presenting a modified version of the selected volume of the
three-dimensional image responsive to a third input.
24. The apparatus of claim 23 wherein the human-machine interface
removes an un-selected volume of the three-dimensional image from
view.
25. The apparatus of claim 23 wherein the human-machine interface
changes transparency of presented tissues within the selected
volume.
26. The apparatus of claim 23 wherein the human-machine interface
filters a selected tissue to remove the selected tissue from
view.
27. The apparatus of claim 23 wherein the human-machine interface
presents the three-dimensional cursor with measurement markings on
at least one edge, surface or side.
28. The apparatus of claim 23 wherein the human-machine interface
receives and implements inputted location indicators.
29. The apparatus of claim 23 wherein the human-machine interface
receives and implements inputted annotations.
30. The apparatus of claim 23 wherein the human-machine interface
changes a size dimension of the three-dimensional cursor responsive
to a fourth input.
31. The apparatus of claim 30 wherein the human-machine interface
changes a geometric shape of the three-dimensional cursor
responsive to a fifth input.
32. The apparatus of claim 23 wherein the human-machine interface
automatically generates and presents a statistical representation
of the selected volume of the three-dimensional image.
33. The apparatus of claim 23 wherein the human-machine interface
presents at least one tissue type with false color.
34. The apparatus of claim 23 wherein the human-machine interface
presents volumetric changes over time with false color.
35. The apparatus of claim 23 wherein the human-machine interface
presents multiple computed tomography images associated with the
selected volume using reference lines.
36. The apparatus of claim 23 wherein the human-machine interface
presents multiple axial computed tomography images associated with
the selected volume using reference lines.
37. The apparatus of claim 23 wherein the human-machine interface
presents a maximum intensity projection (MIP) image of a positron
emission tomography (PET) scan with the three-dimensional cursor
overlaid thereon to indicate orientation and location of the
selected volume.
38. The apparatus of claim 23 wherein the human-machine interface
presents a radiology report enhanced with information obtained
using the three-dimensional cursor.
39. The apparatus of claim 23 wherein the human-machine interface
automatically calculates and presents a quantitative analysis and a
qualitative analysis associated with multiple time points.
40. The apparatus of claim 23 wherein the human-machine interface
presents inputted registration markers.
41. The apparatus of claim 40 wherein the human-machine interface
automatically calculates volumetric change based on the
registration markers.
42. The apparatus of claim 40 wherein the human-machine interface
automatically re-orients the selected volume of the
three-dimensional image based on the registration markers.
43. The apparatus of claim 23 wherein the human-machine interface
presents multiple volumes selected with the three-dimensional
cursor to designate a pre-operative planning pathway for guiding
surgical intervention.
44. The apparatus of claim 23 wherein the human-machine interface
presents the selected volume with an augmented reality, virtual
reality or mixed reality headset.
Description
TECHNICAL FIELD
[0001] Aspects of this disclosure are generally related to
human-machine interfaces, and more particularly to cursors.
BACKGROUND
[0002] The typical arrow-shaped cursor presented by a computer
operating system is zero-dimensional. A zero-dimensional cursor
designates the location of a single point in a space such as a
two-dimensional window presented on a monitor. Mouse buttons can be
used in combination with movement of the cursor to select objects
in the two-dimensional space, but at any given instant of time a
zero-dimensional cursor position designates only a single point in
space.
[0003] The current standard for diagnostic radiologists reviewing
computed tomography (CT) or magnetic resonance imaging (MRI)
studies is a slice-by-slice method. A conventional keyboard,
monitor, and mouse with a zero-dimensional cursor are used for
manipulating the images. The use of mouse buttons and cursor
movement for manipulating the images can become burdensome. For
example, many images are included in radiology studies that are
performed for the follow up of cancer to determine the response to
treatment. The ability to recognize and analyze differences between
images can be important. As an example, the recent Investigation of
Serial Studies to Predict Your Therapeutic Response with Imaging
and Molecular Analysis (I-SPY) trial tracked the changes in the
tumor over multiple magnetic resonance imaging (MM) scans during
the administration of neoadjuvant chemotherapy (NACT). It has been
noted that the phenotypic appearance (e.g., shape, margins) of a
tumor correlated with the pathologic response to NACT. A more
efficient and accurate interface for manipulating and presenting
medical images would therefore have utility.
[0004] Known techniques for 3D viewing of medical images are
described in U.S. Pat. No. 9,349,183, Method and Apparatus for
Three Dimensional Viewing of Images, issued to Douglas, U.S. Pat.
No. 8,384,771, Method and Apparatus for Three Dimensional Viewing
of Images, issued to Douglas, Douglas, D. B., Petricoin, E. F.,
Liotta L., Wilson, E. D3D augmented reality imaging system: proof
of concept in mammography. Med Devices (Auckl), 2016;9:277-83,
Douglas, D. B., Boone, J. M., Petricoin, E., Liotta, L., Wilson, E.
Augmented Reality Imaging System: 3D Viewing of a Breast Cancer. J
Nat Sci. 2016;2(9), and Douglas, D. B., Wilke, C. A., Gibson, J.
D., Boone, J. M., Wintermark, M. Augmented Reality: Advances in
Diagnostic Imaging. Multimodal Technologies and Interaction,
2017;1(4):29. In D3D imaging, the radiologist wears an augmented
reality (AR), mixed reality (MR) or virtual reality (VR) headset
and uses a joystick or gaming controller. Advantages include
improved depth perception and human machine interface. Still, there
are several challenges faced with this approach. First, an area of
interest (e.g. tumor) may be in close proximity to structures that
are similar in composition/density. Isolating the area of interest
for better examination may be difficult. Second, many soft tissues
in the body are mobile and deformable, so it can be difficult to
achieve the best orientation to properly compare the tumor at
multiple time points. Efficiently aligning the orientation to do so
may be difficult. Third, certain portions of a tumor can respond to
treatment and decrease in size while other portions of a tumor
demonstrate increases in size. The pattern of tumor shrinkage has
important prognostic implications. Furthermore, composition and
complex morphologic features including spiculations (spikes
extending from the surface), irregular margins and enhancement also
have important implications. Consequently, there is a need for a
system that facilitates recognition of the subtle, yet important
changes in size, shape and margins. Fourth, a patient with
metastatic cancer has several areas of interest in different areas
of the body. It is difficult and time consuming to find each of the
areas of interest at every time point to determine interval change.
Consequently, there is a need for a system that enables the
observer to do this efficiently.
SUMMARY
[0005] All examples, aspects and features mentioned in this
document can be combined in any technically possible way.
[0006] In accordance with an aspect of the invention a method
comprises: generating a three-dimensional cursor that has a
non-zero volume; responsive to a first input, moving the
three-dimensional cursor within a three-dimensional image;
responsive to a second input, selecting a volume of the
three-dimensional image designated by the three-dimensional cursor;
and responsive to a third input, presenting a modified version of
the selected volume of the three-dimensional image. In some
implementations presenting the modified version of the selected
volume of the three-dimensional image comprises removing an
un-selected volume of the three-dimensional image from view. In
some implementations presenting the modified version of the
selected volume of the three-dimensional image comprises changing
transparency of presented tissues within the selected volume. In
some implementations presenting the modified version of the
selected volume of the three-dimensional image comprises filtering
a selected tissue to remove the selected tissue from view. In some
implementations presenting the three-dimensional cursor with
measurement markings on at least one edge, surface or side. In some
implementations presenting the modified version of the selected
volume of the three-dimensional image comprises presenting inputted
location indicators. In some implementations presenting the
modified version of the selected volume of the three-dimensional
image comprises presenting inputted annotations. Some
implementations comprise changing a size dimension of the
three-dimensional cursor responsive to a fourth input. Some
implementations comprise changing a geometric shape of the
three-dimensional cursor responsive to a fifth input. Some
implementations comprise automatically generating a statistical
representation of the selected volume of the three-dimensional
image. In some implementations presenting the modified version of
the selected volume of the three-dimensional image comprises
presenting at least one tissue type with false color. In some
implementations presenting the modified version of the selected
volume of the three-dimensional image comprises presenting
volumetric changes over time with false color. Some implementations
comprise presenting multiple computed tomography images associated
with the selected volume using reference lines. Some
implementations comprise presenting multiple axial computed
tomography images associated with the selected volume using
reference lines. Some implementations comprise presenting a maximum
intensity projection (MIP) image of a positron emission tomography
(PET) scan with the three-dimensional cursor overlaid thereon to
indicate orientation and location of the selected volume. Some
implementations comprise presenting a radiology report enhanced
with information obtained using the three-dimensional cursor. Some
implementations comprise automatically calculating and presenting a
quantitative analysis and a qualitative analysis associated with
multiple time points. Some implementations comprise presenting the
modified version of the selected volume of the three-dimensional
image comprises presenting inputted registration markers. Some
implementations comprise automatically calculating volumetric
change based on the registration markers. Some implementations
comprise automatically re-orienting the selected volume of the
three-dimensional image based on the registration markers. Some
implementations comprise using multiple volumes selected with the
three-dimensional cursor to designate a pre-operative planning
pathway for guiding surgical intervention. Some implementations
comprise presenting the selected volume with an augmented reality,
virtual reality or mixed reality headset.
[0007] In accordance with an aspect of the invention an apparatus
comprises: a computing device; and a human-machine interface
comprising a three-dimensional cursor that has a non-zero volume;
the human-machine interface moving the three-dimensional cursor
within a three-dimensional image responsive to a first input; the
human-machine interface selecting a volume of the three-dimensional
image designated by the three-dimensional cursor responsive to a
second input; and the human-machine interface presenting a modified
version of the selected volume of the three-dimensional image
responsive to a third input. In some implementations, the
human-machine interface removes an un-selected volume of the
three-dimensional image from view. In some implementations, the
human-machine interface changes transparency of presented tissues
within the selected volume. In some implementations, the
human-machine interface filters a selected tissue to remove the
selected tissue from view. In some implementations, the
human-machine interface presents the three-dimensional cursor with
measurement markings on at least one edge, surface or side. In some
implementations, the human-machine interface receives and
implements inputted location indicators. In some implementations,
the human-machine interface receives and implements inputted
annotations. In some implementations, the human-machine interface
changes a size dimension of the three-dimensional cursor responsive
to a fourth input. In some implementations, the human-machine
interface changes a geometric shape of the three-dimensional cursor
responsive to a fifth input. In some implementations, the
human-machine interface automatically generates and presents a
statistical representation of the selected volume of the
three-dimensional image. In some implementations, the human-machine
interface presents at least one tissue type with false color. In
some implementations, the human-machine interface presents
volumetric changes over time with false color. In some
implementations, the human-machine interface presents multiple
computed tomography images associated with the selected volume
using reference lines. In some implementations, the human-machine
interface presents multiple axial computed tomography images
associated with the selected volume using reference lines. In some
implementations, the human-machine interface presents a maximum
intensity projection (MIP) image of a positron emission tomography
(PET) scan with the three-dimensional cursor overlaid thereon to
indicate orientation and location of the selected volume. In some
implementations, the human-machine interface presents a radiology
report enhanced with information obtained using the
three-dimensional cursor. In some implementations, the
human-machine interface automatically calculates and presents a
quantitative analysis and a qualitative analysis associated with
multiple time points. In some implementations, the human-machine
interface presents inputted registration markers. In some
implementations, the human-machine interface automatically
calculates volumetric change after appropriate registration using
the registration markers. In some implementations, the
human-machine interface automatically re-orients the selected
volume of the three-dimensional image based on the registration
markers. In some implementations, the human-machine interface
presents multiple volumes selected with the three-dimensional
cursor to designate a pre-operative planning pathway for guiding
surgical intervention. In some implementations, the human-machine
interface presents the selected volume with an augmented reality,
virtual reality or mixed reality headset.
BRIEF DESCRIPTION OF THE FIGURES
[0008] The patent or application file contains at least one drawing
executed in color. Copies of this patent or patent application
publication with color drawing(s) will be provided by the Office
upon request and payment of the necessary fee.
[0009] FIG. 1A illustrates a 3D cursor selecting a volume of
interest from a three-dimensional medical image.
[0010] FIG. 1B illustrates the volume of interest selected with the
3D cursor;
[0011] unselected portions have been removed from view.
[0012] FIG. 1C illustrates modification of the transparency of the
selected volume of interest.
[0013] FIG. 1D illustrates filtering of selected areas of the
selected volume of interest.
[0014] FIG. 2 illustrates a variant of the 3D cursor of FIG. 1A
with measurement markings on edges and sides.
[0015] FIG. 3 illustrates location indicators and annotations
positioned relative to the portion of the image within the selected
volume of interest.
[0016] FIGS. 4A, 4B, and 4C illustrate three different examples of
geometric shapes of the 3D cursor of FIG. 1A.
[0017] FIG. 5 illustrates presentation of a quantitative analysis
of tissues inside of the volume of interest selected with the 3D
cursor of FIG. 1A.
[0018] FIG. 6 illustrates use of false color and transparency
changes to enhance viewing of the selected volume of interest.
[0019] FIG. 7 illustrates association of multiple computed
tomography (CT) images of the chest in lung windows with the
interactive 3D cursor using reference lines.
[0020] FIG. 8 illustrates association of multiple axial computed
tomography (CT) slices of the chest in lung windows with the
interactive 3D cursor using reference lines.
[0021] FIG. 9 illustrates a maximum intensity projection (MIP)
image of a fludeoxyglucose (18F) positron emission tomography (PET)
scan in which two varying sized interactive 3D cursors are overlaid
to indicate 3D cursor shape, size, orientation, and location when
respective volumes of interest were selected.
[0022] FIG. 10 illustrates a radiology report enhanced with
information obtained using the interactive 3D cursor and including
quantitative and qualitative analysis.
[0023] FIG. 11 illustrates a radiology report enhanced with
information obtained using the interactive 3D cursor, and including
added quantitative and qualitative analysis at multiple time
points.
[0024] FIGS. 12A, 12B and 12C illustrate a technique for correction
for mis-registration at multiple time points using three or more
markers.
[0025] FIG. 13 illustrates use of multiple interactive 3D cursors
to select volumes of interest to designate a safe pre-operative
planning pathway for guiding surgical intervention.
[0026] FIG. 14 illustrates use of the interactive 3D cursor in an
educational setting.
[0027] FIG. 15 illustrates process steps on a radiologist's review
of a patient's image with integration of the interactive 3D
cursor.
[0028] FIG. 16 illustrates a system for use of the interactive 3D
cursor.
DETAILED DESCRIPTION
[0029] Some aspects, features and implementations described herein
may include machines such as computers, electronic components,
radiological components, optical components, and processes such as
computer-implemented steps. It will be apparent to those of
ordinary skill in the art that the computer-implemented steps may
be stored as computer-executable instructions on a non-transitory
computer-readable medium. Furthermore, it will be understood by
those of ordinary skill in the art that the computer-executable
instructions may be executed on a variety of tangible processor
devices. For ease of exposition, not every step, device or
component that may be part of a computer or data storage system is
described herein. Those of ordinary skill in the art will recognize
such steps, devices and components in view of the teachings of the
present disclosure and the knowledge generally available to those
of ordinary skill in the art. The corresponding machines and
processes are therefore enabled and within the scope of the
disclosure.
[0030] FIG. 1A illustrates a 3D (three-dimensional) cursor 100
overlaid on a three-dimensional medical image 102. In the
illustrated example, the 3D cursor 100 defines a cubic volume of
interest. The medical image 102 could include any portion of a
body, or an entire body, for example and without limitation. For
purposes of explanation the medical image 102 includes different
types of tissue. More specifically, the image includes a background
material 104, such as fat, a lobulated mass 106, a tubular-shaped
vein 108, and an artery 110. The 3D cursor 100 can be moved
relative to the image, e.g. in three dimensions, such as by
manipulating an IO device such as a 3D mouse, for example and
without limitation. A button click or other input designates
(selects) the portion of the image that is located inside the
three-dimensional volume of the 3D cursor 100. Distinguishing
between a 3D image portion selected by a 3D cursor and other
unselected image portions is described in US 2016/0026266 and U.S.
Pat. No. 8,384,771, both of which are incorporated by
reference.
[0031] FIG. 1B illustrates the selected image portion of FIG. 1A.
More particularly, unselected portions of the image located outside
of an image portion 112 selected with the 3D cursor 100 have been
filtered-out or otherwise completely removed from view.
Consequently, the removed portions of the image do not obstruct or
hinder the view of the selected image portion. Moreover, the
selected image portion 112 can be manipulated and viewed as a
separate and distinct image from the larger medical image 102 from
which it was selected.
[0032] FIG. 1C illustrates modification of the transparency of the
selected image portion 112. More specifically, transparency may be
decreased and/or increased such that tissues and other features can
be better observed, e.g. such that overlapping tissues and features
are visible. For example, tissues and features located proximate to
the back of the selected image portion such as lobulated mass 106
can be seen through overlapping tissues and features located
proximate to the front of the selected image portion such as vein
108, when transparency is sufficiently increased. The transparency
may be manipulated with the IO device to achieve various levels of
transparency. Further, different levels of transparency may be
applied to different portions of the selected image portion.
[0033] FIG. 1D illustrates filtering of selected areas or tissues
of the selected image portion 112 to remove those areas or tissues
from view. In the illustrated example the background material 104,
vein 108, and an artery 110 have been removed from view, leaving
only the lobulated mass 106. The tissues to be filtered (removed
from view) may be selected based on geometric shape, color,
brightness, density, and any other of a variety of available image
data, either alone or in combination. Moreover, a designated volume
defined by a geometric shape may be removed, e.g. a geometric shape
that traverses tissue boundaries.
[0034] Transparency modification and tissue filtering facilitate
presentation of certain tissue types of concern, both within the
cursor and outside of the cursor. Currently, the medical
professional must see through any tissue within the cursor but
external to the tissue type of concern from the viewing point of
the medical professional, thus degrading the visibility of the
tissue of concern. The illustrated improvements enable the medical
professional to change the transparency of any tissue within the
cursor-defined volume but external to the tissue type of concern.
Alternatively, tissue types not of concern are subtracted from the
volume contained within the interactive 3D cursor, leaving only the
tissue of concern in the presented image. Multiple interactive 3D
cursors in combination can be used to obtain varying patterns of
tissue subtraction. This helps to overcome the limitations of
degraded visibility due to tissue within the cursor but external to
the tissue type of concern from the viewing point of the medical
professional.
[0035] FIG. 2 illustrates an implementation of the 3D cursor 100
with dimensional measurement markings. Dimensional measurement
markings may be available as a feature that can be turned ON and
OFF. In the illustrated example, the 3D cursor is a 2 cm by 2 cm by
2 cm cube. The dimensional measurement markings include tick marks
200, 202, and 204 that respectively designate 1 mm, 5 mm, and 1 cm
increments along the edges of the cube (and thus representing three
dimensions). Tick marks that represent different magnitudes may be
uniquely represented to facilitate visual size determination of the
lobulated mass 106 that represents the lesion of interest. 1 cm
markings 206 are presented in each of two dimensions on each side
of the cube.
[0036] The dimensional measurement markings can help serve as a
reference for radiologist's activities to include visual
assessment, orientation, comparisons with prior scans or
measurements. Advantages may include mitigating the current lack of
metrics are available to the medical professional to understand the
size of the cursor and/or of the tissue elements contained within
the cursor. This implementation places measurement metrics on each
edge and side of the cursor to help enable the medical professional
to rapidly understand the size of the subtended volume within the
cursor. In the case where the cursor encapsulates a volume of
concern such as a tumor, the three-dimensional size could be
recorded in the medical professional report. This can help the
visual assessment of each portion of the tumor to aid in the
assessment of small changes in size of findings including
lobulations of a mass's margin and spiculations.
[0037] Referring to FIG. 3, location indicators 300 and annotations
302 may be placed by the radiologist or by automated techniques to
highlight locations or regions of concern within the interactive 3D
cursor. The location indicators may specify a point or region
within the volume of the 3D cursor. Annotations can be added
manually by the radiologist or by automated techniques to describe
areas that are of concern, e.g., growing, spiculation, irregular
margin, indistinct margin, etc. If spiculations are on the surface
of a tumor, this could be an indicator of potential malignancy. The
location indicators, such as, but not limited to, arrow(s) pointing
to key regions of interest within/outside the 3D cursor helps to
overcome the limitation of the inability to mark key points within
the cursor. This feature will be useful in discussions between
medical professions regarding a patient's condition. It will also
be useful in communicating imaging findings between a medical
professional and a patient.
[0038] Referring to FIGS. 4A, 4B, and 4C, the 3D cursor may be may
be implemented in a wide variety of different shapes. Examples
include but are not limited to cube, cuboid, cylinder, sphere,
ellipsoid, cone and tetrahedron. The shapes are not necessarily
regular, and the lengths of edges may be resized, e.g. overall
geometric shape scaling or changing individual edges, sides, or
surfaces. For example, FIGS. 4A and 4B illustrate cuboid 3D cursors
400, 402 for which edge length has been set or selected based on
the dimensions and orientation of the respective feature of
interest 404, 406. FIG. 4C illustrates a spherical 3D cursor 408
for which the diameter may be set or selected based on the
dimensions of the feature of interest 410. In addition to
dimensional changes, cursor geometric shape may be changed.
[0039] The ability to change the size, shape, and individual
dimensions of the 3D cursor enables the cursor to be customized
based on the particular volume of interest to the medical
professional. A fixed-shape, fixed-size cursor might be too large
or too small, e.g. so as to include a significant amount of tissue
not of interest. For example, in examining the lungs, placement of
a cube-shaped cursor could cause ribs to be included in the image.
Changing the shape of the 3D cursor would help to overcome this
limitation. Customization could be accomplished by wide variety of
techniques, possibly including but not limited to selecting an
edge, side or vertex of the original 3D cursor with a second type
of cursor 412, and then "clicking and dragging" the selected edge,
side, or vertex in the desired direction to expand or reduce the
volume of the original 3D cursor. The interface may also enable
selection and change between multiple 3D geometric shapes, e.g.
changing from cuboid to spherical. Scrolling on the conventional
slices while simultaneously drawing shapes can also be performed to
generate the prescribed 3D cursor volume. The interactive 3D cursor
thus provides an efficient interface for tissue subtraction to
provide enhanced visualization of the tumor.
[0040] FIG. 5 illustrates presentation of a quantitative analysis
500 of all tissues inside a volume selected with the 3D cursor. The
illustrated example includes a bar graph but it is to be understood
that any of a wide variety of charts, graphs, and other techniques
for presentation of data might be implemented. Quantitative
analysis can help the radiologist understand how a feature of
interest such as tumor 502 (e.g., the lobulated mass 106, FIG. 1B)
is changing in volume 504 over multiple time points. The interface
may include a statistical representation of the tissue types,
possibly including but not limited to a histogram bar chart to
depict the volume (e.g., number of voxels per unit volume) of the
different types of tissue within the cursor, distinct markings for
different types of tissue such as, but not limited to, color coding
the bars of the histogram bar chart.
[0041] FIG. 6 illustrates an implementation of the interactive 3D
cursor 100 with false color and transparency to enhance viewing.
False color and transparency may be dynamically adjusted and turned
ON and OFF. Different false colors may be applied to different
tissue types within the volume of the 3D cursor. The colors could
be selected to correspond to the colors used in the statistical
representation (FIG. 5). Alternatively, a respective unique false
color could be selected for each different tissue type, or tissue
types of particular interest or concern, and/or additional features
of concern, e.g., irregular margin, indistinct margin, spiculation,
etc. In the illustrated example, the background material 104 (fat)
is depicted in light gray, the artery 110 is depicted in red, the
vein 108 is depicted in blue, and the lobulated mass 106 is
multicolored. Different colors may be selected or used to indicate
stability of the lobulated mass 106 over time. For example, green
may be used to indicate a stable volume 112 while orange is used to
denote a slow growth volume 114, thereby providing a visual warning
indicator. Red may be used to indicate high rate of growth or
concerning margin volume 116. The extent of the volume of the
lobulated mass can be determined automatically, e.g. based on
density. Moreover, changes in volume of sub-regions of the
lobulated mass may also be automatically determined, and color
coding may be automatically implemented. This can help the
radiologist understand how the mass is changing in volume over
multiple time points.
[0042] FIG. 7 illustrates association of multiple computed
tomography (CT) images of the chest in lung windows with the
interactive 3D cursor 100 using reference lines 700. The
illustrated example includes an axial image 702, a sagittal image
704, and a coronal image 706 of the chest in lung windows. An
advantage is enhanced ability to cross reference the 3D cursor to
the original 2D slices 702, 704, 706 from which total 3D volume was
obtained. Medical professionals have experience and familiarity
with 2D slices and may feel more confident in their findings given
the capability to switch back and forth between the 2D and 3D
volumetric approaches. A small display adjacent to the interactive
3D cursor could indicate which 2D slices contain tissue within in
the interactive 3D cursor. Then the medical professional could
direct the system to automatically select those slices which have
tissue within the cursor and display them on a nearby 2D display
unit. A corresponding visible boundary of the 3D cursor (e.g., red)
projected on each of the slices may be presented.
[0043] FIG. 8 illustrates association of multiple axial computed
tomography (CT) slices 800, 802, 804, 806 of the chest in lung
windows with the interactive 3D cursor 100 using reference lines
808. The multiple axial computed tomography (CT) slices of the
chest in lung windows show the location of the 3D cursor, i.e. the
slice area that includes a cross-section of the 3D cursor, which in
the illustrated example has selected a left upper lobe mass.
Boundaries 810 of the 3D cursor in the slices are depicted in a
color, e.g. red. Within the 3D cursor the lung cancer mass 106 is
depicted in gray, surrounded by black that indicates non-cancerous
lung tissue. This implementation helps the medical professional to
rapidly visualize where the interactive 3D cursor is located
relative to the slice images and the body. It also enables the
medical professional to visualize the entire volumetric data with
the interactive 3D cursor accurately positioned within the volume.
Transparency of tissue within the 3D volume could be changed so
that the interactive 3D cursor would stand out. This would help
avoid left--right orientation mistakes that might occur during
treatment. Multiple interactive 3D cursors which could be of
differing sizes and/or shapes could be created and displayed.
[0044] FIG. 9 illustrates overlay of 3D cursors 100a, 100b on a
maximum intensity projection (MIP) image 900 of a fludeoxyglucose
(18F) positron emission tomography (PET) scan. Two different-sized
interactive 3D cursors are used to highlight two separate areas of
concern, including 3D cursor 100a for a right lung mass and cursor
100b for a vertebral body metastasis. This helps to automatically
transfer data (e.g., picture of tissue within the cursor and
statistical representations) from the viewing modality to the
report of findings. Selection of key data through human machine
interface such as, but limited to, a screen capture can be
automatically transferred to the report of findings. This would
provide quantitative results within the report together with
qualitative impressions of the medical professional.
[0045] FIG. 10 illustrates a radiology report 1000 enhanced with
information obtained from the interactive 3D cursor. Qualitative
findings 1002 and quantitative findings 1004 may be included along
with patient identifying information 1006, clinical history 1008,
comparisons 1010, conclusions 1012, and recommendations 1014. Also
included are a selected volume image 1016 and statistical graphic
1018. This helps to quantitatively track changes in volumes of
concern (e.g., tumors) over time.
[0046] FIG. 11 illustrates a radiology report 1100 enhanced with
information obtained from the interactive 3D cursor at multiple
time points. Qualitative findings 1002 and quantitative findings
1004 may be included along with patient identifying information
1006, clinical history 1008, comparisons 1010, conclusions 1012,
and recommendations 1014. Also included are selected volume images
1102, 1104 from different time points and respective statistical
graphics 1106, 1108 from those time points. Follow up reports can
include current and prior exams 1110, 1112 with quantitative
analysis and qualitative analysis on how the lesion has changed
over time. This may facilitate selection of a lesion (e.g., tumor)
at multiple time points using an interactive 3D cursor; qualitative
assessment of the lesion at multiple time points; and, quantitative
assessment of the lesion at multiple time points. This would enable
the medical professional to better assess how a particular lesion
is changing over time. A report of current findings as outlined in
the previous implementation could be placed in a report together
with the data obtained from an earlier examination. This would
enable tracking over time the progress of treatment or that of
changes in tissues of interest/concern.
[0047] FIGS. 12A, 12B, and 12C illustrate a registration technique
by which mis-registration can be corrected at multiple time points
through the use of three or more markers 12, 14, 16. Initially, the
mass 106 within each 3D cursor 100 image is noted using different
locations within the interactive 3D cursor and different
orientations. Next, the user marks similar locations on each image
of the mass with registration markers. In the illustrated example,
a yellow marker 12, a red marker 14, and a blue marker 16
correspond to the same respective parts of the mass on each scan.
Finally, tissues within the interactive 3D cursor are aligned in
accordance with markers. Many soft tissues within the body can
change in orientation from one scan to the next due to patient
movement. Corresponding mis-registration can limit the ability to
properly track how a lesion changes over time. This technique
provides a method to correct for such mis-registration. Three or
more recognizable spots of the lesion (e.g., tumor) can be marked
with a false color, arrow, or other registration mark. Then, these
locations can be automatically aligned with one another. Shadows
can be added to help bring out depth perception. Proper alignment
will accurately align the shadows. This enhances visual assessment
for how a lesion is changing over time to include changes in tumor
composition, size and morphology.
[0048] FIG. 13 illustrates use of multiple image volumes selected
with the 3D cursor to designate a safe pre-operative planning
pathway to guide surgical intervention. In the illustrated example,
multiple green interactive 3D cursors 1300 mark a surgeon-selected
dissection pathway that is deemed safe in the pre-operative
setting. The interactive 3D cursor 100 containing the cancerous
lesion 106 is shown at a distal end of the planned surgical path
represented by abutting or overlapping volumes selected with the 3D
cursors 1300. The selected path that the surgeon will excise avoids
the artery 110 with a minimum clearance of 10 mm. This provides the
advantage of 3D depiction of possible surgical cuts. The path could
include, but is not limited to, one or more of the following
properties: a serpentine shape; measurements could subsequently be
made to measure absolute distance between a point on the planned
path to some region of concern (e.g., artery); the path could also
be projected on a head mounted display at different intervals
during the course of the operation. This feature would facilitate
surgical planning as well as a potential to improve accuracy of the
surgery.
[0049] FIG. 14 illustrates use of the interactive 3D cursor in an
educational setting. Students 1400 are depicted wearing AR
(augmented reality) headsets 1402 and an instructor 1404 is
pointing to an abnormality on the board 1406. This facilitates
presentation of medical information (e.g., anatomy) in a classroom
environment. The interactive 3D cursor could be placed around the
organ of interest and other parts of the body could be eliminated.
Items from implementations discussed above such as metrics and
arrows could be used. The students would be provided 3D head
displays and joined into a display system so that they could see
the tissue within the interactive 3D cursor. This would eliminate
any confusion on the part of the students as to what specific
detail in the imagery was being discussed.
[0050] FIG. 15 illustrates process steps on a radiologist's review
of a patient's image with integration of the interactive 3D cursor
into his/her practice. Step 1 is to create an interactive 3D cursor
volume and shape that approximates the size and shape of patient
organ/tissue corresponding to the item currently being inspected on
the checklist. Step 2 is to position the interactive 3D cursor over
the organ/tissue to be inspected. The interactive 3D cursor as it
is located within the total 3D image volume may be presented on a
display. Step 3 is to subtract from view all tissue external to the
interactive 3D cursor. The interactive 3D cursor may be rotated to
permit viewing from multiple angles. If interactive cursors are
used at multiple time points to track how a particular lesion
(e.g., tumor) changes over time, the 3D cursors can be rotated in
synchrony with on another. Step 4 is to generate a statistical
representation e.g., a histogram of tissue densities--color coded
with the types of tissue that are suspicious. Step 5 is to subtract
from view additional tissue within the interactive 3D cursor as
deemed appropriate by the medical professional. Step 6 is to
inspect the volume within the cursor and identify region(s) of
interest and place indicators, annotations, and registration
markers relative to region(s) of interest. Step 7 is to extract a
statistical representation and capture imagery showing indicators,
annotations, and registration markers and residual tissue within
the interactive 3D cursor to be inserted into the medical
professional's report. Step 8 is to use cross-referencing as
described the above to confirm findings. Step 9 is to iterate on
the other items on the checklist until finished. Step 10 is to
prepare the report of the medical professional's findings. This
procedure provides an opportunity to enhance medical image review
process by medical professionals.
[0051] FIG. 16 illustrates a system for use of the interactive 3D
cursor. A medical imaging device 1600 is connected to a computer
workstation 1602. A wide variety of medical imaging devices and
computer workstations could be used. Images are captured by the
medical imaging device and sent to the computer workstation. The
computer workstation includes non-volatile storage,
computer-readable memory, processors, and a variety of other
resources including but not limited to IO devices that provide a
human-machine interface. In the illustrated example, the IO devices
include a monitor 1604, keyboard 1606, 3D mouse 1608, and VR
headset 1610. The IO devices are used to prompt a software program
that runs on the computer workstation to perform the various
process steps and implement the various features that have already
been described above.
[0052] There are multiple potential advantages of the interactive
3D cursor. For example, there is reduction in time spent for
classification of multiple lesions. The radiologist doesn't have to
sort through many prior imaging studies to find the lesion and the
interactive 3D cursor will save time. There is reduction in error
when tracking multiple lesions, i.e. reducing the likelihood of
mistakes when identifying different specific lesions that are
nearby one another when comparing multiple scans. One possibility
is to analyze the images obtained using the 3D cursor and using
multiple uniquely tagged (e.g. numbered) cursors for any suspicious
regions. The medical profession could then switch to slices for
confirmation.
[0053] Several features, aspects, embodiments and implementations
have been described. Nevertheless, it will be understood that a
wide variety of modifications and combinations may be made without
departing from the scope of the inventive concepts described
herein. Accordingly, those modifications and combinations are
within the scope of the following claims.
* * * * *