U.S. patent application number 14/582255 was filed with the patent office on 2016-06-30 for 3d image capture apparatus with cover window fiducials for calibration.
The applicant listed for this patent is 3M INNOVATIVE PROPERTIES COMPANY. Invention is credited to Aya Eid, James L. Graham, II, Eric S. Hansen, Vladimir Kryzhniy, Shannon D. Scott, David B. Stegall, Zhisheng Yun.
Application Number | 20160191901 14/582255 |
Document ID | / |
Family ID | 56151420 |
Filed Date | 2016-06-30 |
United States Patent
Application |
20160191901 |
Kind Code |
A1 |
Stegall; David B. ; et
al. |
June 30, 2016 |
3D IMAGE CAPTURE APPARATUS WITH COVER WINDOW FIDUCIALS FOR
CALIBRATION
Abstract
A 3D imaging apparatus with enhanced depth of field to obtain
electronic images of an object for use in generating a 3D digital
model of the object. The apparatus includes a housing having
mirrors positioned to receive an image from an object external to
the housing and provide the image to an image sensor. The optical
path between the object and the image sensor includes an aperture
element having apertures for providing the image along multiple
optical channels with a lens positioned within each of the optical
channels. The apparatus also includes a transparent cover
positioned within the optical path and having a plurality of
fiducials. The depth of field of the apparatus includes the cover,
allowing the fiducials to be used to calibrate the apparatus or
verify and correct the existing calibration of it.
Inventors: |
Stegall; David B.; (St.
Paul, MN) ; Kryzhniy; Vladimir; (St. Paul, MN)
; Hansen; Eric S.; (Arden Hills, MN) ; Eid;
Aya; (St. Paul, MN) ; Scott; Shannon D.;
(Hudson, WI) ; Yun; Zhisheng; (Woodbury, MN)
; Graham, II; James L.; (Woodbury, MN) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
3M INNOVATIVE PROPERTIES COMPANY |
St. Paul |
MN |
US |
|
|
Family ID: |
56151420 |
Appl. No.: |
14/582255 |
Filed: |
December 24, 2014 |
Current U.S.
Class: |
348/49 |
Current CPC
Class: |
A61B 1/00057 20130101;
H04N 17/002 20130101; A61B 1/00096 20130101; H04N 5/2253 20130101;
A61B 1/00172 20130101; H04N 13/246 20180501; G02B 27/32 20130101;
A61B 1/24 20130101; A61C 9/0053 20130101; H04N 5/2256 20130101;
H04N 13/257 20180501; A61B 1/00177 20130101; H04N 13/236 20180501;
G02B 27/0075 20130101; G03B 17/17 20130101; H04N 5/2252
20130101 |
International
Class: |
H04N 13/02 20060101
H04N013/02; G02B 27/32 20060101 G02B027/32; G02B 27/00 20060101
G02B027/00; G03B 17/17 20060101 G03B017/17; H04N 5/225 20060101
H04N005/225; G06T 7/00 20060101 G06T007/00 |
Claims
1. A 3D imaging apparatus, comprising: a housing; an image sensor
within the housing; a first mirror and a second mirror, the first
and second mirrors positioned to receive an image from an object
external to the housing and provide the image to the image sensor;
and an aperture element having a plurality of apertures, located
along an optical path between the object and the image sensor, for
providing the image along a plurality of optical channels
corresponding with the apertures to the image sensor; and a
transparent cover positioned within the optical path and having a
plurality of fiducials detectable by the image sensor, wherein a
depth of field of the apparatus includes the transparent cover.
2. The 3D imaging apparatus of claim 1, wherein the image sensor is
positioned substantially parallel to an object plane of the
object.
3. The 3D imaging apparatus of claim 1, wherein the aperture
element is located between the first and second mirrors and the
image sensor.
4. The 3D imaging apparatus of claim 1, wherein at least one of the
fiducials comprises a circle with an X in the circle.
5. The 3D imaging apparatus of claim 1, wherein at least one of the
fiducials comprises a first opaque color and a second opaque color
different from the first color.
6. The 3D imaging apparatus of claim 1, wherein at least one of the
fiducials comprises an opaque dot.
7. The 3D imaging apparatus of claim 1, wherein the fiducials are
located on an inside surface of the cover.
8. The 3D imaging apparatus of claim 1, further comprising a lens
positioned within each of the optical channels between the aperture
element and the image sensor.
9. The 3D imaging apparatus of claim 1, further comprising a light
source adjacent the cover for illuminating the object.
10. The 3D imaging apparatus of claim 1, wherein the image sensor
comprises a single image sensor partitioned into multiple regions
corresponding with the plurality of optical channels.
11. A 3D imaging apparatus, comprising: a housing; an image sensor
within the housing; a mirror positioned to receive an image from an
object external to the housing and provide the image to the image
sensor; and an aperture element having a plurality of apertures,
located along an optical path between the object and the image
sensor, for providing the image along a plurality of optical
channels corresponding with the apertures to the image sensor; and
a transparent cover positioned within the optical path and having a
plurality of fiducials detectable by the image sensor, wherein a
depth of field of the apparatus includes the transparent cover.
12. The 3D imaging apparatus of claim 11, wherein the image sensor
is positioned substantially perpendicular to an object plane of the
object.
13. The 3D imaging apparatus of claim 11, wherein the aperture
element is located between the mirror and the image sensor.
14. The 3D imaging apparatus of claim 11, wherein at least one of
the fiducials comprises a circle with an X in the circle.
15. The 3D imaging apparatus of claim 11, wherein at least one of
the fiducials comprises a first opaque color and a second opaque
color different from the first color.
16. The 3D imaging apparatus of claim 11, wherein at least one of
the fiducials comprises an opaque dot.
17. The 3D imaging apparatus of claim 11, wherein the fiducials are
located on an inside surface of the cover.
18. The 3D imaging apparatus of claim 11, further comprising a lens
positioned within each of the optical channels between the aperture
element and the image sensor.
19. The 3D imaging apparatus of claim 10, further comprising a
light source adjacent the cover for illuminating the object.
20. The 3D imaging apparatus of claim 10, wherein the image sensor
comprises a single image sensor partitioned into multiple regions
corresponding with the plurality of optical channels.
Description
BACKGROUND
[0001] Three-dimensional (3D) image scanners are typically
calibrated after assembly. The calibration process permits the
scanners to produce accurate 3D measurements of solid objects
placed in the field of view of the system. In addition, the
calibration process characterizes the thermal sensitivity of the
scanner during operation and removes the thermal-dependent error
from 3D measurements. As the scanner ages over the long-term, there
is the possibility that the physical state of the scanner can drift
from the originally calibrated state. The drifted state can cause
small but measurable errors in the 3D measurements of solid
objects. A field calibration target can be used to correct a more
severe aging-related drift from the initial calibration. However,
the field target calibration requires active participation by the
user of the scanner, which can be inconvenient and not necessarily
used when needed to recalibrate the scanner.
SUMMARY
[0002] A first 3D imaging apparatus, consistent with the present
invention, includes a housing and an image sensor within the
housing. First and second mirrors are positioned to receive an
image from an object external to the housing and provide the image
to the image sensor. An aperture element having a plurality of
apertures is located along an optical path between the object and
the image sensor for providing the image along a plurality of
optical channels to the image sensor. The apparatus also includes a
transparent cover positioned within the optical path and having a
plurality of fiducials. The depth of field of the apparatus
includes the transparent cover along with the fiducials.
[0003] A second 3D imaging apparatus, consistent with the present
invention, includes a housing and an image sensor within the
housing. A mirror is positioned to receive an image from an object
external to the housing and provide the image to the image sensor.
An aperture element having a plurality of apertures is located
along an optical path between the object and the image sensor for
providing the image along a plurality of optical channels to the
image sensor. The apparatus also includes a transparent cover
positioned within the optical path and having a plurality of
fiducials. The depth of field of the apparatus includes the
transparent cover along with the fiducials.
BRIEF DESCRIPTION OF THE DRAWINGS
[0004] The accompanying drawings are incorporated in and constitute
a part of this specification and, together with the description,
explain the advantages and principles of the invention. In the
drawings,
[0005] FIG. 1 is a side view of a 3D imager with depth of field
extension;
[0006] FIG. 2 is a diagram illustrating use of two fold mirrors for
depth of field extension;
[0007] FIG. 3 is a diagram illustrating use of two concave mirrors
for depth of field extension;
[0008] FIG. 4 is a perspective view of the 3D imager of FIG. 1;
[0009] FIG. 5 is an exploded perspective view of the 3D imager of
FIG. 1;
[0010] FIG. 6 is an exploded side view of the 3D imager of FIG.
1;
[0011] FIG. 7 is a side view of an alternative 3D imager with depth
of field extension;
[0012] FIG. 8 is a perspective view of the 3D imager of FIG. 7;
[0013] FIG. 9 is an exploded perspective view of the 3D imager of
FIG. 7;
[0014] FIG. 10 is a diagram illustrating two optical elements for
each optical channel in a 3D imager;
[0015] FIG. 11 is a diagram illustrating three optical elements for
each optical channel in a 3D imager;
[0016] FIG. 12 is a diagram illustrating four optical elements for
each optical channel in a 3D imager;
[0017] FIG. 13 is a diagram illustrating two image data regions on
an image sensor in a 3D imager for obtaining multiple views in a 3D
system;
[0018] FIG. 14 is a diagram illustrating three image data regions
on an image sensor in a 3D imager for obtaining multiple views in a
3D system;
[0019] FIG. 15 is a diagram illustrating a first type of cover
window fiducials for use relating to calibration;
[0020] FIG. 16 is a diagram illustrating a second type of cover
window fiducials for use relating to calibration; and
[0021] FIG. 17 is a diagram illustrating a third type of cover
window fiducials for use relating to calibration.
DETAILED DESCRIPTION
[0022] Embodiments include using fiducials on a cover window of a
3D scanner for use in calibrating the scanner or checking the
calibrated state of the scanner. An example of a 3D scanner having
a cover window within its depth of field is disclosed in U.S.
patent application Ser. No. 14/277,113, entitled "3D Image Capture
Apparatus with Depth of Field Extension," and filed May 14, 2014,
which is incorporated herein by reference as if fully set forth.
Systems to generate 3D images or models based upon image sets from
multiple views are disclosed in U.S. Pat. Nos. 7,956,862 and
7,605,817, both of which are incorporated herein by reference as if
fully set forth. These systems can be included in a housing
providing for hand-held use, and an example of such a housing is
disclosed in U.S. Pat. No. D674,091, which is incorporated herein
by reference as if fully set forth.
3D Image Capture Apparatus
[0023] FIG. 1 is a side view of a 3D imager 10 with depth of field
extension through the use of two mirrors. System 10 includes a
housing 12, mirrors 18 and 20, an aperture element 22, lenses 24,
and an image sensor 28. Housing 12 has an angled tip 17 with mirror
18 secured adjacent an interior surface of the tip. A mechanical
holder 26 is used to hold mirror 20, aperture element 22, and
lenses 24 in position over image sensor 28. A circuit board 30 can
receive electronic signals from image sensor 28 representing the
images and transmit the signals for further processing to generate
a 3D model of the object. Housing 12 includes a transparent cover
14 and light sources 16 adjacent the cover to illuminate an object
to be imaged. In this design, image sensor 28 is positioned
substantially parallel to an object plane of the object. The imager
has a depth of field 32 which includes housing 12, in particular a
bottom surface 13 of the housing. The depth of field can
alternatively include and extend into the inside of housing 12. By
having the depth of field include the housing, imager 10 can be
placed directly on (in physical contact with) an object to be
imaged, such as on teeth for intra-oral scanning.
[0024] For the configuration of imager 10 of FIG. 1, the image
plane (image sensor surface plane) 42 is positioned along the
horizontal plane with object plane 34, as shown in FIG. 2. Mirrors
36 and 38 provide an image of an object at object plane 34 through
a lens 40 to an image sensor at image plane 42. If the image sensor
surface is normal to the optical axis of lens 40, to achieve good
image quality over the entire field of view of object plane 34,
object plane 34 needs to be parallel to image plane 42. If .alpha.
is the angle of mirror 38 to image plane 42 and .beta. is the angle
of the mirror 36 to object plane 34, to have a good image quality
over the lens field of view, mirrors 36 and 38 have the following
relationship: .alpha.+.beta.=90.degree..
[0025] FIGS. 1 and 2 show a configuration using two planar fold
mirrors. For the two fold mirrors configuration, either of the two
fold mirrors, or both of the mirrors, can be implemented with
concave mirrors. If concave mirrors are used, the position of the
image sensor can be adjusted to compensate for the focus of the
concave mirror and obtain sharp images.
[0026] FIG. 3 illustrates a system using two concave mirrors.
Mirrors 46 and 48 provide an image of an object at object plane 44
through a lens 50 to an image sensor at image plane 52. If the
image sensor surface is normal to the optical axis of lens 50, to
achieve good image quality over the entire field of view of object
plane 44, object plane 44 needs to be parallel to image plane 52.
If .alpha.' is the angle of mirror 48 to image plane 52 and .beta.'
is the angle of the mirror 46 to object plane 44, to have a good
image quality over the lens field of view, mirrors 46 and 48 have
the following relationship: .alpha.'+.beta.'=90.degree..
[0027] FIGS. 4-6 are perspective, exploded perspective, and
exploded side views, respectively, of 3D imager 10 of FIG. 1. FIG.
5 illustrates apertures 23 in aperture element 22 to create
multiple channels. Although three apertures are shown, aperture
element 22 can alternatively have two apertures for a two channel
system.
[0028] The components of imager 10 can be implemented with, for
example, the following. Mirrors 18 and 20 can be aluminum or silver
coated on optical glass or metal. Mirror 18 can alternatively be a
prism, and mirror 20 can alternatively be a planar mirror plate. A
prism is used for mirror 20 for ease of holding the mirror in place
on holder 26. Mirrors 18 and 20 can optionally be one piece of
material with mirrors on both ends. Mirrors 18 and 20 are
preferably positioned at 50.degree. and 40.degree., respectively,
from the image plane. The angles of the mirrors should total
90.degree. for the image sensor to obtain images normal to the
target, and each of the angles can thus be adjusted for desired
placement in the housing. Lenses 24 can include separate lenses for
each channel or be a single molded piece of material. Exemplary
lens arrays are provided below. Aperture element 22 can be a
multi-layer metal plate, such as BeCu base with Ni plating, with
holes etched into it for the apertures 23. Holder 26 can be
aluminum or a molded plastic material, and mirror 20, aperture
element 22, and lenses 24 can be adhered to holder 26 or
mechanically held in place on the holder. Light sources 16 can be
light emitting diodes (LEDs). Cover 14 can be optical glass.
Housing 12 can be metal or a plastic material. The various
components of imager 10 in housing 12 can be positioned at
particular distances in the optical path for a desired
performance.
[0029] FIG. 7 is a side view of an alternative 3D imager 50 with
depth of field extension using one fold mirror. FIGS. 8 and 9 are
perspective and exploded perspective views, respectively, of 3D
imager 50 of FIG. 7. System 50 includes a housing 52, a mirror 58,
an aperture element 60, lenses 62, and an image sensor 64. Housing
52 has an angled tip 57 with mirror 58 secured adjacent an interior
surface of the tip. A circuit board 66 can receive electronic
signals from image sensor 64 representing the images and transmit
the signals for further processing to generate a 3D model of the
object. Housing 52 includes a transparent cover 54 and light
sources 56, such as LEDs, adjacent the cover to illuminate an
object to be imaged. In this design, image sensor 64 is positioned
substantially perpendicular to an object plane of the object. The
imager has a depth of field which includes housing 52, in
particular a bottom surface 53 of the housing. The depth of field
can alternatively include and extend into the inside of housing 52.
By having the depth of field include the housing, imager 50 can be
placed directly on (in physical contact with) an object to be
imaged, such as on teeth for intra-oral scanning.
[0030] FIG. 9 illustrates apertures 61 in aperture element 60 to
create multiple channels. Although three apertures are shown,
aperture element 60 can alternatively have two apertures for a two
channel system. Aperture element 60 can be on prism mirror 58, on
lenses 62, or in between mirror 58 and lenses 62 with gaps on both
sides of aperture element 60. Lenses 62 can be separate lenses or
one molded piece of material for each channel. The fold mirror in
imager 50 can be implemented with a concave mirror or a planar
mirror plate instead of the prism as shown. The components of
imager 50 can be implemented with the exemplary materials provided
above for imager 10.
[0031] Each of the optical channels in the 3D imagers can have
single or multiple optical elements. Multiple elements can achieve
superior imaging quality, large depth of field, and athermalized
system design. FIGS. 10-12 illustrate three options of the optics
for each channel. FIG. 10 illustrates two lenses 70 positioned
along an optical path normal to an image sensor 72. FIG. 11
illustrates three lenses 74 positioned along an optical path normal
to an image sensor 76. FIG. 12 illustrates four lenses 78
positioned along an optical path normal to an image sensor 80.
[0032] The images of the object formed on the image sensor are
located in two regions as shown in FIG. 13 for a two channel system
or three regions as shown in FIG. 14 for a three-channel system. In
FIG. 13, a first view-angle image 84 is captured in region 88 of an
image sensor 82, and second view-angle image 86 is captured in
region 90 of image sensor 82. In FIG. 14, a first view-angle image
94 is captured in region 100 of an image sensor 92, a second
view-angle image 96 is captured in region 102 of image sensor 92,
and a third view-angle image 98 is captured in region 104 of image
sensor 92.
[0033] The image sensors can be implemented with, for example, any
digital imager such as a CMOS or CCD sensor. The image sensor can
include a single sensor, as shown, partitioned into multiple image
data regions. Alternatively, the image sensor can be implemented
with multiple sensors with the image data regions distributed among
them.
Cover Window Fiducials
[0034] FIGS. 15-17 are diagrams illustrating examples of cover
window fiducials for use relating to calibration. FIG. 15
illustrates fiducials 110, 111, 112, and 113 located in corners of
cover 14. The fiducials have a known distance between them, for
example distance 114 between fiducials 110 and 111, and distance
115 between fiducials 110 and 112. These fiducials 110-113 are
indicated as circles filled with an "x." FIG. 16 illustrates
fiducials 120, 121, 122, and 123 located in corners of cover 14.
The fiducials have a known distance between them, for example
distance 124 between fiducials 120 and 121, and distance 125
between fiducials 120 and 122. These fiducials are indicated as
solid dots. FIG. 17 illustrates a row of fiducials 130 on one side
of cover 14 and another row of fiducials 131 on the opposing side
of cover 14. The fiducials have a known distance between them, for
example distance 133 between fiducials in row 130, distance 134
between fiducials in row 131, and distance 132 between the first
fiducials in rows 130 and 131. These fiducials in rows 130 and 131
are indicated as solid dots.
[0035] The fiducials can be implemented with two different opaque
colors. For example, the fiducials in FIG. 15 can be implemented
with a first opaque color for the circle and a second opaque color,
different from the first color, for the "x." As another example,
the fiducials in FIGS. 16 and 17 can be implemented with a first
opaque color for a portion of the solid dot and a second opaque
color, different from the first color, for another portion of the
solid dot. Aside from the fiducials shown in FIGS. 15-17, the
fiducials can be implemented with other shapes such as, for
example, triangles or squares. The fiducials are preferably located
at the edges of the cover window to be outside the main view of the
image sensor although still within view of and detectable by the
image sensor. Alternatively, the fiducials can be located anywhere
on the cover window within view of and detectable by the image
sensor. Although shown on cover 14 of scanner 10, the fiducials can
also be located on cover 54 of scanner 50 or on cover windows of
other 3D or multi-view scanners.
[0036] By placing these fiducials as precise features on the cover
window of a 3D scanner where the depth of field includes the cover
window, a permanent distance-measuring standard with virtually no
aging-related changes can be built into the scanner. During
operation of the system, the fiducial locations can acquired and
compared with the expected specifications. In particular, the
distance between the fiducials as detected by the image sensor and
associated system processing images from the image sensor can be
compared with the actual known distance on the cover. Any
discrepancy can then be used to correct errors caused by the aging
or even short-term temperature related variations of the scanner.
The fiducials can also be used for initial calibration of the
scanner.
[0037] The cover, such as transparent glass, can provide a
water-tight seal that does not interfere with imaging and permits
the frequent submersion into disinfecting solution. The fiducials
can include a variety of shapes and characteristics, and they
should be chosen so as to minimize the likelihood of interfering
with the scanning experience or calibration process. The fiducials
should also be chosen to maximize the likelihood of finding their
images during a typical scan. In particular, the fiducials
preferably are features that the scanner can detect via the image
sensor and associated processing and are not very noticeable to a
user. The fiducials are preferably opaque and consequently produce
a dark image of the fiducials in the video frames captured by the
scanner. Since the cover lies within the depth of field of the
system, the fiducials will also be in good focus. Ideally, the
fiducials are located on the inner surface of the cover so as to
not be exposed to the outside environment that could damage them
over time. Alternatively, the fiducials can be located on the outer
surface of the cover or embedded within the cover.
[0038] The distance separating the fiducials can be specified to a
manufacturer, where photolithographic processes can produce the
fiducials to within tolerances of .+-.1 micron, for example. In the
case that the fiducials are placed along the corners of a 5 mm
square configuration, for example, the resulting measurement
standard or ground truth would have a 0.02% tolerance along the
sides of the square configuration or a 0.014% tolerance along a
diagonal. Such an error is more than 5 times better than the
desired accuracy of the entire system for many embodiments.
[0039] By incorporating fiducials into the cover of a scanner, the
user need not be an active participant in the diagnostic process.
The fiducials can be a permanent feature available for measurement
in almost every video frame during the intended use of the system.
By being available for nearly every video frame, the fiducials
afford the system a method to track the thermal-state and thus
correct or augment the correction to thermal error in real time.
Although the cover when implemented with glass can expand and
contract due to thermal variations of the scanner, the thermal
sensitivity of the scanner is due to a much larger expansion and
contraction of the lens array when implemented with plastic where
mere microns of movement can produce significant fractions of a
percent in 3D model error. In particular, a typical glass cover has
a thermal coefficient of expansion around 10.sup.-6 (per degree
C.), meaning that two fiducials separated by 5 mm on room
temperature glass that warms up by 20.degree. C. would then be
separated by 5.001 mm. Such a small thermal response is far smaller
than the typical system can resolve. Furthermore, the cover with
permanent fiducials would be stable over time and thus provide a
ground truth isotropic correction for any potential long-term
drifting of the scanner from the calibrated state.
* * * * *