U.S. patent application number 17/206581 was filed with the patent office on 2021-07-08 for focus scanning apparatus.
This patent application is currently assigned to 3SHAPE A/S. The applicant listed for this patent is 3SHAPE A/S. Invention is credited to Rune FISKER, Karl-Josef HOLLENBECK, Rasmus KJ R, Henrik OJELUND, Arish A. QAZI, Mike van der POEL.
Application Number | 20210211638 17/206581 |
Document ID | / |
Family ID | 1000005466467 |
Filed Date | 2021-07-08 |
United States Patent
Application |
20210211638 |
Kind Code |
A1 |
FISKER; Rune ; et
al. |
July 8, 2021 |
FOCUS SCANNING APPARATUS
Abstract
A scanner includes a camera, a light source for generating a
probe light incorporating a spatial pattern, an optical system for
transmitting the probe light towards the object and for
transmitting at least a part of the light returned from the object
to the camera, a focus element within the optical system for
varying a position of a focus plane of the spatial pattern on the
object, unit for obtaining at least one image from said array of
sensor elements, unit for evaluating a correlation measure at each
focus plane position between at least one image pixel and a weight
function, a processor for determining the in-focus position(s) of
each of a plurality of image pixels for a range of focus plane
positions, or each of a plurality of groups of image pixels for a
range of focus plane positions, and transforming in-focus data into
3D real world coordinates.
Inventors: |
FISKER; Rune; (Virum,
DK) ; OJELUND; Henrik; (Lyngby, DK) ; KJ R;
Rasmus; (Kobenhavn K, DK) ; van der POEL; Mike;
(Rodovre, DK) ; QAZI; Arish A.; (Toronto, CA)
; HOLLENBECK; Karl-Josef; (Copenhagen O, DK) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
3SHAPE A/S |
Copenhagen K |
|
DK |
|
|
Assignee: |
3SHAPE A/S
Copenhagen K
DK
|
Family ID: |
1000005466467 |
Appl. No.: |
17/206581 |
Filed: |
March 19, 2021 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
16774843 |
Jan 28, 2020 |
|
|
|
17206581 |
|
|
|
|
16433369 |
Jun 6, 2019 |
10595010 |
|
|
16774843 |
|
|
|
|
16217943 |
Dec 12, 2018 |
10326982 |
|
|
16433369 |
|
|
|
|
15974105 |
May 8, 2018 |
10349041 |
|
|
16217943 |
|
|
|
|
14502230 |
Sep 30, 2014 |
10097815 |
|
|
15974105 |
|
|
|
|
13376427 |
Dec 6, 2011 |
8878905 |
|
|
PCT/DK2010/050148 |
Jun 17, 2010 |
|
|
|
14502230 |
|
|
|
|
61231118 |
Aug 4, 2009 |
|
|
|
61187744 |
Jun 17, 2009 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
A61B 5/0068 20130101;
H04N 13/296 20180501; G01B 11/2513 20130101; G01B 11/2518 20130101;
A61B 5/4547 20130101; A61B 5/1075 20130101; G01B 2210/58 20130101;
A61B 5/1076 20130101; A61B 5/1077 20130101 |
International
Class: |
H04N 13/296 20060101
H04N013/296; A61B 5/00 20060101 A61B005/00; G01B 11/25 20060101
G01B011/25; A61B 5/107 20060101 A61B005/107 |
Claims
1. (canceled)
2. An intraoral scanner for determining the 3D geometry and color
of at least a part of the surface of an object in an oral cavity,
the intraoral scanner comprising: at least one camera accommodating
an array of sensor elements; a pattern generator configured to
generate, using a light source, a probe light with a plurality of
configurations in the form of a time-varying illumination pattern;
an optical system for transmitting the probe light towards the
object along an optical path thereby illuminating at least a part
of the object with the time-varying illumination pattern, and for
transmitting at least a part of the light returned from the object
to the at least one camera to form a plurality of 2D images,
wherein the 3D geometry is determined based on the plurality of 2D
images and the time-varying illumination pattern; a tip configured
to be inserted into the oral cavity; and a hardware processor
configured to: selectively switch a color of the probe light,
thereby illuminating the object with different colors at different
times; record different images by the at least one camera at the
different times, thereby recording images of the object with the
different colors; and combine the different colors from the
different images, thereby obtaining the color of the surface of the
object, wherein the intraoral scanner is wireless, and wherein the
at least one camera is a high-speed camera.
3. The intraoral scanner according to claim 2, wherein the pattern
generator includes at least one translucent and/or transparent
pattern element.
4. The intraoral scanner according to claim 2, wherein the pattern
generator is based on a projector such as a liquid crystal display
(LCD) projector or a digital light processing (DLP) projector.
5. The intraoral scanner according to claim 2, wherein the
different colors are provided by at least three monochromatic light
sources of the light source.
6. The intraoral scanner according to claim 5, wherein the at least
three monochromatic light sources are light-emitting diodes (LEDs)
or a segmented LED.
7. The intraoral scanner according to claim 5, wherein the hardware
processor is configured to selectively switch the color of the
probe light by switching on and off the at least three
monochromatic light sources.
8. The intraoral scanner according to claim 2, wherein the optical
system is substantially achromatic.
9. The intraoral scanner according to claim 2, wherein the probe
light is directed towards the object in a direction substantially
parallel with the longitudinal axis of the tip, and wherein the
probe light is further reflected towards the object by a single
reflective mirror located in the tip.
10. The intraoral scanner according to claim 2, wherein the
hardware processor is further configured to locally process raw 3D
data into data for the 3D geometry, such that the data for the 3D
geometry is wirelessly transmitted instead of the raw 3D data, and
wherein the raw 3D data is based on the plurality of 2D images.
11. The intraoral scanner according to claim 2, wherein the
hardware processor is further configured to locally process raw 3D
data into data for the 3D geometry, such that the data for the 3D
geometry is wirelessly transmitted at a reduced data rate in
comparison with transmission of raw 3D data, and wherein the raw 3D
data is based on the plurality of 2D images.
12. The intraoral scanner according to claim 10, wherein the data
for the 3D geometry is wirelessly transmitted to a workstation.
13. The intraoral scanner according to claim 10, wherein the
hardware processor processes the raw 3D data into data for the 3D
geometry in real time.
14. The intraoral scanner according to claim 2, wherein a temporal
correlation measure is computed with sensor signals, in the at
least one camera, recorded at different times while the plurality
of configurations in the form of a time-varying illumination
pattern are changed.
15. The intraoral scanner according to claim 14, wherein the
temporal correlation measure is computed for each individual
sensing element.
16. The intraoral scanner according to claim 14, wherein the
temporal correlation measure is based on a reference signal that is
based on knowledge or on calibration of the plurality of
configurations of the time-varying illumination pattern.
17. The intraoral scanner according to claim 2, wherein the
high-speed camera records images at a frame rate of at least 500
frames per second or at a frame rate of at least 2000 frames per
second.
18. The intraoral scanner according to claim 2, wherein the
intraoral scanner comprises at least two light sources, each of the
light sources configured to operate with the pattern generator.
19. The intraoral scanner according to claim 18, wherein the at
least two light sources are configured to operate with different
wavelengths.
20. The intraoral scanner according to claim 19, wherein the
intraoral oral scanner is configured such that the different
wavelengths can be manually selected, whereby the 3D geometry is
dependent on which light source is being manually selected.
21. The intraoral scanner according to claim 2, wherein the
intraoral scanner further comprises polarization optics.
22. A method for determining the 3D geometry and color of at least
a part of the surface of an object in an oral cavity, said method
comprising the steps of: using a light source and a pattern
generator to generate a probe light with a plurality of
configurations in the form of a time-varying illumination pattern;
transmitting the probe light, through a tip of intraoral scanner
inserted into the oral cavity, towards the object along an optical
path, thereby illuminating at least a part of the object with the
time-varying illumination pattern; transmitting at least a part of
the light returned from the object to at least one camera
accommodating an array of sensor elements to form a plurality of 2D
images, wherein the 3D geometry is determined based on the
plurality of 2D images and the time-varying illumination pattern;
selectively switching a color of the probe light, thereby
illuminating the object with different colors at different times;
recording different images by the at least one camera at the
different times, thereby recording images of the object with the
different colors; combining the different colors from the different
images to obtain the color of the surface of the object; and
wirelessly transmitting the data for the 3D geometry.
23. The method according to claim 22, wherein the plurality of 2D
images is recorded using a high-speed camera.
24. The method according to claim 22, wherein the method further
comprises a step of locally processing, on the intraoral scanner,
raw 3D data into data for the 3D geometry, such that the data for
the 3D geometry is wirelessly transmitted instead of the raw 3D
data, and wherein the raw 3D data is based on the plurality of 2D
images.
25. The method according to claim 22, wherein the method further
comprises a step of locally processing, on the intraoral scanner,
raw 3D data into data for the 3D geometry, such that the data for
the 3D geometry is wirelessly transmitted at a reduced data rate in
comparison with transmission of raw 3D data, and wherein the raw 3D
data is based on the plurality of 2D images.
26. The method according to claim 24, wherein the step of
wirelessly transmitting the data for the 3D geometry is to a
workstation.
27. The method according to claim 24, wherein the step of locally
processing is in real time.
28. The method according to claim 22, where the method further
comprises a step of computing a temporal correlation measure with
sensor signals, in the at least one camera, recorded at different
times while the plurality of configurations in the form of a
time-varying illumination pattern are changed.
29. The method according to claim 28, wherein the temporal
correlation measure is computed for each individual sensing
element.
30. The method according to claim 28, wherein the temporal
correlation measure is based on a reference signal that is based on
knowledge or on calibration of the plurality of configurations of
the time-varying illumination pattern.
31. The method according to claim 22, wherein method further
comprises a step of manually selecting a wavelength for which the
intraoral scanner obtains the 3D geometry.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] The present application is a continuation of U.S. Ser. No.
16/774,843, filed on Jan. 28, 2020, which is a continuation of U.S.
Ser. No. 16/433,369, filed on Jun. 6, 2019, which is a continuation
of U.S. Ser. No. 16/217,943, filed on Dec. 12, 2018, which is a
continuation of U.S. Ser. No. 15/974,105, filed on May 8, 2018,
which is a continuation of U.S. Ser. No. 14/502,230, filed on Sep.
30, 2014, now U.S. Pat. No. 10,097,815 B2, which in turn is a
continuation of U.S. Ser. No. 13/376,427, filed on Dec. 6, 2011,
now U.S. Pat. No. 8,878,905 B2, which is a national stage
application of PCT/DK2010/050148, filed on Jun. 17, 2010, and which
claims the benefit of U.S. 61/187,744, filed on Jun. 17, 2009, and
U.S. 61/231,118, filed on Aug. 4, 2009, the contents of all of
which are incorporated herein by reference in their entirety.
FIELD OF THE INVENTION
[0002] The present disclosure relates to an apparatus and a method
for optical 3D scanning of surfaces. The principle of the apparatus
and method according to the disclosure may be applied in various
contexts. One specific embodiment of the invention is particularly
suited for intraoral scanning, i.e. direct scanning of teeth and
surrounding soft-tissue in the oral cavity. Other dental related
embodiments of the invention are suited for scanning dental
impressions, gypsum models, wax bites, dental prosthetics and
abutments. Another embodiment of the invention is suited for
scanning of the interior and exterior part of a human ear or ear
channel impressions. The disclosure may find use within scanning of
the 3D structure of skin in dermatological or
cosmetic/cosmetological applications, scanning of jewelry or wax
models of whole jewelry or part of jewelry, scanning of industrial
parts and even time resolved 3D scanning, such as time resolved 3D
scanning of moving industrial parts.
BACKGROUND
[0003] The disclosure relates to three dimensional (3D) scanning of
the surface geometry of objects. Scanning an object surface in 3
dimensions is a well known field of study and the methods for
scanning can be divided into contact and non-contact methods. An
example of contact measurements methods are Coordinate Measurement
Machines (CMM), which measures by letting a tactile probe trace the
surface. The advantages include great precision, but the process is
slow and a CMM is large and expensive. Non-contact measurement
methods include x-ray and optical probes.
[0004] Confocal microscopy is an optical imaging technique used to
increase micrograph contrast and/or to reconstruct
three-dimensional images by using a spatial pinhole to eliminate
out-of-focus light or flare in specimens that are thicker than the
focal plane.
[0005] A confocal microscope uses point illumination and a pinhole
in an optically conjugate plane in front of the detector to
eliminate out-of-focus information. Only the light within the focal
plane can be detected. As only one point is illuminated at a time
in confocal microscopy, 2D imaging requires raster scanning and 3D
imaging requires raster scanning in a range of focus planes.
[0006] In WO 00/08415 the principle of confocal microscopy is
applied by illuminating the surface with a plurality of illuminated
spots. By varying the focal plane in-focus spot-specific positions
of the surface can be determined. However, determination of the
surface structure is limited to the parts of the surface that are
illuminated by a spot.
[0007] WO 2003/060587 relates to optically sectioning of a specimen
in microscopy wherein the specimen is illuminated with an
illumination pattern. Focus positions of the image plane are
determined by characterizing an oscillatory component of the
pattern. However, the focal plane can only be adjusted by moving
the specimen and the optical system relative to each other, i.e.
closer to or further away from each other. Thus, controlled
variation of the focal plane requires a controlled spatial relation
between the specimen and the optical system, which is fulfilled in
a microscope. However, such a controlled spatial relation is not
applicable to e.g. a hand held scanner.
[0008] US2007/0109559 A1 describes a focus scanner where distances
are found from the focus lens positions at which maximum reflective
intensity of light beams incident on the object being scanned is
observed. In contrast to the disclosure disclosed here, this prior
art exploits no pre-determined measure of the illumination pattern
and exploits no contrast detection, and therefore, the
signal-to-noise ratio is sub-optimal.
[0009] In WO 2008/125605, means for generating a time-variant
pattern composed of alternating split images are described. This
document describes a scanning method to obtain an optical section
of a scan object by means of two different illumination profiles,
e.g. two patterns of opposite phases. These two images are used to
extract the optical section, and the method is limited to
acquisition of images from only two different illumination
profiles. Furthermore, the method relies on a predetermined
calibration that determines the phase offset between the two
illumination profiles.
SUMMARY
[0010] Thus, an object of the disclosure is to provide a scanner
which may be integrated in a manageable housing, such as a handheld
housing. Further objects of the disclosure are: discriminate
out-of-focus information and provide a fast scanning time.
[0011] This is achieved by a method and a scanner for obtaining
and/or measuring the 3D geometry of at least a part of the surface
of an object, said scanner comprising: [0012] at least one camera
accommodating an array of sensor elements, [0013] means for
generating a probe light incorporating a spatial pattern, [0014]
means for transmitting the probe light towards the object thereby
illuminating at least a part of the object with said pattern in one
or more configurations, [0015] means for transmitting at least a
part of the light returned from the object to the camera, [0016]
means for varying the position of the focus plane of the pattern on
the object while maintaining a fixed spatial relation of the
scanner and the object, [0017] means for obtaining at least one
image from said array of sensor elements, [0018] means for
evaluating a correlation measure at each focus plane position
between at least one image pixel and a weight function, where the
weight function is determined based on information of the
configuration of the spatial pattern; [0019] data processing means
for: [0020] a) determining by analysis of the correlation measure
the in-focus position(s) of: [0021] each of a plurality of image
pixels for a range of focus plane positions, or [0022] each of a
plurality of groups of image pixels for a range of focus plane
positions, and [0023] b) transforming in-focus data into 3D real
world coordinates.
[0024] The method and apparatus described in this disclosure is for
providing a 3D surface registration of objects using light as a
non-contact probing agent. The light is provided in the form of an
illumination pattern to provide a light oscillation on the object.
The variation/oscillation in the pattern may be spatial, e.g. a
static checkerboard pattern, and/or it may be time varying, for
example by moving a pattern across the object being scanned. The
disclosure provides for a variation of the focus plane of the
pattern over a range of focus plane positions while maintaining a
fixed spatial relation of the scanner and the object. It does not
mean that the scan must be provided with a fixed spatial relation
of the scanner and the object, but merely that the focus plane can
be varied (scanned) with a fixed spatial relation of the scanner
and the object. This provides for a hand held scanner solution
based on the present disclosure.
[0025] In some embodiments the signals from the array of sensor
elements are light intensity.
[0026] One embodiment of the invention comprises a first optical
system, such as an arrangement of lenses, for transmitting the
probe light towards the object and a second optical system for
imaging light returned from the object to the camera. In the
preferred embodiment of the invention only one optical system
images the pattern onto the object and images the object, or at
least a part of the object, onto the camera, preferably along the
same optical axis, however along opposite optical paths.
[0027] In the preferred embodiment of the invention an optical
system provides an imaging of the pattern onto the object being
probed and from the object being probed to the camera. Preferably,
the focus plane is adjusted in such a way that the image of the
pattern on the probed object is shifted along the optical axis,
preferably in equal steps from one end of the scanning region to
the other. The probe light incorporating the pattern provides a
pattern of light and darkness on the object. Specifically, when the
pattern is varied in time for a fixed focus plane then the in-focus
regions on the object will display an oscillating pattern of light
and darkness. The out-of-focus regions will display smaller or no
contrast in the light oscillations.
[0028] Generally we consider the case where the light incident on
the object is reflected diffusively and/or specularly from the
object's surface. But it is understood that the scanning apparatus
and method are not limited to this situation. They are also
applicable to e.g. the situation where the incident light
penetrates the surface and is reflected and/or scattered and/or
gives rise to fluorescence and/or phosphorescence in the object.
Inner surfaces in a sufficiently translucent object may also be
illuminated by the illumination pattern and be imaged onto the
camera. In this case a volumetric scanning is possible. Some
planktic organisms are examples of such objects.
[0029] When a time varying pattern is applied a single sub-scan can
be obtained by collecting a number of 2D images at different
positions of the focus plane and at different instances of the
pattern. As the focus plane coincides with the scan surface at a
single pixel position, the pattern will be projected onto the
surface point in-focus and with high contrast, thereby giving rise
to a large variation, or amplitude, of the pixel value over time.
For each pixel it is thus possible to identify individual settings
of the focusing plane for which each pixel will be in focus. By
using knowledge of the optical system used, it is possible to
transform the contrast information vs. position of the focus plane
into 3D surface information, on an individual pixel basis.
[0030] Thus, in one embodiment of the disclosure the focus position
is calculated by determining the light oscillation amplitude for
each of a plurality of sensor elements for a range of focus
planes.
[0031] For a static pattern a single sub-scan can be obtained by
collecting a number of 2D images at different positions of the
focus plane. As the focus plane coincides with the scan surface,
the pattern will be projected onto the surface point in-focus and
with high contrast. The high contrast gives rise to a large spatial
variation of the static pattern on the surface of the object,
thereby providing a large variation, or amplitude, of the pixel
values over a group of adjacent pixels. For each group of pixels it
is thus possible to identify individual settings of the focusing
plane for which each group of pixels will be in focus. By using
knowledge of the optical system used, it is possible to transform
the contrast information vs. position of the focus plane into 3D
surface information, on an individual pixel group basis.
[0032] Thus, in one embodiment of the invention the focus position
is calculated by determining the light oscillation amplitude for
each of a plurality of groups of the sensor elements for a range of
focus planes.
[0033] The 2D to 3D conversion of the image data can be performed
in a number of ways known in the art. I.e. the 3D surface structure
of the probed object can be determined by finding the plane
corresponding to the maximum light oscillation amplitude for each
sensor element, or for each group of sensor elements, in the
camera's sensor array when recording the light amplitude for a
range of different focus planes. Preferably, the focus plane is
adjusted in equal steps from one end of the scanning region to the
other. Preferably the focus plane can be moved in a range large
enough to at least coincide with the surface of the object being
scanned.
[0034] The present disclosure distinguishes itself from WO
2008/125605, because in the embodiments of the present invention
that use a time-variant pattern, input images are not limited to
two illumination profiles and can be obtained from any illumination
profile of the pattern. This is because the orientation of the
reference image does not rely entirely on a predetermined
calibration, but rather on the specific time of the input image
acquisition.
[0035] Thus WO 2008/125605 applies specifically exactly two
patterns, which are realized physically by a chrome-on-glass mask
as illuminated from either side, the reverse side being reflective.
WO 2008/125605 thus has the advantage of using no moving parts, but
the disadvantage of a comparatively poorer signal-to-noise ratio.
In the present disclosure there is the possibility of using any
number of pattern configurations, which makes computation of the
light oscillation amplitude or the correlation measure more
precise.
Definitions
[0036] Pattern: A light signal comprising an embedded spatial
structure in the lateral plane. May also be termed "illumination
pattern".
[0037] Time varying pattern: A pattern that varies in time, i.e.
the embedded spatial structure varies in time. May also be termed
"time varying illumination pattern". In the following also termed
"fringes".
[0038] Static pattern: A pattern that does not vary in time, e.g. a
static checkerboard pattern or a static line pattern.
[0039] Pattern configuration: The state of the pattern. Knowledge
of the pattern configuration at a certain time amounts to knowing
the spatial structure of the illumination at that time. For a
periodic pattern the pattern configuration will include information
of the pattern phase. If a surface element of the object being
scanned is imaged onto the camera then knowledge of the pattern
configuration amounts to knowledge of what part of the pattern is
illuminating the surface element.
[0040] Focus plane: A surface where light rays emitted from the
pattern converge to form an image on the object being scanned. The
focus plane does not need to be flat. It may be a curved
surface.
[0041] Optical system: An arrangement of optical components, e.g.
lenses, that transmit, collimate and/or images light, e.g.
transmitting probe light towards the object, imaging the pattern on
and/or in the object, and imaging the object, or at least a part of
the object, on the camera.
[0042] Optical axis: An axis defined by the propagation of a light
beam. An optical axis is preferably a straight line. In the
preferred embodiment of the invention the optical axis is defined
by the configuration of a plurality of optical components, e.g. the
configuration of lenses in the optical system. There may be more
than one optical axis, if for example one optical system transmits
probe light to the object and another optical system images the
object on the camera. But preferably the optical axis is defined by
the propagation of the light in the optical system transmitting the
pattern onto the object and imaging the object onto the camera. The
optical axis will often coincide with the longitudinal axis of the
scanner.
[0043] Optical path: The path defined by the propagation of the
light from the light source to the camera. Thus, a part of the
optical path preferably coincides with the optical axis. Whereas
the optical axis is preferably a straight line, the optical path
may be a non-straight line, for example when the light is
reflected, scattered, bent, divided and/or the like provided e.g.
by means of beam splitters, mirrors, optical fibers and the
like.
[0044] Telecentric system: An optical system that provides imaging
in such a way that the chief rays are parallel to the optical axis
of said optical system. In a telecentric system out-of-focus points
have substantially same magnification as in-focus points. This may
provide an advantage in the data processing. A perfectly
telecentric optical system is difficult to achieve, however an
optical system which is substantially telecentric or near
telecentric may be provided by careful optical design. Thus, when
referring to a telecentric optical system it is to be understood
that it may be only near telecentric.
[0045] Scan length: A lateral dimension of the field of view. If
the probe tip (i.e. scan head) comprises folding optics to direct
the probe light in a direction different such as perpendicular to
the optical axis then the scan length is the lateral dimension
parallel to the optical axis.
[0046] Scan object: The object to be scanned and on which surface
the scanner provides information. "The scan object" may just be
termed "the object".
[0047] Camera: Imaging sensor comprising a plurality of sensors
that respond to light input onto the imaging sensor. The sensors
are preferably ordered in a 2D array in rows and columns.
[0048] Input signal: Light input signal or sensor input signal from
the sensors in the camera. This can be integrated intensity of
light incident on the sensor during the exposure time or
integration of the sensor. In general, it translates to a pixel
value within an image. May also be termed "sensor signal".
[0049] Reference signal: A signal derived from the pattern. A
reference signal may also be denoted a weight function or weight
vector or reference vector.
[0050] Correlation measure: A measure of the degree of correlation
between a reference and input signal. Preferably the correlation
measure is defined such that if the reference and input signal are
linearly related to each other then the correlation measure obtains
a larger magnitude than if they are not.
[0051] In some cases the correlation measure is a light oscillation
amplitude.
[0052] Image: An image can be viewed as a 2D array of values (when
obtained with a digital camera) or in optics, an image indicates
that there exists a relation between an imaged surface and an image
surface where light rays emerging from one point on said imaged
surface substantially converge on one point on said image
surface.
[0053] Intensity: In optics, intensity is a measure of light power
per unit area. In image recording with a camera comprising a
plurality of individual sensing elements, intensity may be used to
term the recorded light signal on the individual sensing elements.
In this case intensity reflects a time integration of light power
per unit area on the sensing element over the exposure time
involved in the image recording.
Mathematical Notation
[0054] A A correlation measure between the weight function and the
recorded light signal. This can be a light oscillation amplitude.
[0055] I Light input signal or sensor input signal. This can be
integrated intensity of light incident on the sensor during the
exposure time or integration of the sensor. In general, it
translates to a pixel value within an image. [0056] f Reference
signal. May also be called weight value. [0057] n The number of
measurements with a camera sensor and/or several camera sensors
that are used to compute a correlation measure. [0058] H Image
height in number of pixels [0059] W Image width in number of
pixels
[0060] Symbols are also explained as needed in the text.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0061] The scanner preferably comprises at least one beam splitter
located in the optical path. For example, an image of the object
may be formed in the camera by means of a beam splitter. Exemplary
uses of beam splitters are illustrated in the figures.
[0062] In a preferred embodiment of the invention light is
transmitted in an optical system comprising a lens system. This
lens system may transmit the pattern towards the object and images
light reflected from the object to the camera.
[0063] In a telecentric optical system, out-of-focus points have
the same magnification as in-focus points. Telecentric projection
can therefore significantly ease the data mapping of acquired 2D
images to 3D images. Thus, in a preferred embodiment of the
invention the optical system is substantially telecentric in the
space of the probed object. The optical system may also be
telecentric in the space of the pattern and camera.
Varying Focus
[0064] A pivotal point of the disclosure is the variation, i.e.
scanning, of the focal plane without moving the scanner in relation
to the object being scanned. Preferably the focal plane may be
varied, such as continuously varied in a periodic fashion, while
the pattern generation means, the camera, the optical system and
the object being scanned is fixed in relation to each other.
Further, the 3D surface acquisition time should be small enough to
reduce the impact of relative movement between probe and teeth,
e.g. reduce effect of shaking. In the preferred embodiment of the
invention the focus plane is varied by means of at least one focus
element. Preferably the focus plane is periodically varied with a
predefined frequency. Said frequency may be at least 1 Hz, such as
at least 2 Hz, 3, 4, 5, 6, 7, 8, 9 or at least 10 Hz, such as at
least 20, 40, 60, 80 or at least 100 Hz.
[0065] Preferably the focus element is part of the optical system.
I.e. the focus element may be a lens in a lens system. A preferred
embodiment comprises means, such as a translation stage, for
adjusting and controlling the position of the focus element. In
that way the focus plane may be varied, for example by translating
the focus element back and forth along the optical axis.
[0066] If a focus element is translated back and forth with a
frequency of several Hz this may lead to instability of the
scanner. A preferred embodiment of the invention thus comprises
means for reducing and/or eliminating the vibration and/or shaking
from the focus element adjustment system, thereby increasing the
stability of the scanner. This may at least partly be provided by
means for fixing and/or maintaining the centre of mass of the focus
element adjustment system, such as a counter-weight to
substantially counter-balance movement of the focus element; for
example, by translating a counter-weight opposite to the movement
of the focus element. Ease of operation may be achieved if the
counter-weight and the focus element are connected and driven by
the same translation means. This may however, only substantially
reduce the vibration to the first order. If a counter-weight
balanced device is rotated around the counter-weight balanced axis,
there may be issues relating to the torque created by the
counter-weights. A further embodiment of the invention thus
comprises means for reducing and/or eliminating the first order,
second order, third order and/or higher order vibration and/or
shaking from the focus element adjustment system, thereby
increasing the stability of the scanner.
[0067] In another embodiment of the invention more than one optical
element is moved to shift the focal plane. In that embodiment it is
desirable that these elements are moved together and that the
elements are physically adjacent.
[0068] In the preferred embodiment of the invention the optical
system is telecentric, or near telecentric, for all focus plane
positions. Thus, even though one or more lenses in the optical
system may be shifted back and forth to change the focus plane
position, the telecentricity of the optical system is
maintained.
[0069] The preferred embodiment of the invention comprises focus
gearing. Focus gearing is the correlation between movement of the
lens and movement of the focus plane position. E.g. a focus gearing
of 2 means that a translation of the focus element of 1 mm
corresponds to a translation of the focus plane position of 2 mm.
Focus gearing can be provided by a suitable design of the optical
system. The advantage of focus gearing is that a small movement of
the focus element may correspond to a large variation of the focus
plane position. In specific embodiments of the invention the focus
gearing is between 0.1 and 100, such as between 0.1 and 1, such as
between 1 and 10, such as between 2 and 8, such as between 3 and 6,
such as least 10, such as at least 20.
[0070] In another embodiment of the invention the focus element is
a liquid lens. A liquid lens can control the focus plane without
use of any moving parts.
Camera
[0071] The camera may be a standard digital camera accommodating a
standard CCD or CMOS chip with one A/D converter per line of sensor
elements (pixels). However, to increase the frame rate the scanner
according to the disclosure may comprise a high-speed camera
accommodating multiple A/D converters per line of pixels, e.g. at
least 2, 4, 8 or 16 A/D converters per line of pixels.
Pattern
[0072] Another central element of the disclosure is the probe light
with an embedded pattern that is projected on to the object being
scanned. The pattern may be static or time varying. The time
varying pattern may provide a variation of light and darkness on
and/or in the object. Specifically, when the pattern is varied in
time for a fixed focus plane then the in-focus regions on the
object will display an oscillating pattern of light and darkness.
The out-of-focus regions will display smaller or no contrast in the
light oscillations. The static pattern may provide a spatial
variation of light and darkness on and/or in the object.
Specifically, the in-focus regions will display an oscillating
pattern of light and darkness in space. The out-of-focus regions
will display smaller or no contrast in the spatial light
oscillations.
[0073] Light may be provided from an external light source, however
preferably the scanner comprises at least one light source and
pattern generation means to produce the pattern. It is advantageous
in terms of signal-to-noise ratio to design a light source such
that the intensity in the non-masked parts of the pattern is as
close to uniform in space as possible. In another embodiment the
light source and the pattern generation means is integrated in a
single component, such as a segmented LED. A segmented LED may
provide a static pattern and/or it may provide a time varying
pattern in itself by turning on and off the different segments in
sequence. In one embodiment of the invention the time varying
pattern is periodically varying in time. In another embodiment of
the invention the static pattern is periodically varying in
space.
[0074] Light from the light source (external or internal) may be
transmitted through the pattern generation means thereby generating
the pattern. For example the pattern generation means comprises at
least one translucent and/or transparent pattern element. For
generating a time varying pattern a wheel, with an opaque mask can
be used. E.g. the mask comprises a plurality of radial spokes,
preferably arranged in a symmetrical order. The scanner may also
comprise means for rotating and/or translating the pattern element.
For generating a static pattern a glass plate with an opaque mask
can be used. E.g. the mask comprises a line pattern or checkerboard
pattern. In general said mask preferably possesses rotational
and/or translational periodicity. The pattern element is located in
the optical path. Thus, light from the light source may be
transmitted through the pattern element, e.g. transmitted
transversely through the pattern element. The time varying pattern
can then be generated by rotating and/or translating the pattern
element. A pattern element generating a static pattern does not
need to be moved during a scan.
Correlation
[0075] One object of the disclosure is to provide short scan time
and real time processing, e.g. to provide live feedback to a
scanner operator to make a fast scan of an entire tooth arch.
However, real time high resolution 3D scanning creates an enormous
amount of data. Therefore data processing should be provided in the
scanner housing, i.e. close to the optical components, to reduce
data transfer rate to e.g. a cart, workstation or display. In order
to speed up data processing time and in order to extract in-focus
information with an optimal signal-to-noise ratio various
correlation techniques may be embedded/implemented. This may for
example be implemented in the camera electronics to discriminate
out-of-focus information. The pattern is applied to provide
illumination with an embedded spatial structure on the object being
scanned. Determining in-focus information relates to calculating a
correlation measure of this spatially structured light signal
(which we term input signal) with the variation of the pattern
itself (which we term reference signal). In general the magnitude
of the correlation measure is high if the input signal coincides
with the reference signal. If the input signal displays little or
no variation then the magnitude of the correlation measure is low.
If the input signal displays a large spatial variation but this
variation is different than the variation in the reference signal
then the magnitude of the correlation measure is also low. In a
further embodiment of the invention the scanner and/or the scanner
head may be wireless, thereby simplifying handling and operation of
the scanner and increasing accessibility under difficult scanning
situations, e.g. intra-oral or in the ear scanning. However,
wireless operation may further increase the need for local data
processing to avoid wireless transmission of raw 3D data.
[0076] The reference signal is provided by the pattern generating
means and may be periodic. The variation in the input signal may be
periodic and it may be confined to one or a few periods. The
reference signal may be determined independently of the input
signal. Specifically in the case of a periodic variation, the phase
between the oscillating input and reference signal may be known
independently of the input signal. In the case of a periodic
variation the correlation is typically related to the amplitude of
the variation. If the phase between the oscillating input and
reference signals is not known it is necessary to determine both
cosine and sinusoidal part of the input signal before the input
signal's amplitude of variation can be determined. This is not
necessary when the phase is known.
[0077] One way to define the correlation measure mathematically
with a discrete set of measurements is as a dot product computed
from a signal vector, I=(I.sub.1, . . . , I.sub.n), with n>1
elements representing sensor signals and a reference vector,
f=(f.sub.1, . . . , f.sub.n), of same length as said signal vector
of reference weights. The correlation measure A is then given
by
A = f I = i = 1 n f i I i ##EQU00001##
[0078] The indices on the elements in the signal vector represent
sensor signals that are recorded at different times and/or at
different sensors. In the case of a continuous measurement the
above expression is easily generalized to involve integration in
place of the summation. In that case the integration parameter is
time and/or one or more spatial coordinates.
[0079] A preferred embodiment is to remove the DC part of the
correlation signal or correlation measure, i.e., when the reference
vector elements sums to zero (.SIGMA..sub.1=1.sup.n f.sub.i=0). The
focus position can be found as an extremum of the correlation
measure computed over all focus element positions. We note that in
this case the correlation measure is proportional to the sample
Pearson correlation coefficient between two variables. If the DC
part is not removed, there may exist a trend in DC signal over all
focus element positions, and this trend can be dominating
numerically. In this situation, the focus position may still be
found by analysis of the correlation measure and/or one or more of
its derivatives, preferably after trend removal.
[0080] Preferably, the global extremum should be found. However,
artifacts such as dirt on the optical system can result in false
global maxima. Therefore, it can be advisable to look for local
extrema in some cases. If the object being scanned is sufficiently
translucent it may be possible to identify interior surfaces or
surface parts that are otherwise occluded. In such cases there may
be several local extrema that corresponds to surfaces and it may be
advantageous to process several or all extrema.
[0081] The correlation measure can typically be computed based on
input signals that are available as digital images, i.e., images
with a finite number of discrete pixels. Therefore conveniently,
the calculations for obtaining correlation measures can be
performed for image pixels or groups thereof. Correlation measures
can then be visualized in as pseudo-images.
[0082] The correlation measure applied in this disclosure is
inspired by the principle of a lock-in amplifier, in which the
input signal is multiplied by the reference signal and integrated
over a specified time. In this disclosure, a reference signal is
provided by the pattern.
Temporal Correlation
[0083] Temporal correlation involves a time-varying pattern. The
light signal in the individual light sensing elements in the camera
is recorded several times while the pattern configuration is
varied. The correlation measure is thus at least computed with
sensor signals recorded at different times.
[0084] A principle to estimate light oscillation amplitude in a
periodically varying light signal is taught in WO 98/45745 where
the amplitude is calculated by first estimating a cosine and a
sinusoidal part of the light intensity oscillation. However, from a
statistical point of view this is not optimal because two
parameters are estimated to be able to calculate the amplitude.
[0085] In this embodiment of the invention independent knowledge of
the pattern configuration at each light signal recording allows for
calculating the correlation measure at each light sensing
element.
[0086] In some embodiments of the invention the scanner comprises
means for obtaining knowledge of the pattern configuration. To
provide such knowledge the scanner preferably further comprises
means for registering and/or monitoring the time varying
pattern.
[0087] Each individual light sensing element, i.e. sensor element,
in the camera sees a variation in the light signal corresponding to
the variation of the light illuminating the object.
[0088] One embodiment of the invention obtains the time variation
of the pattern by translating and/or rotating the pattern element.
In this case the pattern configuration may be obtained by means of
a position encoder on the pattern element combined with prior
knowledge of the pattern geometry that gives rise to a pattern
variation across individual sensing elements. Knowledge of the
pattern configuration thus arises as a combination of knowledge of
the pattern geometry that results in a variation across different
sensing elements and pattern registration and/or monitoring during
the 3D scan. In case of a rotating wheel as the pattern element the
angular position of the wheel may then be obtained by an encoder,
e.g. mounted on the rim.
[0089] One embodiment of the invention involves a pattern that
possesses translational and/or rotational periodicity. In this
embodiment there is a well-defined pattern oscillation period if
the pattern is substantially translated and/or rotated at a
constant speed.
[0090] One embodiment of the invention comprises means for sampling
each of a plurality of the sensor elements a plurality of times
during one pattern oscillation period, preferably sampled an
integer number of times, such as sampling 2, 3, 4, 5, 6, 7 or 8
times during each pattern oscillation period, thereby determining
the light variation during a period.
[0091] The temporal correlation measure between the light variation
and the pattern can be obtained by recording several images on the
camera during one oscillation period (or at least one oscillation
period). The number of images recorded during one oscillation
period is denoted n. The registration of the pattern position for
each individual image combined with the independently known pattern
variation over all sensing element (i.e. obtaining knowledge of the
pattern configuration) and the recorded images allows for an
efficient extraction of the correlation measure in each individual
sensing element in the camera. For a light sensing element with
label j, the n recorded light signals of that element are denoted
I.sub.1,j, . . . , I.sub.n,j. The correlation measure of that
element, A.sub.j, may be expressed as
A j = i = 1 n f i , j I i , j ##EQU00002##
Here the reference signal or weight function f is obtained from the
knowledge of the pattern configuration. f has two indices i,j. The
variation of f with the first index is derived from the knowledge
of the pattern position during each image recording. The variation
of f with the second index is derived from the knowledge of the
pattern geometry which may be determined prior to the 3D
scanning.
[0092] Preferably, but not necessarily, the reference signal f
averages to zero over time, i.e. for all j we have
i = 1 n f i , j = 0 ##EQU00003##
to suppress the DC part of the light variation or correlation
measure. The focus position corresponding to the pattern being in
focus on the object for a single sensor element in the camera will
be given by an extremum value of the correlation measure of that
sensor element when the focus position is varied over a range of
values. The focus position may be varied in equal steps from one
end of the scanning region to the other.
[0093] To obtain a sharp image of an object by means of a camera
the object must be in focus and the optics of the camera and the
object must be in a fixed spatial relationship during the exposure
time of the image sensor of the camera. Applied to the present
disclosure this should imply that the pattern and the focus should
be varied in discrete steps to be able to fix the pattern and the
focus for each image sampled in the camera, i.e. fixed during the
exposure time of the sensor array. However, to increase the
sensitivity of the image data the exposure time of the sensor array
should be as high as the sensor frame rate permits. Thus, in the
preferred embodiment of the invention images are recorded (sampled)
in the camera while the pattern is continuously varying (e.g. by
continuously rotating a pattern wheel) and the focus plane is
continuously moved. This implies that the individual images will be
slightly blurred since they are the result of a time-integration of
the image while the pattern is varying and the focus plane is
moved. This is something that one could expect to lead to
deterioration of the data quality, but in practice the advantage of
concurrent variation of the pattern and the focus plane is bigger
than the drawback.
[0094] In another embodiment of the invention images are recorded
(sampled) in the camera while the pattern is fixed and the focus
plane is continuously moved, i.e. no movement of the pattern. This
could be the case when the light source is a segmented light
source, such as a segment LED that flashes in an appropriate
fashion. In this embodiment the knowledge of the pattern is
obtained by a combination of prior knowledge of the geometry of the
individual segments on the segmented LED give rise to a variation
across light sensing elements and the applied current to different
segments of the LED at each recording.
[0095] In yet another embodiment of the invention images are
recorded (sampled) in the camera while the pattern is continuously
varying and the focus plane is fixed.
[0096] In yet another embodiment of the invention images are
recorded (sampled) in the camera while the pattern and the focus
plane are fixed.
[0097] The temporal correlation principle may be applied in general
within image analysis. Thus, a further embodiment of the invention
relates to a method for calculating the amplitude of a light
intensity oscillation in at least one (photoelectric) light
sensitive element, said light intensity oscillation generated by a
periodically varying illumination pattern and said amplitude
calculated in at least one pattern oscillation period, said method
comprising the steps of: [0098] providing the following a
predetermined number of sampling times during a pattern oscillation
period: [0099] sampling the light sensitive element thereby
providing the signal of said light sensitive element, and [0100]
providing an angular position and/or a phase of the periodically
varying illumination pattern for said sampling, and [0101]
calculating said amplitude(s) by integrating the products of a
predetermined periodic function and the signal of the corresponding
light sensitive element over said predetermined number of sampling
times, wherein said periodic function is a function of the angular
position and/or the phase of the periodically varying illumination
pattern.
[0102] This may also be expressed as
A = i f ( p i ) I i ##EQU00004##
where A is the calculated amplitude or correlation measure, i is
the index for each sampling, f is the periodic function, p.sub.i is
the angular position/phase of the illumination pattern for sampling
i and I.sub.i is the signal of the light sensitive element for
sampling i. Preferably the periodic function averages to zero over
a pattern oscillation period, i.e.
.SIGMA..sub.if(p.sub.i)=0.
[0103] To generalize the principle to a plurality of light
sensitive elements, for example in a sensor array, the angular
position/phase of the illumination pattern for a specific light
sensitive element may consist of an angular position/phase
associated with the illumination pattern plus a constant offset
associated with the specific light sensitive element. Thereby the
correlation measure or amplitude of the light oscillation in light
sensitive element j may be expressed as
A j = i f ( .theta. j + p i ) I i , j , ##EQU00005##
where .theta..sub.j is the constant offset for light sensitive
element j.
[0104] A periodically varying illumination pattern may be generated
by a rotating wheel with an opaque mask comprising a plurality of
radial spokes arranged in a symmetrical order. The angular position
of the wheel will thereby correspond to the angular position of the
pattern and this angular position may obtained by an encoder
mounted on the rim of the wheel. The pattern variation across
different sensor elements for different position of the pattern may
be determined prior to the 3D scanning in a calibration routine. A
combination of knowledge of this pattern variation and the pattern
position constitutes knowledge of the pattern configuration. A
period of this pattern may for example be the time between two
spokes and the amplitude of a single or a plurality of light
sensitive elements of this period may be calculated by sampling
e.g. four times in this period.
[0105] A periodically varying illumination pattern may generated by
a Ronchi ruling moving orthogonal to the lines and the position is
measured by an encoder. This position corresponds to the angular
position of the generated pattern. Alternatively, a checkerboard
pattern could be used.
[0106] A periodically varying illumination pattern may generated by
a one-dimensional array of LEDs that can be controlled line
wise.
[0107] A varying illumination pattern may generated by a LCD or DLP
based projector.
Optical Correlation
[0108] The abovementioned correlation principle (temporal
correlation) requires some sort of registering of the time varying
pattern, e.g. knowledge of the pattern configuration at each light
level recording in the camera. However, a correlation principle
without this registering may be provided in another embodiment of
the invention. This principle is termed "optical correlation".
[0109] In this embodiment of the invention an image of the pattern
itself and an image of at least a part of the object being scanned
with the pattern projected onto it is combined on the camera. I.e.
the image on the camera is a superposition of the pattern itself
and the object being probed with the pattern projected onto it. A
different way of expressing this is that the image on the camera
substantially is a multiplication of an image of the pattern
projected onto the object with the pattern itself.
[0110] This may be provided in the following way. In a further
embodiment of the invention the pattern generation means comprises
a transparent pattern element with an opaque mask. The probe light
is transmitted through the pattern element, preferably transmitted
transversely through the pattern element. The light returned from
the object being scanned is retransmitted the opposite way through
said pattern element and imaged onto the camera. This is preferably
done in a way where the image of the pattern illuminating the
object and the image of the pattern itself are coinciding when both
are imaged onto the camera. One particular example of a pattern is
a rotating wheel with an opaque mask comprising a plurality of
radial spokes arranged in a symmetrical order such that the pattern
possesses rotational periodicity. In this embodiment there is a
well-defined pattern oscillation period if the pattern is
substantially rotated at a constant speed. We define the
oscillation period as 2.pi./.omega..
[0111] We note that in the described embodiment of the invention
the illumination pattern is a pattern of light and darkness. A
light sensing element in the camera with a signal proportional to
the integrated light intensity during the camera integration time
.delta.t with label j, I.sub.j is given by
I j = K .intg. t t + .delta. t T j ( t ' ) S j ( t ' ) dt '
##EQU00006##
Here K is the proportionality constant of the sensor signal, t is
the start of the camera integration time, T.sub.j is the
time-varying transmission of the part of the rotating pattern
element imaged onto the j'th light sensing element, and S.sub.j is
the time-varying light intensity of light returned from the scanned
object and imaged onto the j'th light sensing element. In the
described embodiment T.sub.j is the the step function substantially
defined by T.sub.j(t)=0 for sin(.omega.t+.PHI..sub.j)>0 and
T.sub.j(t)=1 elsewhere. .PHI..sub.j is a phase dependent on the
position of the j'th imaging sensor.
[0112] The signal on the light sensing element is a correlation
measure of the pattern and the light returned from the object being
scanned. The time-varying transmission takes the role of the
reference signal and the time-varying light intensity of light
returned from the scanned object takes the role of the input
signal. The advantage of this embodiment of the invention is that a
normal CCD or CMOS camera with intensity sensing elements may be
used to record the correlation measure directly since this appears
as an intensity on the sensing elements. Another way of expressing
this is that the computation of the correlation measure takes place
in the analog, optical domain instead of in an electronic domain
such as an FPGA or a PC.
[0113] The focus position corresponding to the pattern being in
focus on the object being scanned for a single sensor element in
the camera will then be given by the maximum value of the
correlation measure recorded with that sensor element when the
focus position is varied over a range of values. The focus position
may be varied in equal steps from one end of the scanning region to
the other. One embodiment of the invention comprises means for
recording and/or integrating and/or monitoring and/or storing each
of a plurality of the sensor elements over a range of focus plane
positions.
[0114] Preferably, the global maximum should be found. However,
artifacts such as dirt on the optical system can result in false
global maxima. Therefore, it can be advisable to look for local
maxima in some cases.
[0115] Since the reference signal does not average to zero the
correlation measure has a DC component. Since the DC part is not
removed, there may exist a trend in DC signal over all focus
element positions, and this trend can be dominating numerically. In
this situation, the focus position may still be found by analysis
of the correlation measure and/or one or more of its
derivatives.
[0116] In a further embodiment of the invention the camera
integration time is an integer number M of the pattern oscillation
period, i.e. .quadrature.t=2.quadrature.M/.quadrature.. One
advantage of this embodiment is that the magnitude of the
correlation measure can be measured with a better signal-to-noise
ratio in the presence of noise than if the camera integration time
is not an integer number of the pattern oscillation period.
[0117] In another further embodiment of the invention the camera
integration time is much longer than pattern oscillation period,
i.e. .quadrature.t>>2.quadrature.M/.quadrature.. Many times
the pattern oscillation time would here mean e.g. camera
integration time at least 10 times the oscillation time or more
preferably such as at least 100 or 1000 times the oscillation time.
One advantage of this embodiment is that there is no need for
synchronization of camera integration time and pattern oscillation
time since for very long camera integration times compared to the
pattern oscillation time the recorded correlation measure is
substantially independent of accurate synchronization.
[0118] Equivalent to the temporal correlation principle the optical
correlation principle may be applied in general within image
analysis. Thus, a further embodiment of the invention relates to a
method for calculating the amplitude of a light intensity
oscillation in at least one (photoelectric) light sensitive
element, said light intensity oscillation generated by a
superposition of a varying illumination pattern with itself, and
said amplitude calculated by time integrating the signal from said
at least one light sensitive element over a plurality of pattern
oscillation periods.
Spatial Correlation
[0119] The above mentioned correlation principles (temporal
correlation and optical correlation) require the pattern to be
varying in time. If the optical system and camera provides a
lateral resolution which is at least two times what is needed for
the scan of the object then it is possible to scan with a static
pattern, i.e. a pattern which is not changing in time. This
principle is termed "spatial correlation". The correlation measure
is thus at least computed with sensor signals recorded at different
sensor sites.
[0120] The lateral resolution of an optical system is to be
understood as the ability of optical elements in the optical
system, e.g. a lens system, to image spatial frequencies on the
object being scanned up to a certain point. Modulation transfer
curves of the optical system are typically used to describe imaging
of spatial frequencies in an optical system. One could e.g. define
the resolution of the optical system as the spatial frequency on
the object being scanned where the modulation transfer curve has
decreased to e.g. 50%. The resolution of the camera is a combined
effect of the spacing of the individual camera sensor elements and
the resolution of the optical system.
[0121] In the spatial correlation the correlation measure refers to
a correlation between input signal and reference signal occurring
in space rather than in time. Thus, in one embodiment of the
invention the resolution of the measured 3D geometry is equal to
the resolution of the camera. However, for the spatial correlation
the resolution of the measured 3D geometry is lower than the
resolution of the camera, such as at least 2 times lower, such as
at least 3 times lower, such as at least 4 times lower, such as
least 5 times lower, such as at least 10 times lower. The sensor
element array is preferably divided into groups of sensor elements,
preferably rectangular groups, such as square groups of sensor
elements, preferably adjacent sensor elements. The resolution of
the scan, i.e. the measured 3D geometry, will then be determined by
the size of these groups of sensor elements. The oscillation in the
light signal is provided within these groups of sensor elements,
and the amplitude of the light oscillation may then be obtained by
analyzing the groups of sensor elements. The division of the sensor
element array into groups is preferably provided in the data
processing stage, i.e. the division is not a physical division
thereby possibly requiring a specially adapted sensor array. Thus,
the division into groups is "virtual" even though the single pixel
in a group is an actual physical pixel.
[0122] In one embodiment of the invention the pattern posseses
translational periodicity along at least one spatial coordinate. In
a further embodiment of the invention the spatially periodic
pattern is aligned with the rows and/or the columns of the array of
sensor elements. For example in the case of a static line pattern
the rows or columns of the pixels in the camera may be parallel
with the lines of the pattern. Or in the case of a static
checkerboard pattern the row and columns of the checkerboard may be
aligned with the rows and columns, respectively, of the pixels in
the camera. By aligning is meant that the image of the pattern onto
the camera is aligned with the "pattern" of the sensor element in
the sensor array of the camera. Thus, a certain physical location
and orientation of the pattern generation means and the camera
requires a certain configuration of the optical components of the
scanner for the pattern to be aligned with sensor array of the
camera.
[0123] In a further embodiment of the invention at least one
spatial period of the pattern corresponds to a group of sensor
elements. In a further embodiment of the invention all groups of
sensor elements contain the same number of elements and have the
same shape. E.g. when the period of a checkerboard pattern
corresponds to a square group of e.g. 2.times.2, 3.times.3,
4.times.4, 5.times.5, 6.times.6, 7.times.7, 8.times.8, 9.times.9,
10.times.10 or more pixels on the camera.
[0124] In yet another embodiment one or more edges of the pattern
is aligned with and/or coincide with one or more edges of the array
of sensor elements. For example a checkerboard pattern may be
aligned with the camera pixels in such a way that the edges of the
image of the checkerboard pattern onto the camera coincide with the
edges of the pixels.
[0125] In spatial correlation independent knowledge of the pattern
configuration allows for calculating the correlation measure at
each group of light sensing. For a spatially periodic illumination
this correlation measure can be computed without having to estimate
the cosine and sinusoidal part of the light intensity oscillation.
The knowledge of the pattern configuration may be obtained prior to
the 3D scanning.
[0126] In a further embodiment of the invention the correlation
measure, A.sub.j, within a group of sensor elements with label j is
determined by means of the following formula:
A j = i = 1 n f i , j I i , j ##EQU00007##
Where n is the number of sensor elements in a group of sensors,
f.sub.j=(f.sub.a,j, . . . f.sub.n,j) is the reference signal vector
obtained from knowledge of the pattern configuration, and
I.sub.j=(I.sub.1,j, . . . I.sub.n,j) is input signal vector. For
the case of sensors grouped in square regions with N sensors as
square length then n=N.sup.2.
[0127] Preferably, but not necessarily, the elements of the
reference signal vector averages to zero over space, i.e. for all j
we have
i = 1 n f i , j = 0 ##EQU00008##
to suppress the DC part of the correlation measure. The focus
position corresponding to the pattern being in focus on the object
for a single group of sensor elements in the camera will be given
by an extremum value of the correlation measure of that sensor
element group when the focus position is varied over a range of
values. The focus position may be varied in equal steps from one
end of the scanning region to the other.
[0128] In the case of a static checkerboard pattern with edges
aligned with the camera pixels and with the pixel groups having an
even number of pixels such as 2.times.2, 4.times.4, 6.times.6,
8.times.8, 10.times.10, a natural choice of the reference vector f
would be for its elements to assume the value 1 for the pixels that
image a bright square of the checkerboard and -1 for the pixels
that image a dark square of the checkerboard.
[0129] Equivalent to the other correlation principles the spatial
correlation principle may be applied in general within image
analysis. In particular in a situation where the resolution of the
camera is higher than what is necessary in the final image. Thus, a
further embodiment of the invention relates to a method for
calculating the amplitude(s) of a light intensity oscillation in at
least one group of light sensitive elements, said light intensity
oscillation generated by a spatially varying static illumination
pattern, said method comprising the steps of: [0130] providing the
signal from each light sensitive element in said group of light
sensitive elements, and [0131] calculating said amplitude(s) by
integrating the products of a predetermined function and the signal
from the corresponding light sensitive element over said group of
light sensitive elements, wherein said predetermined function is a
function reflecting the illumination pattern.
[0132] To generalize the principle to a plurality of light
sensitive elements, for example in a sensor array, the correlation
measure or amplitude of the light oscillation in group j may be
expressed as
A j = i = 1 n f ( i , j ) I i , j , ##EQU00009##
where n is the number of sensor elements in group j, I.sub.i,j is
the signal from the i'th sensor element in group j and f(i,j) is a
predetermined function reflecting the pattern.
[0133] Compared to temporal correlation, spatial correlation has
the advantage that no moving pattern is required. This implies that
knowledge of the pattern configuration may be obtained prior to the
3D scanning. Conversely, the advantage of temporal correlation is
its higher resolution, as no pixel grouping is required.
[0134] All correlation principles, when embodied with an image
sensor that allows very high frame rates, enable 3D scanning of
objects in motion with little motion blur. It also becomes possible
to trace moving objects over time ("4D scanning"), with useful
applications for example in machine vision and dynamic deformation
measurement. Very high frame rates in this context are at least
500, but preferably at least 2000 frames per second.
Transforming Correlation Measure Extrema to 3D World
Coordinates
[0135] Relating identified focus position(s) for camera sensor or
camera sensor groups to 3D world coordinates may be done by ray
tracing through the optical system. Before such ray tracing can be
performed the parameters of the optical system need to be known.
One embodiment of the invention comprises a calibration step to
obtain such knowledge. A further embodiment of the invention
comprises a calibration step in which images of an object of known
geometry are recorded for a plurality of focus positions. Such an
object may be a planar checkerboard pattern. Then, the scanner can
be calibrated by generating simulated ray traced images of the
calibration object and then adjusting optical system parameters as
to minimize the difference between the simulated and recorded
images.
[0136] In a further embodiment of the invention the calibration
step requires recording of images for a plurality of focus
positions for several different calibration objects and/or several
different orientations and/or positions of one calibration
object.
[0137] With knowledge of the parameters of the optical system, one
can employ backward ray tracing technique to estimate the 2D->3D
mapping. This requires that the scanner's optical system be known,
preferably through calibration. The following steps can be
performed: [0138] 1. From each pixel of the image (at the image
sensor), trace a certain number of rays, starting from the image
sensor and through the optical system (backward ray tracing).
[0139] 2. From the rays that emit, calculate the focus point, the
point where all these rays substantially intersect. This point
represents the 3D coordinate of where a 2D pixel will be in focus,
i.e., in yield the global maximum of light oscillation amplitude.
[0140] 3. Generate a look up table for all the pixels with their
corresponding 3D coordinates. The above steps are repeated for a
number of different focus lens positions covering the scanner's
operation range.
Specular Reflections
[0141] High spatial contrast of the in-focus pattern image on the
object is often necessary to obtain a good signal to noise ratio of
the correlation measure on the camera. This in turn may be
necessary to obtain a good estimation of the focus position
corresponding to an extremum in the correlation measure. This
sufficient signal to noise ratio for successful scanning is often
easily achieved in objects with a diffuse surface and negligible
light penetration. For some objects, however, it is difficult to
achieve high spatial contrast.
[0142] A difficult kind of object, for instance, is an object
displaying multiple scattering of the incident light with a light
diffusion length large compared to the smallest feature size of the
spatial pattern imaged onto the object. A human tooth is an example
of such an object. The human ear and ear canal are other examples.
In case of intra oral scanning, the scanning should preferably be
provided without spraying and/or drying the teeth to reduce the
specular reflections and light penetration. Improved spatial
contrast can be achieved by preferential imaging of the specular
surface reflection from the object on the camera. Thus, one
embodiment of the invention comprises means for
preferential/selectively imaging of specular reflected light and/or
diffusively reflected light. This may be provided if the scanner
further comprises means for polarizing the probe light, for example
by means of at least one polarizing beam splitter. A polarizing
beam splitter may for instance be provided for forming an image of
the object in the camera. This may be utilized to extinguish
specular reflections, because if the incident light is linearly
polarized a specular reflection from the object has the property
that it preserves its polarization state
[0143] The scanner according to the invention may further comprise
means for changing the polarization state of the probe light and/or
the light reflected from the object. This can be provided by means
of a retardation plate, preferably located in the optical path. In
one embodiment of the invention the retardation plate is a quarter
wave retardation plate. A linearly polarized light wave is
transformed into a circularly polarized light wave upon passage of
a quarter wave plate with an orientation of 45 degrees of its fast
axis to the linear polarization direction. This may be utilized to
enhance specular reflections because a specular reflection from the
object has the property that it flips the helicity of a circularly
polarized light wave, whereas light that is reflected by one or
more scattering events becomes depolarized.
The Field of View (Scanning Length)
[0144] In one embodiment of the invention the probe light is
transmitted towards the object in a direction substantially
parallel with the optical axis. However, for the scan head to be
entered into a small space such as the oral cavity of a patient it
is necessary that the tip of the scan head is sufficiently small.
At the same time the light out of the scan head need to leave the
scan head in a direction different from the optical axis. Thus, a
further embodiment of the invention comprises means for directing
the probe light and/or imaging an object in a direction different
from the optical axis. This may be provided by means of at least
one folding element, preferably located along the optical axis, for
directing the probe light and/or imaging an object in a direction
different from the optical axis. The folding element could be a
light reflecting element such as a mirror or a prism. In one
embodiment of the invention a 45 degree mirror is used as folding
optics to direct the light path onto the object. Thereby the probe
light is guided in a direction perpendicular to the optical axis.
In this embodiment the height of the scan tip is at least as large
as the scan length and preferably of approximately equal size.
[0145] One embodiment of the invention comprises at least two light
sources, such as light sources with different wavelengths and/or
different polarization. Preferably also control means for
controlling said at least two light sources. Preferably this
embodiment comprises means for combining and/or merging light from
said at least two light sources. Preferably also means for
separating light from said at least two light sources. If waveguide
light sources are used they may be merged by waveguides. However,
one or more diffusers may also be provided to merge light
sources.
[0146] Separation and/or merging may be provided by at least one
optical device which is partially light transmitting and partially
light reflecting, said optical device preferably located along the
optical axis, an optical device such as a coated mirror or coated
plate. One embodiment comprises at least two of said optical
devices, said optical devices preferably displaced along the
optical axis. Preferably at least one of said optical devices
transmits light at certain wavelengths and/or polarizations and
reflects light at other wavelengths and/or polarizations.
[0147] One exemplary embodiment of the invention comprises at least
a first and a second light source, said light sources having
different wavelength and/or polarization, and wherein [0148] a
first optical device reflects light from said first light source in
a direction different from the optical axis and transmits light
from said second light source, and [0149] a second optical device
reflects light from said second light source in a direction
different from the optical axis. Preferably said first and second
optical devices reflect the probe light in parallel directions,
preferably in a direction perpendicular to the optical axis,
thereby imaging different parts of the object surface. Said
different parts of the object surface may be at least partially
overlapping.
[0150] Thus, for example light from a first and a second light
source emitting light of different wavelengths (and/or
polarizations) is merged together using a suitably coated plate
that transmits the light from the first light source and reflects
the light from the second light source. At the scan tip along the
optical axis a first optical device (e.g. a suitably coated plate,
dichroic filter) reflects the light from the first light source
onto the object and transmits the light from the second light
source to a second optical device (e.g. a mirror) at the end of the
scan tip, i.e. further down the optical axis. During scanning the
focus position is moved such that the light from the first light
source is used to project an image of the pattern to a position
below the first optical device while second light source is
switched off. The 3D surface of the object in the region below the
first optical device is recorded. Then the first light source is
switched off and the second light source is switched on and the
focus position is moved such that the light from the second light
source is used to project an image of the pattern to a position
below the second optical device. The 3D surface of the object in
the region below the second optical device is recorded. The region
covered with the light from the two light sources respectively may
partially overlap.
[0151] In another embodiment of the invention the probe light is
directed in a direction different from the optical axis by means of
a curved fold mirror. This embodiment may comprise one or more
optical elements, such as lenses, with surfaces that may be
aspherical to provide corrected optical imaging.
[0152] A further embodiment of the invention comprises of at least
one translation stage for translating mirror(s) along the optical
axis. This allows for a scan tip with a smaller height than the
scan length. A large scan length can be achieved by combining
several scans with the mirror(s) in different positions along the
optical axis.
[0153] In another embodiment of the invention the probe light is
directed in a direction different from the optical axis by means of
at least one grating that provides anamorphic magnification so that
the image of the pattern on the object being scanned is stretched.
The grating may be blazed. In this embodiment the light source
needs to be monochromatic or semi-monochromatic.
[0154] The abovementioned embodiments suitable for increasing the
scan length may comprise control means for providing a coordination
of the light sources and the focus element.
Color Scanning
[0155] One embodiment of the invention is only registering the
surface topology (geometry) of the object being scanned. However,
another embodiment of the invention is being adapted to obtain the
color of the surface being scanned, i.e. capable of registering the
color of the individual surface elements of the object being
scanned together with the surface topology of the object being
scanned. To obtain color information the light source needs to be
white or to comprise at least three monochromatic light sources
with colors distributed across the visible part of the
electromagnetic spectrum.
[0156] To provide color information the array of sensor elements
may be a color image sensor. The image sensor may accommodate a
Bayer color filter scheme. However, other color image sensor types
may be provided, such as a Foveon type color image sensor, wherein
the image sensor provides color registration in each sensor
element.
[0157] One embodiment of the invention comprises means selecting
one color of the probe light at a time, i.e. selectively switching
between different colors of the probe light, thereby illuminating
the object with different colors. If a white light source is used
then some kind of color filtering must be provided. Preferably
comprising a plurality of color filters, such as red, green and
blue color filters, and means for inserting said color filters
singly in front of the white light source, thereby selecting a
color of the probe light.
[0158] In one embodiment of the invention color filters are
integrated in the pattern generation means, i.e. the pattern
generation means comprises color filters, such as translucent
and/or transparent parts that are substantially monochromatically
colored. For example a pattern element such as a rotating wheel
with an opaque mask and where the translucent/transparent parts are
color filters. For example one third of the wheel is red, one third
is green and one third is blue.
[0159] Probe light of different colors may also be provided by at
least three monochromatic light sources, such as lasers or LED's,
said light sources having wavelengths distributed across the
visible part of the wavelength spectrum. This will in general also
require means for merging said light sources. For example suitable
coated plates. In the case of waveguide light sources, the merging
may be provided by a waveguide element.
[0160] To handle the different colors of the probe light the
optical system is preferably substantially achromatic.
[0161] One embodiment of the invention comprises means for
switching between at least two colors, preferably three colors,
such as red, green and blue, of the probe light for a focal plane
position. I.e. for a single focal plane position it is possible to
switch between different colors of the probe light. For example by
switching on and off different monochromatic light sources (having
one only light source turned on at a time) or by applying different
color filters. Furthermore, the amplitude of the light signal of
each of a plurality of the sensor elements may be determined for
each color for each focal plane positions. I.e. for each focus
position the color of the probe light may be switched. The embedded
time varying pattern provides a single color oscillating light
signal and the amplitude of the signal in each sensor element may
be determined for that color. Switching to the next color the
amplitude may be determined again. When the amplitude has been
determined for all colors the focus position is changed and the
process is repeated. The color of the surface being scanned may
then be obtained by combining and/or weighing the color information
from a plurality of the sensor elements. E.g. the color expressed
as e.g. an RGB color coordinate of each surface element can be
reconstructed by appropriate weighting of the amplitude signal for
each color corresponding to the maximum amplitude. This technique
may also be applied when a static pattern is provided where the
color of at least a part of the pattern is varying in time.
[0162] To decrease the amount of data to be processed the color
resolution of the imaging may be chosen to be less than the spatial
resolution. The color information is then provided by data
interpolation. Thus, in one embodiment of the invention the
amplitude of the light signal of each of a plurality of the sensor
elements is determined for each color for selected full color focal
plane positions, and the amplitude of the light signal of each of a
plurality of the sensor elements is determined for one color for
each focal plane position. Then the color of the surface being
scanned may be obtained by interpolating the color information from
full color focal plane positions. Thus, for example the amplitude
is registered for all colors at an interval of N focus positions;
while one color is selected for determination of the amplitude at
all focus positions. N is a number which could be e.g. 3, 5, or 10.
This results in a color resolution which is less than the
resolution of the surface topology. This technique may also be
applied when a static pattern is provided where the color of at
least a part of the pattern is varying in time.
[0163] Another embodiment of the invention does not register full
color information and employs only two light sources with different
colors. An example of this is a dental scanner that uses red and
blue light to distinguish hard (tooth) tissue from soft (gum)
tissue.
Impression Scanning
[0164] One embodiment of the invention is adapted to impression
scanning, such as scanning of dental impressions and/or ear canal
impressions.
Small Cavity Scanner
[0165] Specific applications of the scanner according to the
invention relates to scanning of cavities, in particular body
cavities. Scanning in cavities may relate to scanning of objects in
the cavity, such as scanning of teeth in a mouth. However, scanning
of e.g. the ear relate to scanning of the inner surface of the
cavity itself. In general scanning of a cavity, especially a small
cavity, requires some kind of probe for the scanner. Thus, in one
embodiment of the invention the point of emission of probe light
and the point of accumulation of reflected light is located on a
probe, said probe being adapted to be entered into a cavity, such
as a body cavity.
[0166] In another embodiment of the invention the probe is adapted
to scan at least a part of the surface of a cavity, such as an ear
canal. The ability to scan at least a part of the external part of
the ear and/or the ear canal and make a virtual or real model of
the ear is essential in the design of modern custom-fitted hearing
aid (e.g. ear shell or mold). Today, scanning of ears is performed
in a two-step process where a silicone impression of the ear is
taken first and the impression is subsequently scanned using an
external scanner in a second step.
[0167] Thus, one embodiment of the invention comprises [0168] a
housing accommodating the camera, pattern generation means, focus
varying means and data processing means, and [0169] at least one
probe accommodating a first optical system, preferably a
substantially elongated probe.
[0170] Preferably, the point of emission of probe light and the
point of accumulation of light returned from the scanned object is
located on said probe. The optical system in the probe is for
transmitting the probe light from the housing toward the object and
also for transmitting and/or imaging light returned from the object
back towards the housing where the camera is located. Thus, the
optical system in the probe may comprise a system of lenses. In one
embodiment of the invention probe may comprise at least one optical
fibre and/or a fibre bundle for transmitting/transporting/guiding
the probe light and/or the returned light from the object surface.
In this case the optical fibre(s) may act as an optical relay
system that merely transports light (i.e. probe light and returned
light) inside the probe. In one embodiment of the invention the
probe is endoscopic. The probe may be rigid or flexible. Use of
optical fibre(s) in the probe may e.g. provide a flexible probe
with a small diameter.
[0171] In one embodiment of the invention the light is transmitted
to the object and imaged by means of only the optical system in the
probe, the first optical system. However, in a further embodiment
of the invention the housing may further comprise a second optical
system.
[0172] In a further embodiment of the invention the probe is
detachable from the housing. Then preferably a first point of
emission of probe light and a first point of accumulation of
returned light is located on the probe, and a second point of
emission of probe light and a second point of accumulation of
returned light is located on the housing. This may require optical
systems in both the housing and the probe. Thus, a scan may be
obtained with the probe attached to the housing. However, a scan
may also be obtained with the probe detached from the housing, i.e.
the housing may be a standalone scanner in itself. For example the
probe may be adapted to be inserted into and scanning the inside of
a cavity, whereas the housing may be adapted to scanning of
exterior surfaces. The attachment of the probe may include
mechanical and/or electrical transfer between the housing and the
probe. For instance attaching the probe may provide an electrical
signal to the control electronics in the housing that signals the
current configuration of the device.
[0173] In one embodiment of the invention the probe light is
directed toward the object in a direction substantially parallel
with the optical axis and/or the longitudinal axis of the probe. In
a further embodiment the probe comprises a posterior reflective
element, such as a mirror, for directing the probe light in a
direction different from the optical axis, preferably in a
direction perpendicular to the optical axis. Applying to the
abovementioned example with a stand-alone scanner housing with the
probe detached, the probe light may exit the housing in a direction
parallel with the optical axis of the optical system in the housing
(i.e. the second optical system), whereas with the probe attached
the probe light may be directed in a direction different than the
optical axis of the optical system of the probe (i.e. the first
optical system). Thereby the probe is better adapted to scanning a
cavity.
[0174] In some embodiments of this invention, waste heat generated
in the scanner is used to warm the probe such that no or less
condensation occurs on the probe when the probe is inside the body
cavity, e.g. the mouth. Waste heat can, e.g., be generated by the
processing electronics, the light source, and/or the mechanism that
moves the focus element.
[0175] In some embodiments of this invention, the scanner provides
feedback to the user when the registration of subsequent scans to a
larger model of the 3D surface fails. For example, the scanner
could flash the light source.
[0176] Further, the probe may comprise means for rotating/spinning
the reflective element, preferably around an axis substantially
parallel with the optical axis and/or the longitudinal axis of the
probe. Thereby the probe may be adapted to provide a scan
360.degree. around the optical axis and/or the longitudinal axis of
the probe, preferably without rotation of probe and/or scanner.
[0177] In a further embodiment of the invention a plurality of
different probes matches the housing. Thereby different probes
adapted to different environments, surfaces, cavities, etc. may be
attached to the housing to account for different scanning
situations. A specific example of this is when the scanner
comprises a first probe being adapted to scan the interior part of
a human ear and a second probe being adapted to scan the exterior
part of said human ear. Instead of a second probe it may be the
housing itself, i.e. with the probe detached, that is adapted to
scan the exterior part of said human ear. I.e. the housing may be
adapted to perform a 3D surface scan. In other words: the housing
with the probe attached may be adapted to scan the interior part of
a human ear and the housing with the probe detached may be adapted
to scan the exterior part of said human ear. Preferably, means for
merging and/or combining 3D data for the interior and exterior part
of the ear provided, thereby providing a full 3D model of a human
ear.
[0178] For handheld embodiments of this invention, a pistol-like
design is ergonomic because the device rests comfortably inside the
hand of the operator, with most of the mass resting on top of the
hand and/or wrist. In such a design, it is advantageous to be able
to orient the above-mentioned posterior reflective in multiple
positions. For example, it could be possible to rotate a probe with
the posterior reflective element, with or without the step of
detaching it from the main body of the scanning device. Detachable
probes may also be autoclavable, which is a definitely advantage
for scanners applied in humans, e.g., as medical devices. For
embodiments of this invention that realize a physically moving
focus element by means of a motor, it is advantageous to place this
motor inside a grip of the pistol-like shape.
Use of Motion, Gravity, and Magnetic Sensors
[0179] Handheld embodiments of the invention preferably include
motion sensors such as accelerometers and/or gyros. Preferably,
these motion sensors are small like microelectromechanical systems
(MEMS) motion sensors. The motion sensors should preferably measure
all motion in 3D, i.e., both translations and rotations for the
three principal coordinate axes. The benefits are: [0180] A) Motion
sensors can detect vibrations and/or shaking. Scans such affected
can be either discarded or corrected by use of image stabilization
techniques. [0181] B) Motion sensors can help with stitching and/or
registering partial scans to each other. This advantage is relevant
when the field of view of the scanner is smaller than the object to
be scanned. In this situation, the scanner is applied for small
regions of the object (one at a time) that then are combined to
obtain the full scan. In the ideal case, motion sensors can provide
the required relative rigid-motion transformation between partial
scans' local coordinates, because they measure the relative
position of the scanning device in each partial scan. Motion
sensors with limited accuracy can still provide a first guess for a
software-based stitching/registration of partial scans based on,
e.g., the Iterative Closest Point class of algorithms, resulting in
reduced computation time. [0182] C) Motion sensors can be used
(also) as a remote control for the software that accompanies the
invention. Such software, for example, can be used to visualize the
acquired scan. With the scanner device now acting as a remote
control, the user can, for example, rotate and/or pan the view (by
moving the remote control in the same way as the object on the
computer screen should "move"). Especially in clinical application,
such dual use of the handheld scanner is preferable out of hygienic
considerations, because the operator avoids contamination from
alternative, hand-operated input devices (touch screen, mouse,
keyboard, etc).
[0183] Even if it is too inaccurate to sense translational motion,
a 3-axis accelerometer can provide the direction of gravity
relative to the scanning device. Also a magnetometer can provide
directional information relative to the scanning device, in this
case from the earth's magnetic field. Therefore, such devices can
help with stitching/registration and act as a remote control
element.
[0184] The present invention relates to different aspects including
the scanner device described above and in the following, and
corresponding methods, devices, uses and/or product means, each
yielding one or more of the benefits and advantages described in
connection with the first mentioned aspect, and each having one or
more embodiments corresponding to the embodiments described in
connection with the first mentioned aspect and/or disclosed in the
appended claims.
[0185] In particular, disclosed herein is a method for obtaining
and/or measuring the 3D geometry of at least a part of the surface
of an object, said method comprising the steps of: [0186]
generating a probe light incorporating a spatial pattern, [0187]
transmitting the probe light towards the object along the optical
axis of an optical system, thereby illuminating at least a part of
the object with said pattern, [0188] transmitting at least a part
of the light returned from the object to the camera, [0189] varying
the position of the focus plane of the pattern on the object while
maintaining a fixed spatial relation of the scanner and the object,
[0190] obtaining at least one image from said array of sensor
elements, [0191] evaluating a correlation measure at each focus
plane position between at least one image pixel and a weight
function, where the weight function is determined based on
information of the configuration of the spatial pattern; [0192]
determining by analysis of the correlation measure the in-focus
position(s) of: [0193] each of a plurality of image pixels in the
camera for said range of focus plane positions, or [0194] each of a
plurality of groups of image pixels in the camera for said range of
focus planes, and [0195] transforming in-focus data into 3D real
world coordinates.
[0196] Disclosed is also a computer program product comprising
program code means for causing a data processing system to perform
the method, when said program code means are executed on the data
processing system.
[0197] Disclosed is also a computer program product, comprising a
computer-readable medium having stored there on the program code
means.
[0198] Another aspect of the invention relates to a scanner for
obtaining and/or measuring the 3D geometry of at least a part of
the surface of an object, said scanner comprising: [0199] at least
one camera accommodating an array of sensor elements, [0200] means
for generating a probe light, [0201] means for transmitting the
probe light towards the object thereby illuminating at least a part
of the object, [0202] means for transmitting light returned from
the object to the camera, [0203] means for varying the position of
the focus plane on the object, [0204] means for obtaining at least
one image from said array of sensor elements, [0205] means for:
[0206] a) determining the in-focus position(s) of: [0207] each of a
plurality of the sensor elements for a range of focus plane
positions, or [0208] each of a plurality of groups of the sensor
elements for a range of focus plane positions, and [0209] b)
transforming in-focus data into 3D real world coordinates; [0210]
wherein the scanner further comprises counter-weight means for
counter-balancing the means for varying the position of the focus
plane.
[0211] Disclosed is also a method for obtaining and/or measuring
the 3D geometry of at least a part of the surface of an object,
said method comprising the steps of: [0212] accommodating an array
of sensor elements, [0213] generating a probe light, [0214]
transmitting the probe light towards the object thereby
illuminating at least a part of the object, [0215] transmitting
light returned from the object to the camera, [0216] varying the
position of the focus plane on the object, [0217] obtaining at
least one image from said array of sensor elements, [0218]
determining the in-focus position(s) of: [0219] each of a plurality
of the sensor elements for a range of focus plane positions, or
[0220] each of a plurality of groups of the sensor elements for a
range of focus plane positions, and [0221] transforming in-focus
data into 3D real world coordinates; [0222] wherein the method
further comprises counter-balancing the means for varying the
position of the focus plane.
[0223] Another aspect of the invention relates to a handheld 3D
scanner with a grip at an angle of more than 30 degrees from the
scanner's main optical axis, for use in intraoral or in-ear
scanning.
BRIEF DESCRIPTION OF THE DRAWINGS
[0224] FIG. 1: A schematic presentation of a first example
embodiment of the device according to the invention.
[0225] FIG. 2: A schematic presentation of a second example
embodiment of the device according to the invention (optical
correlation).
[0226] FIGS. 3A through 3C: Schematic presentations of example
embodiments of patterns according to the invention.
[0227] FIG. 4: A schematic presentation of a first example
embodiment of a flat scan tip with large scan length, using a
plurality of (dichroic) mirrors and light sources.
[0228] FIG. 5: A schematic presentation of a third example
embodiment of a flat scan tip with a large scan length, using a
curved mirror.
[0229] FIG. 6: A schematic presentation of a fourth example
embodiment of a flat scan tip with large scan length, using a
diffractive grating.
[0230] FIG. 7: A schematic presentation of an example embodiment of
a mass-balanced focus lens scanner.
[0231] FIG. 8: A schematic presentation of an example embodiment of
a device for simultaneous scanning of a surface shape and
color.
[0232] FIG. 9: A schematic presentation of an example embodiment of
a device for scanning the at least a part of the external part of
the human ear and/or a part of the ear canal of a human ear.
[0233] FIGS. 10A and 10B: Schematics showing how a scanner
embodiment can be used to both scan the outer and inner ear,
respectively.
[0234] FIG. 11: Schematic of a scanner probe embodiment used to
scan a narrow body cavity, such as a human ear.
[0235] FIGS. 12A through 12D: Examples of mirror configurations to
be used with a scanner probe.
[0236] FIG. 13: A schematic representation of the reference signal
values/weight values per pixel for a checkerboard pattern in an
idealized optical system.
[0237] FIGS. 14A through 14E: Illustration of the process of
generating a fused reference signal, visualized as images.
[0238] FIG. 15: Top: Example image with projected pattern showing
on a human tooth. Bottom: The correlation measure for the series of
focus lens positions at the group of pixels framed in the top part
of the figure.
[0239] FIG. 16: Example fused correlation measure image of an
intraoral scene.
[0240] FIG. 17: Example of a handheld intraoral scanner with a
pistol-like grip and a removable tip.
[0241] It will be understood that the ray traces and lenses
depicted in the figures are for purpose of illustration only, and
depict optical paths generally in the discussed systems. The ray
traces and lens shapes should not be understood to limit the scope
of the invention in any sense including the magnitude, direction,
or focus of light rays or bundles passing through various optical
components, not withstanding any variations in number, direction,
shape, position or size thereof, except as expressly indicated in
the following detailed description of the exemplary embodiments
illustrated in the drawings.
DETAILED DESCRIPTION OF THE DRAWINGS
[0242] A functional hand held 3D surface scanner should preferably
have the following properties: [0243] 1) Telecentricity in the
space of the object being scanned, [0244] 2) possibility to shift
the focal plane while maintaining telecentricity and magnification
[0245] 3) simple focusing scheme that involves tuning of optical
components only in the handle of the device and not in the probe
tip, and [0246] 4) a total size consistent with a hand held
scanning device.
[0247] The scanner embodiment illustrated in FIG. 1 is a hand-held
scanner with all components inside the housing (head) 100. The
scanner head comprises a tip which can be entered into a cavity, a
light source 110, optics 120 to collect the light from the light
source, pattern generation means 130, a beam splitter 140, an image
sensor and electronics 180, a lens system which transmits and
images the light between the pattern, the object being scanned, and
the image sensor (camera) 180. The light from the light source 110
travels back and forth through the optical system 150. During this
passage the optical system images the pattern 130 onto the object
being scanned 200 and further images the object being scanned onto
the image sensor 181. The lens system includes a focusing element
151 which can be adjusted to shift the focal imaging plane of the
pattern on the probed object 200. One way to embody the focusing
element is to physically move a single lens element back and forth
along the optical axis. The device may include polarization optics
160. The device may include folding optics 170 which directs the
light out of the device in a direction different to the optical
axis of the lens system, e.g. in a direction perpendicular to the
optical axis of the lens system. As a whole, the optical system
provides an imaging of the pattern onto the object being probed and
from the object being probed to the camera. One application of the
device could be for determining the 3D structure of teeth in the
oral cavity. Another application could be for determining the 3D
shape of the ear canal and the external part of the ear.
[0248] The optical axis in FIG. 1 is the axis defined by a straight
line through the light source 110, optics 120 and the lenses in the
optical system 150. This also corresponds to the longitudinal axis
of the scanner illustrated in FIG. 1. The optical path is the path
of the light from the light source 110 to the object 220 and back
to the camera 180. The optical path may change direction, e.g. by
means of beam splitter 140 and folding optics 170.
[0249] The focus element is adjusted in such a way that the image
of the pattern on the scanned object is shifted along the optical
axis, preferably in equal steps from one end of the scanning region
to the other. When the pattern is varied in time in a periodic
fashion for a fixed focus position then the in-focus regions on the
object will display an spatially varying pattern. The out-of-focus
regions will display smaller or no contrast in the light variation.
The 3D surface structure of the probed object is determined by
finding the plane corresponding to an extremum in the correlation
measure for each sensor in the camera's sensor array or each group
of sensor in the camera's sensor array when recording the
correlation measure for a range of different focus positions 300.
Preferably one would move the focus position in equal steps from
one end of the scanning region to the other.
Pattern Generation
[0250] An embodiment of the pattern generation means is shown in
FIG. 3a: A transparent wheel with an opaque mask 133 in the form of
spokes pointing radially from the wheel center. In this embodiment
the pattern is time-varied by rotating the wheel with a motor 131
connected to the wheel with e.g. a drive shaft 132. The position of
the pattern in time may be registered during rotation. This can be
achieved by e.g. using a position encoder on the rim of the pattern
134 or obtaining the shaft position directly from motor 131.
[0251] FIG. 3b illustrates another embodiment of the pattern
generation means: A segmented light source 135, preferably a
segmented LED. In this embodiment the LED surface is imaged onto
the object under investigation. The individual LED segments 136 are
turned on and off in a fashion to provide a known time-varying
pattern on the object. The control electronics 137 of the time
varying pattern is connected to the segmented light source via
electrical wires 138. The pattern is thus integrated into the light
source and a separate light source is not necessary.
[0252] FIG. 3c illustrates a static pattern as applied in a spatial
correlation embodiment of this invention. The checkerboard pattern
shown is preferred because calculations for this regular pattern
are easiest.
Temporal Correlation
[0253] FIG. 1 is also an exemplary illustration of the temporal
correlation wherein an image of the pattern on and/or in the object
is formed on the camera. Each individual light sensing element in
the camera sees a variation in the signal level corresponding to
the variation of the illumination pattern on the object. The
variation is periodic in the exemplary illustration. The light
variation for each individual light sensing element will have a
constant phase offset relative to the pattern position.
[0254] The correlation measure may be obtained by recording n
images on the camera during at least one oscillation period. n is
an integer number greater than one. The registration of the pattern
position for each individual image combined with the phase offset
values for each sensing element and the recorded images allows for
an efficient extraction of the correlation measure in each
individual sensing element in the camera using the following
formula,
A j = i = 1 n f i , j I i , j ##EQU00010##
Here A.sub.j is the estimated correlation measure of sensing
element j, I.sub.1,j, . . . I.sub.n,j are the n recorded signals
from sensing element j, f.sub.1,j, . . . f.sub.n,j are then
reference signal values obtained from the knowledge of the pattern
configuration for each image recording. f has two indices i,j. The
variation of f with the first index is derived from the knowledge
of the pattern position during each image recording. The variation
of f with the second index is derived from the knowledge of the
pattern geometry which may be determined prior to the 3D
scanning.
[0255] The focus position corresponding to the pattern being in
focus on the object for a single sensor in the camera will be given
by an extremum in the recorded correlation measure of that sensor
when the focus position is varied over a range of values,
preferably in equal steps from one end of the scanning region to
the other.
Spatial Correlation
[0256] In an example of the spatial correlation scheme, one image
of the object with projected checkerboard pattern is recorded with
as high resolution as allowed by the image sensor. The scheme in
the spatial correlation in is then to analyze groups of pixels in
the recorded image and extract the correlation measure in the
pattern. An extremum in the obtained correlation measures indicates
the in-focus position. For simplicity, one can use a checkerboard
pattern with a period corresponding to n=N.times.N pixels on the
sensor and then analyze the correlation measure within one period
of the pattern (in the general case the pattern need not be
quadratic N.times.N). In the best case, it will be possible to
align the pattern so that the checkerboard edges coincide with the
pixel edges but the scanning principle does not rely upon this.
FIG. 16 shows this for the case n=4.times.4=16. For a sensor with
W.times.H=1024.times.512 pixels, this would correspond to obtaining
256.times.128 correlation measure points from one image. Extraction
of the correlation measure A.sub.j within an N.times.N group of
pixels with label j is given by
A j = i = 1 n f i , j I i , j ##EQU00011##
where f=(f.sub.1,j, . . . f.sub.n,j) is the reference signal vector
obtained from knowledge of the pattern configuration, and
I.sub.j=(I.sub.1,j, . . . I.sub.n,j) is input signal vector.
[0257] To suppress any DC part in the light we prefer that for all
j that
0 = i = 1 n f i , j ##EQU00012##
[0258] For the situation depicted in FIG. 16 for instance,
f.sub.i,j=-1 for the pixels corresponding to the dark parts of the
pattern, and f.sub.i,j=+1 otherwise. If the pattern edge was not
aligned with the edges of the pixels, or if the optical system was
not perfect (and thus in all practical applications), then
f.sub.i,j would assume values between -1 and +1 for some i. A
detailed description of how to determine the reference function is
given later.
Optical Correlation
[0259] An example of the optical correlation shown in FIG. 2. In
this embodiment an image is formed on the camera 180 which is a
superposition of the pattern 130 with the probed object 200. In
this embodiment the pattern is of a transmissive nature where light
is transmitted through the pattern and the image of the pattern is
projected onto the object and back again. In particular this
involves retransmission of the light through the pattern in the
opposite direction. An image of the pattern onto the camera is then
formed with the aid of a beam splitter 140. The result of this
arrangement is an image being formed on the camera which is a
superposition of the pattern itself and the object being probed. A
different way of expressing this is that the image on the camera is
substantially a multiplication of an image of the pattern projected
onto the object with the pattern itself.
[0260] The variation is periodic in the exemplary illustration. The
correlation measure between the light variation on the object and
the pattern for a given focus distance may be obtained by time
integrating the camera signal over a large number of oscillation
periods so that exact synchronization of pattern oscillation time
and camera integration time is not important. The focus position
corresponding to the pattern being in focus on the object for a
single sensor in the camera will be given by the maximum recorded
signal value of that sensor when the focus position is varied over
a range of values, preferably in equal steps from one end of the
scanning region to the other.
Finding the Predetermined Reference Function
[0261] In the following, the process for computing the reference
signal f is described for a spatial correlation embodiment of this
invention, and depicted in a stylized way in FIGS. 14A-14E.
[0262] The process starts by recording a series of images of the
checkerboard pattern as projected, e.g., on a flat surface,
preferably oriented orthogonally to the optical axis of the
scanner. The images are taken at different positions of the
focusing element, in effect covering the entire travel range of
said focus element. Preferably, the images are taken at equidistant
locations.
[0263] As the focus plane generally is not a geometrical plane,
different regions of the flat surface will be in focus in different
images. Examples of three such images are shown in FIGS. 14A-14C,
where 1700 is an in-focus region. Note that in this stylized
figure, transitions between regions in and out of focus,
respectively, are exaggerated in order to demonstrate the principle
more clearly. Also, in general there will be many more images than
just the three used in this simple example.
[0264] In-focus regions within an image are found as those of
maximum intensity variance (indicating maximum contrast) over the
entire said series of images. The region to compute variance over
need not be the same as the pixel group dimension used in spatial
correlation, but should be large enough to contain the both dark
and light regions of the pattern, and it must be the same for all
images in the series.
[0265] Finally, a "fused image" (FIG. 14D) is generated by
combining all the in-focus regions of the series (FIGS. 14A-14C).
Note that in real applications, the fused image will generally not
be a perfect checkerboard of black and white, but rather include
intermediate gray values as caused by an imperfect optical system
and a checkerboard that is not perfectly aligned with the camera
sensors. An example of part of a real fused image is shown in FIG.
14E.
[0266] The pixel intensities within this image can be interpreted
as a "weight image" with same dimensions as the original image of
the pattern. In other words, the pixel values can be interpreted as
the reference signal and the reference vector/set of weight values
f.sub.j=(f.sub.1,j, . . . f.sub.n,j) for the n pixels in the pixel
group with index j can be found from the pixel values.
[0267] For convenience in the implementation of the calculations,
especially when carried out on an FPGA, the fused image can be
sub-divided into pixel groups. The DC part of the signal can then
be removed by subtracting the within-group intensity mean from each
pixel intensity value. Furthermore, one can then normalize by
dividing by the within-group standard deviation. The thus processed
weight values are an alternative description of the reference
signal.
[0268] Because of the periodic nature of the "fused image" and thus
the "weight image", the latter can be compressed efficiently, thus
minimizing memory requirements in the electronics that can
implement the algorithm described here. For example, the PNG
algorithm can be used for compression.
The "Correlation Image"
[0269] A "correlation" image is generated based on the "fused
image" and the set of images recorded with the camera during a
scan. For spatial correlation based on an N.times.N checkerboard
pattern, recall that within-group correlation measure is
A.sub.j=.SIGMA..sub.i=1.sup.N.times.Nf.sub.i,jI.sub.i,j,
where f.sub.j=(f.sub.1,j, . . . f.sub.n,j) are values from the
fused image, and I.sub.j=(I.sub.1,j, . . . I.sub.n,j) are values
from a recorded image on the camera. The pixel groupings used in
any DC removal and possibly normalization that yielded the fused
image are the same as in the above calculation. For each image
recorded by the scanner during a sweep of the focusing element,
there will thus be an array of (H/N).times.(W/N) values of A. This
array can be visualized as an image.
[0270] FIG. 15 (top section) shows one example correlation measure
image, here of part of a human tooth and its edge. A pixel group of
6.times.6 pixels is marked by a square 1801. For this example pixel
group, the series of correlation measures A over all images within
a sweep of the focusing element is shown in the chart in the bottom
section of FIG. 15 (cross hairs). The x-axis on the chart is the
position of the focusing element, while the y-axis shows the
magnitude of A. Running a simple Gaussian filter over the raw
series results in a smoothed series (solid line). In the figure the
focus element is in the position that gives optimal focus for the
example group of pixels. This fact is both subjectively visible in
the picture, but also determined quantitatively as the maximum of
the series of A. The vertical line 1802 in the bottom section of
FIG. 15 indicates the location of the global extremum and thus the
in-focus position. Note that in this example, the location of the
maxima in the smoothed and the raw series, respectively, are
visually indistinguishable. In principle, however, it is possible
and also advantageous to find the maximum location from the
smoothed series, as that can be between two lens positions and thus
provide higher accuracy.
[0271] The array of values of A can be computed for every image
recorded in a sweep of the focus element. Combining the global
extrema (over all images) of A in all pixel groups in the same
manner the fused image was combined, one can obtain a pseudo-image
of dimension (H/N).times.(W/N). This we call the "fused correlation
image". An example of a fused correlation image of some teeth and
gingiva is shown in FIG. 16. As can be seen, it is useful for
visualization purposes.
Increasing Field of View
[0272] For the scan head to be entered into a small space such as
the oral cavity of a patient it is necessary that the tip of the
scan head is sufficiently small. At the same time the light out of
the scan head need to leave the scan head in a direction different
from the optical axis, e.g. at a direction perpendicular to the
optical axis. In one embodiment of the invention a 45 degree mirror
is used as folding optics 170 direct the light path onto the
object. In this embodiment the height of the scan tip need to be at
least as large as the scan length.
[0273] Another embodiment of the invention is shown in FIG. 4. This
embodiment of the invention allows for a scan tip with a smaller
height (denoted b in the figure) than the scan length (denoted a in
the figure). The light from two sources 110 and 111 emitting light
of different colors/wavelengths is merged together using a suitably
coated plate (e.g. a dichroic filter) 112 that transmit the light
from 110 and reflects the light from 111. At the scan tip a
suitably coated plate (e.g. a dichroic filter) 171 reflects the
light from one source onto the object and transmits the light from
the other source to a mirror at the end of the scan tip 172. During
scanning the focus position is moved such that the light from 110
is used to project an image of the pattern to a position below 171
while 111 is switched off. The 3D surface of the object in the
region below 171 is recorded. Then 110 is switched off and 111 is
switched on and the focus position is moved such that the light
from 111 is used to project an image of the pattern to a position
below 172. The 3D surface of the object in the region below 172 is
recorded. The region covered with the light from 110 and 111
respectively may partially overlap.
[0274] Another embodiment of the invention that allows for a scan
tip with a smaller height (denoted b in the figure) than the scan
length (denoted a in the figure) is shown in FIG. 5. In this
embodiment the fold optics 170 comprises a curved fold mirror 173
that may be supplemented with one or two lens elements 175 and 176
with surfaces that may be aspherical to provide corrected optical
imaging.
[0275] Another embodiment of the invention that allows for a scan
tip with a smaller height (denoted b in the figure) than the scan
length (denoted a in the figure) is shown in FIG. 6. In this
embodiment the fold optics 170 comprises a grating 177 that
provides anamorphic magnification so that the image of the pattern
on the object being scanned is stretched. The grating may be
blazed. The light source 110 needs to be monochromatic or
semi-monochromatic in this embodiment.
Achieving High Spatial Contrast of Pattern Projected onto Difficult
Objects
[0276] High spatial contrast of the in-focus pattern image on the
object is necessary to obtain a high correlation measure signal
based on the camera pictures. This in turn is necessary to obtain a
good estimation of the focus position corresponding to the position
of an extremum of the correlation measure. This necessary condition
for successful scanning is easily achieved in objects with a
diffuse surface and negligible light penetration. For some objects,
however, it is difficult to achieve high spatial constrast, or more
generally variation.
[0277] A difficult kind of object, for instance, is an object
displaying multiple scattering with a light diffusion length large
compared to the smallest feature size of the spatial pattern imaged
onto the object. A human tooth is an example of such an object. The
human ear and ear canal are other examples. Improved spatial
variation in such objects can be achieved by preferential imaging
of the specular surface reflection from the object on the camera.
An embodiment of the invention applies polarization engineering
shown in FIG. 1. In this embodiment the beam splitter 140 is a
polarizing beam splitter that transmits respectively reflects two
orthogonal polarization states, e.g. S- and P-polarization states.
The light transmitted through the lens system 150 is thus of a
specific polarization state. Before leaving the device the
polarization state is changed with a retardation plate 160. A
preferred type of retardation plate is a quarter wave retardation
plate. A linearly polarized light wave is transformed into a
circularly polarized light wave upon passage of a quarter wave
plate with an orientation 45 degrees of its fast axis to the linear
polarization direction. A specular reflection from the object has
the property that it flips the helicity of a circularly polarized
light wave. Upon passage of the quarter wave retardation plate by
the specularly reflected light the polarization state becomes
orthogonal to the state incident on the object. For instance an
S-polarization state propagating in the downstream direction toward
the object will be returned as a P-polarization state. This implies
that the specularly reflected light wave will be directed towards
the image sensor 181 in the beam splitter 140. Light that enters
into the object and is reflected by one or more scattering events
becomes depolarized and one half of this light will be directed
towards the image sensor 181 by the beam splitter 140.
[0278] Another kind of difficult object is an object with a shiny
or metallic-looking surface. This is particularly true for a
polished object or an object with a very smooth surface. A piece of
jewelry is an example of such an object. Even very smooth and shiny
objects, however, do display an amount of diffuse reflection.
Improved spatial contrast in such objects can be achieved by
preferential imaging of the diffuse surface reflection from the
object on the camera. In this embodiment the beam splitter 140 is a
polarizing beam splitter that transmits respectively reflects two
orthogonal polarization states, e.g. S- and P-polarization states.
The light transmitted through the lens system 150 is thus of a
specific polarization state. A diffuse reflection from the object
has the property that it loses its polarization. This implies that
half of the diffusely reflected light wave will be directed towards
the image sensor 181 in the beam splitter 140. Light that enters
into the object and is reflected by specular polarization preserves
its polarization state and thus none of it will be directed towards
the image sensor 181 by the beam splitter 140.
Reducing Shaking Caused by Focus Element
[0279] During scanning the focus position is changed over a range
of values, preferably provided by a focusing element 151 in the
optical system 150. FIG. 7 illustrates an example of how to reduce
shaking caused by the oscillating focus element. The focusing
element is a lens element 152 that is mounted on a translation
stage 153 and translated back and forth along the optical axis of
said optical system with a mechanical mechanism 154 that includes a
motor 155. During scanning the center of mass of the handheld
device is shifted due to the physical movement of the lens element
and holder. This results in an undesirable shaking of the handheld
device during scanning. The situation is aggravated if the scan is
fast, e.g. a scan time of less than one second. In one
implementation of the invention the shifting of the center of mass
is eliminated by moving a counter-weight 156 in a direction
opposite to the lens element in such a way that the center of mass
of the handheld device remains fixed. In the preferred
implementation the focus lens and the counter-weight are
mechanically connected and their opposite movement is driven by the
same motor.
Color Measurement
[0280] An embodiment of a color 3D scanner is shown in FIG. 8.
Three light sources 110, 111, and 113 emit red, green, and blue
light. The light sources are may be LEDs or lasers. The light is
merged together to overlap or essentially overlap. This may be
achieved by means of two appropriately coated plates 112 and 114.
Plate 112 transmits the light from 110 and reflects the light from
111. Plate 114 transmits the light from 110 and 111 and reflects
the light from 113. The color measurement is performed as follows:
For a given focus position the amplitude of the time-varying
pattern projected onto the probed object is determined for each
sensor element in the sensor 181 by one of the above mentioned
methods for each of the light sources individually. In the
preferred embodiment only one light source is switched on at the
time, and the light sources are switched on after turn. In this
embodiment the optical system 150 may be achromatic. After
determining the amplitude for each light source the focus position
is shifted to the next position and the process is repeated. The
color expressed as e.g. an RGB color coordinate of each surface
element can be reconstructed by appropriate weighting of the
amplitude signal for each color corresponding the maximum
amplitude.
[0281] One specific embodiment of the invention only registers the
amplitude for all colors at an interval of P focus positions; while
one color is selected for determination of the amplitude at all
focus positions. P is a number which could be e.g. 3, 5, or 10.
This results in a color resolution which is less than the
resolution of the surface topology. Color of each surface element
of the probed object is determined by interpolation between the
focus positions where full color information is obtained. This is
in analogy to the Bayer color scheme used in many color digital
cameras. In this scheme the color resolution is also less than the
spatial resolution and color information need to be
interpolated.
[0282] A simpler embodiment of the 3D color scanner does not
register full color information and employs only two light sources
with different colors. An example of this is a dental scanner that
uses red and blue light to distinguish hard (tooth) tissue from
soft (gum) tissue.
Ear Scanner Embodiment
[0283] FIGS. 9-12 schematically illustrate an embodiment of a
time-varying structured light illumination-based scanner for direct
scanning of human ears by scanning both the exterior (outer) and
interior (inner) part of a human ear by use of a common scanner
exterior handle and a detachable probe. This embodiment is
advantageous in that it allows for non-intrusive scanning using a
probe designed to be inserted into small cavities, such as a human
ear. This is done in part by positioning the bulky and essential
parts of the scanner, such as the scanner camera, light source,
electronics and focusing optics outside the closely confined part
of the ear canal.
[0284] The ability to scan the outer and inner part of human ears
and make a virtual or real model of the ear is essential in the
design of modern custom-fitted hearing aid (e.g. ear shell or
mold). Today, scanning of ears is performed in a two-step process
where a silicone impression of the ear is taken first and the
impression is subsequently scanned using an external scanner in a
second step. The process of making the impression suffers from
several drawbacks which will shortly be described in the following.
One major drawback comes from frequent poor quality impressions
taken by qualified clinic professionals due to the preparation and
techniques required. Inaccuracies may arise because the impression
material is known to expand during hardening and that deformation
and creation of fractures in the impression are often created when
the impression is removed from the ear. Another drawback is related
to health risks involved with taking the impression due to
irritation and allergic responses, damage to the tympanic membrane
and infections. Finally, the impression process is an uncomfortable
experience for many patients, especially for young children, who
often require impressions taken at regular intervals (e.g. every
four months) to accommodate the changing dimensions of the ear
canal. In short, these drawbacks can be overcome if it is possible
to scan the outer and inner ear in a non-intrusive way and obtain a
registration between the inner and outer ear surfaces.
[0285] The following is not restricted to ear scanning but can be
used to scan any small bodily cavity. FIG. 9 is a schematic of an
embodiment of such a scanner. The scanner consists of two main
parts--a scanner exterior 1001 and a scanner probe 1002. The
scanner exterior may be used without the probe to obtain a larger
field-of-view needed e.g. to scan the exterior part of the ear
1102, or the first part of the ear canal up to the first bend. The
large field-of-view of the scanner exterior is important to obtain
good registration between individual sub-scans and high global
accuracy. By attaching a scanner probe 1002 to the scanner exterior
1001, the combined scanner allows for scanning of small and bent
cavity surfaces, such as the interior part of an ear 1203. In this
way and using the same system, the combined scanner exterior and
probe are able to both scan larger external areas along with
smaller internal areas.
[0286] In FIG. 9 the exterior part of the scanner embodiment 1001
consists of a diverging light source 1003 (laser, LED, Tungsten or
another type) which is collimated using collimation optics 1004.
The collimated light is used to illuminate a transparent object
1005 (e.g. glass) with an opaque pattern, e.g. fringes on it. The
pattern is subsequently imaged onto the object to be scanned using
a suitable optical system. The pattern is observed using a similar
optical system and a camera 1006, where the latter is positioned
outside the cavity. The 3D information is obtained from the 2D
images by observing the light oscillation created by the movement
of the pattern across the scan object as contained in the
individual pixel amplitude.
[0287] To facilitate movement of the pattern, the fringe pattern
1005 is rotating in one embodiment. In another embodiment, the
fringe pattern is positioned on a translating plate that moves in a
plane perpendicular to the optical axis with a certain oscillation
frequency. The light to and from the scan object is projected
through a beam splitter arrangement 1007, which consists of a prism
cube in one embodiment and in another embodiment consists of an
angled plate or membrane. The beam splitter serves to transmit the
source light further down the system, while at the same time guide
the reflected light from the scan object back to the camera, which
is positioned on an axis perpendicular to the axis of the light
source and beam splitter.
[0288] To move the focus plane the scanner exterior includes
focusing optics, which in one embodiment consists of a single
movable lens 1008. The purpose of the focusing optics is to
facilitate movement of the plane of focus for the whole imaging
system in the required scanning range and along the optical axis.
In one embodiment, the focusing optics of the scanner exterior 1101
includes an objective that can focus the light directly, without
any use of additional optics, as shown in FIG. 10A. In another
embodiment, the scanner exterior is supplied with a wide-angle
objective designed with a large field-of-view, e.g. sufficiently
large for scanning the exterior part of a human ear 1102.
[0289] The optical part of the scanner probe consists of an
endoscopic optical relay system 1009 followed by a probe objective
1010, both of which are of sufficiently small diameter to fit into
the canal of a human ear. These optical systems may consist of both
a plurality of optical fibers and lenses and serve to transport and
focus the light from the scanner exterior onto the scan object 1014
(e.g. the interior surface of an ear), as well as to collimate and
transport the reflected light from the scan object back to the
scanner exterior. In one embodiment, the probe objective provides
telecentric projection of the fringe pattern onto the scan object.
Telecentric projection can significantly ease the data mapping of
acquired 2D images to 3D images. In another embodiment, the chief
rays (center ray of each ray bundle) from the probe objective are
diverging (non-telecentric) to provide the camera with an
angle-of-view larger than zero, as shown in FIG. 9.
[0290] The position of the focus plane is controlled by the
focusing optics 1008 and can be moved in a range large enough to at
least coincide with the scan surface 1014. A single sub-scan is
obtained by collecting a number of 2D images at different positions
of the focus plane and at different positions of the fringe
pattern, as previously described. As the focus plane coincides with
the scan surface at a single pixel position, the fringe pattern
will be projected onto the surface point in-focus and with high
contrast, thereby giving rise to a large variation, or amplitude,
of the pixel value over time. For each pixel it is thus possible to
identify individual settings of the focusing optics for which each
pixel will be in-focus. By using knowledge of the optical system,
it is possible to transform the contrast information vs. position
of the focus plane into 3D surface information, on an individual
pixel basis.
[0291] In one embodiment, a mirror arrangement 1011, consisting of
a single reflective mirror, or prism, or an arrangement of mirrors,
are located after the probe objective 1010. This arrangement serves
to reflect the rays to a viewing direction different from that of
the of the probe axis. Different example mirror arrangements are
found in FIGS. 12A-12D. In one particular embodiment, the angle
between the mirror normal and the optical axis is approximately 45
degrees, thus providing a 90 degree view with respect to the probe
axis--an arrangement ideal for looking round corners. A transparent
window 1012 is positioned adjacent to the mirror and as part of the
probe casing/shell, to allow the light to pass between the probe
and the scan object, while keeping the optics clean from outside
dirt particles.
[0292] To reduce the probe movement required by a scanner operator,
the mirror arrangement may be rotated using a motor 1013. In one
embodiment, the mirror arrangement rotates with constant velocity.
By full rotation of a single mirror, it is in this way possible to
scan with 360 degree coverage around the probe axis without
physically moving the probe. In this case, the probe window 1012 is
required to surround/go all around the probe to enable viewing in
every angle. In another embodiment, the mirror rotates with a
certain rotation oscillation frequency. In yet another embodiment,
the mirror arrangement tilt with respect to the probe axis is
varied with a certain oscillation frequency.
[0293] A particular embodiment uses a double mirror instead of a
single mirror (FIGS. 12B and 12D). In a special case, the normal of
the two mirrors are angled approx. 90 degrees with respect to each
other. The use of a double mirror helps registration of the
individual sub-scans, since information of two opposite surfaces in
this way is obtained at the same time. Another benefit of using a
double mirror is that only 180 degrees of mirror rotation is
required to scan a full 360 degrees. A scanner solution employing
double mirrors may therefore provide 360 degrees coverage in less
time than single mirror configurations.
"Pistol-Like" Grip
[0294] FIG. 17 shows an embodiment of the scanner with a
pistol-like grip 2001. This form is particularly ergonomic. The
scanner in FIG. 17 is designed for intra-oral scanning of teeth.
The tip 2002 can be removed from the main body of the scanner and
can be autoclaved. Furthermore, the tip can have two positions
relative to the main body of the scanner, namely looking down (as
in FIG. 17) and looking up. Therefore, scanning the upper and the
lower mouth of a patient is equally comfortable for the operator.
Note that the scanner shown in FIG. 17 is an early prototype with
several cables attached for testing purposes only.
[0295] Although some embodiments have been described and shown in
detail, the invention is not restricted to them, but may also be
embodied in other ways within the scope of the subject matter
defined in the following claims. In particular, it is to be
understood that other embodiments may be utilised and structural
and functional modifications may be made without departing from the
scope of the present invention.
[0296] In device claims enumerating several means, several of these
means can be embodied by one and the same item of hardware. The
mere fact that certain measures are recited in mutually different
dependent claims or described in different embodiments does not
indicate that a combination of these measures cannot be used to
advantage.
[0297] It should be emphasized that the term "comprises/comprising"
when used in this specification is taken to specify the presence of
stated features, integers, steps or components but does not
preclude the presence or addition of one or more other features,
integers, steps, components or groups thereof.
[0298] The features of the method described above and in the
following may be implemented in software and carried out on a data
processing system or other processing means caused by the execution
of computer-executable instructions. The instructions may be
program code means loaded in a memory, such as a RAM, from a
storage medium or from another computer via a computer network.
Alternatively, the described features may be implemented by
hardwired circuitry instead of software or in combination with
software.
* * * * *