U.S. patent application number 15/419334 was filed with the patent office on 2017-05-18 for method for optically scanning and measuring an environment.
The applicant listed for this patent is FARO Technologies, Inc.. Invention is credited to Daniel Flohr, Dag Frommhold, Martin Heide, Oliver Knorzer, Martin Ossig.
Application Number | 20170139046 15/419334 |
Document ID | / |
Family ID | 58643516 |
Filed Date | 2017-05-18 |
United States Patent
Application |
20170139046 |
Kind Code |
A1 |
Ossig; Martin ; et
al. |
May 18, 2017 |
METHOD FOR OPTICALLY SCANNING AND MEASURING AN ENVIRONMENT
Abstract
A method for optically scanning, measuring and displaying a
point cloud is provided. The method includes emitting, by a laser
scanner, an emission light beam and receiving a reflection light
beam that is reflected from an object. A control device determines
for measurement points projected on a plane corresponding to a
screen, wherein at least some measurement points are displayed on a
display device. One or more pixels are gap filled to generate a
visual appearance of a surface on the display device. Wherein the
gap filling includes a first horizontal search in a first direction
of a first measured point of the measurement points followed by a
second horizontal search in a second direction of the first
measured point. The gap filling further includes a first vertical
search in a third direction of the measured point, followed by a
second vertical search in a fourth direction.
Inventors: |
Ossig; Martin; (Tamm,
DE) ; Flohr; Daniel; (Stuttgart, DE) ;
Frommhold; Dag; (Neuffen, DE) ; Knorzer; Oliver;
(Korntal-Muenchingen, DE) ; Heide; Martin;
(Stuttgart, DE) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
FARO Technologies, Inc. |
Lake Mary |
FL |
US |
|
|
Family ID: |
58643516 |
Appl. No.: |
15/419334 |
Filed: |
January 30, 2017 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
15140909 |
Apr 28, 2016 |
|
|
|
15419334 |
|
|
|
|
13697031 |
Apr 29, 2013 |
9329271 |
|
|
PCT/EP2011/001662 |
Apr 1, 2011 |
|
|
|
15140909 |
|
|
|
|
61362810 |
Jul 9, 2010 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G01C 15/002 20130101;
G01S 7/4808 20130101; G01S 7/4817 20130101; G01S 17/42 20130101;
G01S 7/003 20130101; G01S 17/89 20130101; G01S 7/51 20130101 |
International
Class: |
G01S 17/89 20060101
G01S017/89; G01S 7/48 20060101 G01S007/48; G01S 7/481 20060101
G01S007/481; G01S 7/51 20060101 G01S007/51 |
Foreign Application Data
Date |
Code |
Application Number |
May 10, 2010 |
DE |
10 2010 020 925.2 |
Claims
1. A method for optically scanning, measuring and displaying a
point cloud, the method comprising: emitting, by a light emitter of
a laser scanner, an emission light beam; receiving, by a light
receiver, a reflection light beam, wherein a reflection light beam
of the emission light beam is reflected from an object;
determining, by a control and evaluation device for measurement
points projected on a plane corresponding to a screen on a display
device, at least the distance from the object to a center of the
laser scanner, wherein at least some measurement points are
displayed on the display device; determining which points are
visible on the display device based at least in part on a viewpoint
of the display device; and gap filling one or more pixels to
generate a visual appearance of a surface on the display device,
wherein the gap filling includes a first horizontal search in a
first direction of a first measured point of the measurement points
followed by a second horizontal search in a second direction of the
first measured point, the second direction being opposite the first
direction and wherein the gap filling further includes a first
vertical search in a third direction of the measured point, the
third direction being perpendicular to the first direction,
followed by a second vertical search in a fourth direction, the
fourth direction being opposite the third direction.
2. The method of claim 1, further comprising: determining if
additional pixels could be filled in a horizontal or vertical gap
filling step; and repeating the gap filling of one or more pixels
until it was determined that no additional pixels could be
filled.
3. The method of claim 2, further comprising displaying an image
having the surface on the display.
4. The method of claim 1, wherein the first horizontal search
further comprises: identifying a first pixel, either empty, or
corresponding to the first measured point; determining when a
second pixel is located in the first direction is within a
predetermined radius; determining an adaptive maximum search radius
that is anti-proportional to a first distance to the first measured
point; determining if the second pixel is located within the
adaptive maximum search radius; and determining a second distance
of the second pixel.
5. The method of claim 4, wherein the second horizontal search
further comprises: identifying a third pixel is located within the
predetermined radius; determining if the third pixel is located
within the same adaptive maximum search radius; and determining a
third distance of the third pixel.
6. The method of claim 5, wherein the gap filling further
comprises: determining the difference between the first distance
and the second distance is less than a threshold; determining the
difference between the second distance and the third distance is
less than a threshold; and assigning a depth and color to the first
pixel by interpolating depth and color of the second pixel and the
third pixel.
7. The method of claim 6, wherein the interpolation is
weighted.
8. The method of claim 7, wherein the weighting is 1/r.sub.1.sup.2
and 1/r.sub.2.sup.2 wherein r.sub.1 is the distance in pixel units
from the second to the first pixel, and r.sub.2 is the distance in
pixel units from the third to the first pixel.
9. The method of claim 5, wherein the gap filling further
comprises: determining the difference between the first distance
and the second distance is less than a threshold; determining the
difference between the second distance and the third distance is
greater than a threshold; determining the third distance is greater
than the second distance; and assigning the depth and color of the
third pixel to the first pixel.
10. The method of claim 5, wherein the gap filling further
comprises: determining the difference between the first distance
and the second distance is less than a threshold; determining the
difference between the second distance and the third distance is
greater than a threshold; determining the third distance is less
than the second distance; and assigning the depth and color of the
second pixel to the first pixel.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] The present application is a continuation-in-part
application of U.S. application Ser. No. 15/140,909 filed on Apr.
28, 2016, which is a continuation application of U.S. application
Ser. No. 13/697,031 filed on Apr. 29, 2013. U.S. application Ser.
No. 13/697,031 is a National Stage Application of PCT Application
No. PCT/EP2011/001662, filed on Apr. 1, 2011, which claims the
benefit of U.S. Provisional Patent Application No. 61/362,810,
filed on Jul. 9, 2010, and of pending German Patent Application No.
DE 10 2010 020 925.2, filed on May 10, 2010, and which are hereby
incorporated by reference.
BACKGROUND
[0002] The invention relates to a system and method for optically
scanning and measuring an environment, and in particular, to a
system and method for generating a display image from a point
cloud.
[0003] Metrology devices, such as laser scanners for example, may
generate large volumes of coordinate data of points located on the
surfaces of the scanned area. These types of devices may be used to
generate three-dimensional models of an area, such as a home or
building, a crime scene, or an archeological site for example.
Often with these types of scans, the data may be acquired from
multiple positions to capture all of the desired surfaces and avoid
having blank areas where a surface was in the "shadow" of another
object. As a result in several data-sets of coordinate data are
generated that are registered together to define a single data-set,
sometimes colloquially referred to as a "point cloud" since the
data is represented as a group of points in space without
surfaces.
[0004] It should be appreciated that from a graphical display of a
point cloud, it may be difficult to visualize the surfaces of the
scanned area. This is due to the close proximity of points (from
any user point of view) within the point cloud that may lie on
different planes. For example, if the user point of view of the
point cloud is looking down on a table, there will be points within
the field of view from the table surface, along with the floor that
is underneath the table surface or even the surface on the
underside of the table.
[0005] Where the point cloud is relatively dense, meaning that the
points on a surface are dense, the generation of surfaces in the
displayed image for visualizing the scanned area may be created,
albeit computationally intensive. However, in some applications,
the point cloud may have a lower density of points resulting in
gaps in the data set between the points of the point cloud. As a
result, it may be difficult to generate a desired displayed
image.
[0006] Accordingly, while existing metrology devices and point
cloud display systems are suitable for their intended purposes the
need for improvement remains, particularly in providing a system
for filling in pixels on a graphical display to generate a
displayed image of a point cloud.
SUMMARY
[0007] According to an embodiment of the present invention a method
for optically scanning, measuring and displaying a point cloud is
provided. The method includes emitting, by a light emitter of a
laser scanner, an emission light beam. A light receiver receives a
reflection light beam, wherein a reflection light beam of the
emission light beam is reflected from an object. A control and
evaluation device determines for measurement points projected on a
plane corresponding to a screen on a display device, at least the
distance from the object to a center of the laser scanner, wherein
at least some measurement points are displayed on the display
device. The points are visible are determined on the display device
based at least in part on a viewpoint of the display device. One or
more pixels are gap filled to generate a visual appearance of a
surface on the display device. Wherein the gap filling includes a
first horizontal search in a first direction of a first measured
point of the measurement points followed by a second horizontal
search in a second direction of the first measured point, the
second direction being opposite the first direction. The gap
filling further includes a first vertical search in a third
direction of the measured point, the third direction being
perpendicular to the first direction, followed by a second vertical
search in a fourth direction, the fourth direction being opposite
the third direction.
BRIEF DESCRIPTION OF THE DRAWINGS
[0008] Embodiments of the present invention are explained in more
detail below on the basis of exemplary embodiments illustrated in
the drawings, in which
[0009] FIG. 1 is a schematic illustration of a laser scanner in the
environment including a display device, and
[0010] FIG. 2 is a partial sectional illustration of the laser
scanner.
[0011] FIG. 3 is a schematic illustration of a scanned area with
measured points shown on surfaces;
[0012] FIG. 4 is a schematic illustration of the assignment and
filling of the pixels with a view onto the plane, wherein the
adjacent pixels are on the same surface;
[0013] FIG. 5 is a schematic illustration of the assignment and
filling of the pixels, according to FIG. 4, with a view onto the
plane, in accordance with an embodiment;
[0014] FIG. 6 is a schematic illustration of the assignment and
filling of the pixels with a view onto the plane, wherein the
adjacent pixels are located on different surfaces, in accordance
with an embodiment;
[0015] FIG. 7 is a schematic illustration of the assignment and
filling of the pixels, according to FIG. 3, with a view onto the
plane, in accordance with an embodiment;
[0016] FIG. 8 is a flow diagram of a method of filling pixels in
accordance with an embodiment;
[0017] FIG. 9-FIG. 11 are flow diagrams of a method of performing a
horizontal row pixel filling in accordance with an embodiment;
[0018] FIG. 12-FIG. 16 are schematic illustrations of a sequence of
horizontal and vertical pixel filling steps performed using the
methods of FIG. 6;
DETAILED DESCRIPTION
[0019] Referring to FIG. 1 and FIG. 2, a laser scanner 10 is
provided as a device for optically scanning and measuring the
environment of the laser scanner 10. The laser scanner 10 has a
measuring head 12 and a base 14. The measuring head 12 is mounted
on the base 14 as a unit that can be rotated about a vertical axis.
The measuring head 12 has a rotary mirror 16 that can be rotated
about a horizontal axis. The point of intersection between the two
axes of rotation is designated as the center C.sub.10 of the laser
scanner 10.
[0020] The measuring head 12 also has a light emitter 17 for
emitting an emission light beam 18. The emission light beam 18 may
be a laser beam in the range of wavelength of approximately 300 to
1600 nm, for example, 790 nm, 905 nm or less than 400 nm, but other
electro-magnetic waves having, for example, a greater wavelength
can be used. The emission light beam 18 may be amplitude-modulated
with, for example, a sinusoidal or rectangular-waveform modulation
signal. The emission light beam 18 is passed from the light emitter
17 onto the rotary mirror 16 where it is deflected and then emitted
into the environment. A reception light beam 20, which is reflected
by or otherwise scattered from an object O, is captured again by
the rotary mirror 16, deflected and passed onto a light receiver
21. The direction of the emission light beam 18 and of the
reception light beam 20 results from the angular positions of the
rotary mirror 16 and the measuring head 12, which depend on the
positions of their respective rotary drives which are, in turn,
detected by a respective angular encoder.
[0021] A control and evaluation device 22 has a data link
connection to the light emitter 17 and to the light receiver 21 in
the measuring head 12, parts thereof being arranged also outside
the measuring head 12, for example as a computer connected to the
base 14. The control and evaluation device 22 determines, for a
multiplicity of measurement points X, the distance d of the laser
scanner 10 from the illuminated point on the object O (e.g. the
table T, FIG. 3), and from the propagation times of emission light
beam 18 and reception light beam 20. In an embodiment, the distance
d is determined by the control and evaluation device 22 based on
the phase shift between the two light beams 18 and 20. In another
embodiment, the distance d is determined by the control and
evaluation device based on a measured round trip time between an
outgoing emission pulse of light 18 and a returning reception pulse
of light 20. In another embodiment, the distance d is determine by
the control and evaluation device based on change in optical
wavelength of a reception beam of light 20 relative to an emission
beam of light, the emission beam of light having an optical
wavelength that is linearly swept (chirped) in time. Many other
methods are known in the art and may be used to determine the
distance d.
[0022] Through use of the relatively rapid rotation of the mirror
16, scanning takes place along a circular line. Also, through use
of the relatively slow rotation of the measuring head 12 relative
to the base 14, the entire space is gradually scanned with the
circular lines. The totality of the measurement points X of such a
measurement may be referred to as a scan. The center C.sub.10 of
the laser scanner 10 defines for such a scan the origin of the
local stationary reference system. The base 14 is stationary in
this local stationary reference system. In other embodiments, a
different type of beam steering method is used to direct the
emission beam 18 to different points X in the environment. In an
embodiment, a pair of galvanometer steering mirrors are used to
rapidly direct the emission beam 18 to desired points X. In another
embodiment, the emission beam 18 rotates along with the entire
measuring head 12. Many types of beam steering mechanisms are known
in the art and may be used.
[0023] In addition to the distance d to the center C.sub.10 of the
laser scanner 10, each measurement point X comprises a brightness
value which may also be determined by the control and evaluation
device 22. Here the term brightness may be understood to refer to a
level of optical power or a level of irradiance (optical power per
unit area).The brightness is a gray-tone value which is determined,
for example, by integration of the bandpass-filtered and amplified
signal of the light receiver 21 over a measuring period which is
assigned to the measurement point X. Through use of a color camera,
it is optionally possible to generate pictures, by which colors (R,
G, B) can be assigned as a value to the measurement points X in
addition to the brightness or comprising the brightness.
[0024] A display device 30 is connected to the control and
evaluation device 22. The display device 30 can be integrated into
the laser scanner 10, for example into the measuring head 12 or
into the base 14, or it can be an external unit, for example part
of a computer which is connected to the base 14. The display device
30 has a graphics card 32 and a screen 34 which can be arranged
separately from one another or as a structural unit. The control
and evaluation device 22 provides 3D data of the scan.
[0025] It should be appreciated that when the measurement points X,
which are positioned in a three-dimensional space, are displayed on
a two-dimensional display device 30, it may be difficult for the
user to visualize the object O on the display. For example,
referring to FIG. 3 a table T has been scanned with the laser
scanner 10 and the measurement points X have been acquired. When
the user displays the measurement points X from a viewpoint V, the
user will see not only the measurement points X.sub.T from the
tabletop, but also the measurement points X.sub.F. As a result,
without a visual reference such as a surface of the table T or the
surfaces of the legs of the table, it will be difficult for the
user to distinguish between the measurement points X.sub.T and the
measurement points X.sub.F. It should be appreciated that in an
embodiment where the measured object O is a structure, such as the
inside of a building and the measured points may be on opposite
sides of a wall. Accordingly, embodiments herein provide advantages
generating on display device 30 an image representing surfaces on
the object O by filling in the pixels between measured points. The
method of filling in the pixels between the measured points may
sometimes be referred to a gap filling or pixel filling.
[0026] In many cases, objects in an environment are scanned with a
laser scanner 10 placed in a plurality of locations. This is
illustrated in FIG. 3, where a scanner 10A is located at a first
location at a first time and the same or different scanner 10B is
located at a different location at a second time. In some
embodiments, multiple scanners such as 10A, 10B are used
simultaneously. In an embodiment, the scanner 10A is positioned to
scan the top side of the table T and to a lesser extent floor
points to the side of the table. The scanner 10B is positioned to
scan mostly points beneath the table T rather than on top of the
table. When the measurement points X.sub.T are registered with the
measurement points X.sub.F, the result is a point cloud having
points distributed over three-dimensional space. From the
perspective of a viewer V, such a collection of sparsely located
points may not clearly indicate which points are closer to the
viewer and which are farther from the viewer. Hence a display of
the collected points in the point cloud may intersperse the floor
points and the tabletop points, with the surfaces not clearly
distinguished for the viewer V.
[0027] Referring now to FIGS. 4-7, with continuing reference to
FIG. 1 and FIG. 2, the graphic card 32 converts the 3-D data into
2-D data (e.g., rendering data), which are displayed on the screen
34. The 3-D data are the measurement points X, wherein several
scans from different positions of the laser scanner 10 can be
assembled into one scene. For representing the 2-D data, there are
pixels P, i.e., adjacent, small polygonal surfaces (e.g. squares or
hexagons), which are arranged in a two-dimensional plane E which
corresponds to the screen 34. In an embodiment, a starting point
for the representation on the screen 34 is the projection of the
measurement points X onto a plane E with a viewer (e.g., eye,
camera), at a certain viewpoint V. The projection appears to be in
perspective (i.e., the viewpoint V is in the finite) or
orthographical (i.e., the viewpoint V in is the infinite). In an
embodiment, the projected measurement points X are assigned to
single pixels P.sub.0, P.sub.1, P.sub.2. A Z-buffer, which is a
two-dimensional array (field) of values for the pixels P. In this
Z-buffer, a field element (depth z) is assigned to each pixel P.
The depth z of each projected measurement point X corresponds to
the distance of the measurement point X to the plane E with respect
to the viewpoint V. The field of the pixels P and the Z-buffer may
be treated in the same way as the images.
[0028] The viewpoint V may be arbitrary per se and is usually
changed several times when regarding the scan and/or the scene. In
other words, the user may elect to view the measurement point data
from a variety of perspectives, with the changes to the displayed
measurement points X re-rendered.
[0029] Since the measurement points X are punctiform, with gaps in
between, and the pixels P usually, in the case of nearby objects O,
have a higher density in the plane E than do the projections of the
measurement points X, a gap-filling method is carried out to fill
as many pixels P as possible for an improved visual representation
for the user. In an embodiment, the graphic card 32 carries out
this method by processing the 3-D data following a parallel method
described further herein below, the parallel processing method
based at least in part on the indication of the viewpoint V and of
the plane E.
[0030] In an embodiment, initially only those pixels P are filled
to which the projection of a measurement point X is assigned, i.e.,
which exactly cover one measurement point X, such as pixels
P.sub.0, P.sub.1, P.sub.2 for example. These pixels P are filled
with the value of the assigned measurement point X, i.e.,
brightness (gray-scale value) and, where applicable, color. All
other pixels P, which do not exactly correspond with a projection
of a measurement point X, i.e., which are "in between" are empty at
first, for example are set to a value of zero (e.g. empty or
blank). Each of the depths z, i.e., the field elements of the
Z-buffer, which are assigned to the initially filled pixels P, is
set to that depth z.sub.0, z.sub.1, z.sub.2, which corresponds to
the distance of the assigned measurement point X to the plane E. In
an embodiment, sometimes referred to as perspective projection, the
distance z1 is marked from an object point to a point P1 on a plane
E along a line directed to a viewer point V. In this embodiment,
the distance z1 is the distance from the object point to point P1.
In another embodiment, sometimes referred to as orthographic
projection, the distance is a perpendicular distance from the
object point to the point P1 rather than along the line aimed
toward V.
[0031] All other field elements of the Z-buffer (e.g., depths z)
are set to a very large value (relative to the scale of the
measured points), for example, to approximate an infinite depth.
If, when the projection of the measurement points X is made, it
turns out that two measurement points X are available for one pixel
P, the measurement point having the smaller depth z is selected and
the other one is rejected, so that covered surfaces and covered
edges, such measurement points X.sub.F on the floor of FIG. 3 for
example, are not visible.
[0032] According to an embodiment, the gap-filling method is
performed in dependence on the depth z.sub.0, z.sub.1, z.sub.2,
i.e., on the distance to the plane E. The graphic card 32 selects
all pixels P in parallel with respect to time. By way of example,
one selected pixel P.sub.0 is regarded now. The assigned depth z,
i.e., field element of the Z-buffer, contains the depth z.sub.0.
For each selected pixel P.sub.0 the adjacent pixels P.sub.1,
P.sub.2, are searched consecutively, i.e., alternating between to
the left and to the right, and above and below. If the adjacent
pixel P.sub.1 is not yet filled or if its depth z is bigger than
the depth z.sub.0 of the selected pixel P.sub.0, it is skipped and
the second next pixel P is taken as adjacent pixel P.sub.1, if
necessary iteratively. If an adjacent pixel P.sub.1, the depth
z.sub.1 of which is smaller than the depth z.sub.0 of the selected
pixel P.sub.0, is found in one of the directions, a change to the
next direction takes place, and it is looked for the adjacent pixel
P.sub.2 (e.g., the depth z.sub.2 of which is smaller than the depth
z.sub.0 of the selected pixel P.sub.0). It is possible to define a
maximum number of skipped pixels, i.e., if the adjacent pixel
P.sub.1 or P.sub.2 is not yet found after skipping the maximum
number of skipped pixels, the search for P.sub.1 or P.sub.2 is
aborted. In one embodiment, the search for pixels to fill about an
initial pixel P.sub.0 is limited to those pixels within a radius R
of the pixel P.sub.0 on the two-dimensional pixel array. In another
embodiment, the search is also limited by another radius in 3-D
coordinates, perpendicular to the viewing direction.
[0033] If the adjacent pixels P.sub.1 and P.sub.2 to the selected
pixel P.sub.0 have been found in opposite directions, with the
depths z.sub.1 and z.sub.2 of the adjacent pixels P.sub.1 and
P.sub.2 being smaller than the depth z.sub.0, it is checked whether
P.sub.1 and P.sub.2 are on the same plane, i.e., whether the amount
of the difference of z.sub.2 and z.sub.1 is below a threshold value
for the depth z.sub.crit, i.e.,
|z.sub.2-z.sub.1|<z.sub.crit
applies. In such a case, the selected pixel P.sub.0 is filled with
the value which is interpolated between P.sub.1 and P.sub.2, i.e.,
brightness (gray-scale level) and, if applicable color. The
assigned field element of the Z-buffer is likewise set to the
interpolated depth between z.sub.1 and z.sub.2. Interpolation
weights depend on the distance of the selected pixel P.sub.0 from
P.sub.1 and P.sub.2 in plane E.
[0034] If the difference of the depths is too big, i.e., the
condition
|z.sub.2-z.sub.1|>z.sub.crit
applies, it is assumed that P.sub.1 and P.sub.2 are located on
different planes. The selected pixel P.sub.0 is then filled with
the value, i.e., brightness and, if applicable color, of, for
example, the more remote pixel P.sub.1 or P.sub.2, and the assigned
field element of the Z-buffer with the bigger depth z.sub.1 or
z.sub.2. In another embodiment, the value and the depth of pixel
P.sub.1 or P.sub.2 having a larger depth z.sub.1 or z.sub.2 is
transferred or assigned to the other pixel. In the case of more
than two adjacent pixels P .sub.1, P.sub.2, the average value of
the majority, i.e., of the adjacent pixels P.sub.1, P.sub.2, which
are located on the same surface, can be transferred.
[0035] Selected pixels P.sub.0, which are already filled with
values of the measurement points, are overwritten by the
interpolation of the values of the adjacent pixels P.sub.1 and
P.sub.2. In another embodiment, a selected pixel P.sub.0, which is
already filled, remains unvaried if |z.sub.0-z.sub.1| is smaller
than a threshold, i.e. if P.sub.0 and P.sub.1 already belong to the
same surface.
[0036] If pixels P have been skipped when finding the pixels
P.sub.1 and P.sub.2, because they were not filled or because their
depth z was too big, their adjacent pixels P.sub.1, P.sub.2 are the
same as with the selected pixel P.sub.0, so that the skipped pixels
P.sub.F and the assigned field elements of the Z-buffer, within the
framework of the selection taking place in parallel, are likewise
filled either with a value which is interpolated between the pixels
P.sub.1 and P.sub.2 and/or the depths z.sub.1 and z.sub.2
(depending on the distance of the selected pixel P.sub.0 from
P.sub.1 and P.sub.2 in plane E) or with the value and/or the depth
z.sub.1 or z.sub.2 of the more remote one among pixels P.sub.1 or
P.sub.2 (or the average value of the majority) as shown in FIG.
7.
[0037] Due to the selection taking place in parallel, filling with
the value and/or the depth z.sub.1 or z.sub.2 of the more remote
among the pixels P.sub.1 or P.sub.2 on account of a difference of
depths which is too big, leads to the closer-by pixel P.sub.1 or
P.sub.2 forming an edge. Even if no adjacent pixel P.sub.1 or
P.sub.2 is found, the depth z.sub.1 or z.sub.2 of which is smaller
than the depth z.sub.0 of the selected pixel P.sub.0, since the
selected pixel P.sub.0 is at the side of the screen 34, an edge is
generated, since these selected pixels P.sub.0 at the edge are not
filled then.
[0038] Once the pixels along a horizontal row are filled, the same
process may be used to fill the vertical gaps between the
measurement points P.sub.3, P.sub.4 in a similar manner to that
described herein above for the horizontal gap filling. As will be
discussed in more detail below, the gap-filling process may be
iteratively performed until the image is created.
[0039] Gap-filling may take place in the control and evaluation
device 22 or by software running on an external computer. Due to
the savings in time by a parallel selection, the hardware-based
gap-filling on the graphic card 32 may be used together with the
programming interface of the latter.
[0040] Referring now to FIG. 8 an embodiment is shown of a method
100 for acquiring measurement points and displaying the point cloud
using gap filling of pixels. The method 100 begins in block 102
where the object O is scanned with a metrology device, such as but
not limited to a laser scanner 10 for example. It should be
appreciated that the scanned object O may comprise multiple
objects, such as the interior rooms and hallways of a building for
example. The step of scanning the object O results in the
generation of a point cloud representing measured points X on the
surfaces of the object O.
[0041] The method 100 then proceeds to block 104 where it is
determined which of the measured points X are visible on the
display device 30. As discussed herein above the three-dimensional
point cloud is projected onto the two-dimensional screen 34 of
display device 30 to allow the user to view the point cloud. The
method then proceeds to block 106 where the pixels are gap filled
to provide the visual appearance of surfaces in the display device
30. In an embodiment, the step of gap filling in block 106 includes
a horizontal search to the left of a pixel containing measured
point in block 108 and then a search to the right of the pixel in
block 110. The step of gap filling in block 106 further includes
the steps of a vertical search above (e.g. towards the top of the
display device 34) the pixel containing the measured point in block
112 and then a search below the pixel in block 114. In an
embodiment, the vertical search of blocks 112, 114 follow the
horizontal search of blocks 108, 110, it should be appreciated,
however, that this is for exemplary purposes and the claims should
not be so limited. In other embodiments, the vertical search
precedes the horizontal search. In an embodiment, the vertical
search is performed sequentially with the horizontal search. In
another embodiment, the horizontal and vertical searches are
performed simultaneously in both directions.
[0042] The gap filling steps may be performed for all of the pixels
containing a measured point that are being displayed on the display
device. In an embodiment, the pixel filling steps are the same as
those described herein above with respect to FIGS. 4-7. Once the
gap filling is performed, the method 100 then proceeds to query
block 116 where it is determined whether all of the pixels have
been evaluated for gap filling. As will be discussed in more detail
below, gaps or empty pixels may occur in some cases where the pixel
is not bounded by a pixel containing a measured point or a
previously filled pixel. To resolve this issue, the gap filling of
block 106 may be an iterative process. In one embodiment, the
process performs for a fixed number of iterations. In another
embodiment, the process performs until a stop condition is
achieved. In one embodiment, the stop condition is that no
additional pixels could be filled in the previous iteration.
[0043] When the query block 116 returns an affirmative, meaning
that there are unevaluated pixels, the method loops back to block
106. When query block 116 returns a negative, the method 100
proceeds to block 118 where the image is displayed on the display
device 30.
[0044] Referring now to FIGS. 9-11, an embodiment of a method 200
for the gap filling steps of blocks 108-110. It should be
appreciated that while method 200 is described with reference to
horizontal gap filling, this is for exemplary purposes and the
steps of method 200 may also be performed for vertical gap filling
as well.
[0045] The method 200 begins in block 202 where a pixel P.sub.0 is
identified. The pixel P.sub.0 is a pixel that contains an image of
the measured point X that is projected from the three-dimensional
point cloud onto the two-dimensional display device 30. The method
200 then proceeds to block 204 where a search is performed of
pixels to the left of the pixel P.sub.0. As used herein, the terms
"left" and "right" indicate opposite sides in the row of pixel
P.sub.0, this is done for clarity purposes and no intended
orientation of the display device 34 or the point cloud data is
intended.
[0046] The method 200 then proceeds to query block 206 where it is
determined if a pixel P.sub.1 is found within a radius R (FIG. 7).
In the exemplary embodiment, the radius R is initially a
predetermined value defined by the user, such as five pixels for
example. When the query block 206 returns a negative (no pixel
P.sub.1 found), the method 200 proceeds to block 208 and no gap
filling is performed. When the query block 206 returns a positive
(a pixel P.sub.1 is found), the method proceeds to block 210 where
a maximum search radius in pixels is determined where the radius is
non-proportional or anti-proportional to a distance d1, where the
distance d1 is the same as z1 of FIG. 4. The search is performed in
pixel space. It should be appreciated that each pixel may represent
a different size in 3D space. For example, if a ray extends through
each pixel from the view point into space, the distance between the
rays will increase with the distance or depth from the viewpoint
(perspective projection). In an embodiment, the search re-projects
the pixels into 3D space and objects within a predetermined
distance.
[0047] The method 200 then proceeds to query block 212 where it is
determined whether the distance (x0, y0), (x1, y1) is between the
measured points of pixels P.sub.0, P.sub.1. When this distance is
not within the radius determined in block 210, the query block 212
returns a negative and proceeds to block 214 where the direct
neighboring pixel P.sub.1 to the left of (x0, y0) is evaluated in
query block 216. In query block 216, it is determined whether the
pixel is non-empty and has a depth less than the depth to the
measured point X of P.sub.0. When query block 216 returns a
negative, meaning either the pixel is empty or it has a depth
greater than the measured point X of P.sub.0, the method 200 loops
back to block 208. When the query block 216 returns a positive, the
method 200 proceeds to block 218 where the values of (x1, y1) and
d1 are set to this pixel (e.g. pixel P.sub.1). Once the values of
block 218 are set, or when the query block 212 returns a positive,
the method 200 proceeds to block 211 (FIG. 10).
[0048] In block 211, a search is performed of the pixels to the
right of the pixel P.sub.0. The method then proceeds to query block
213 where it is determined whether pixel P2 is within the radius R
(e.g. 5 pixels). When the query block 213 returns a negative, the
method 200 proceeds to block 208 and no gap filling is performed.
When the query block 213 returns a positive, the a maximum search
radius is determined in block 215 that is non-proportional or
anti-proportional to the distance d2, where the distance d2 is
equivalent to the distance z2 of FIG. 4. The method 200 then
proceeds to query block 217 where it is determined if the pixel
distance (x0, y0), (x2, y2) is within the radius determined in
block 215.
[0049] When query block 217 returns a negative, the method 200
proceeds to block 219 where the direct neighboring pixel to the
right of (x0, y0) is evaluated. This provides for more even gap
filling even when the search criteria is not fulfilled. In an
embodiment, for directly neighboring pixels, the 3D-criteria is
ignored unless better pixels are identified. In query block 221, it
is determined if the pixel is non-empty and has a depth less than
the depth to the measured point X of P.sub.0. When query block 221
returns a negative, meaning either the pixel is empty or it has a
depth greater than the measured point X of P.sub.0, the method 200
loops back to block 208 and no gap filling is performed. When the
query block 221 returns a positive, the method 200 proceeds to
block 222 where the values of (x2, y2) and d2 are set to this pixel
(e.g. pixel P.sub.2). Once the values of block 222 are set, or when
the query block 217 returns a positive, the method 200 proceeds to
block 224 (FIG. 11).
[0050] In query block 224, it is determined if the difference
between the distances d0 and d1 (e.g. the distance from the
viewpoint V to the measured points x0 and x1 of FIG. 6) is greater
than a threshold (e.g. Z.sub.crit). In one embodiment, the
threshold is 2.5 centimeters. When query block 224 returns a
negative, indicating that that the measured points already lie on
the same plane, the method 200 proceeds to block 208 and no gap
filling is performed. When query block 224 returns a positive, the
method 200 proceeds to query block 226 where it is determined
whether the difference between the distances d1 and d2 (e.g. the
distance from the viewpoint V to the measured points x1 and x2 of
FIG. 6) is less than a threshold. In one embodiment, the threshold
of block 226 is 10 centimeters.
[0051] When query block 226 returns a positive, meaning the
difference in the depths is less than the threshold and that the
points lie on the same plane, the method 200 proceeds to block 228
where the depth and color of pixels P1 and P2 is interpolated and
assigned to the intermediate or intervening pixels. In one
embodiment, the depth and color interpolation is weighted with
weights (1/r.sub.1.sup.2) and (1/r.sub.2.sup.2), where r.sub.1 is
the distance in pixel units from P1 to P0, and r.sub.2 is the
distance in pixel units from P2 to P0.
[0052] When query block 226 returns a negative, the method 200
proceeds to query block 230 where it is determined if the depth d2
is greater than the depth d1. When the query block 230 returns a
positive, the color and depth values from the pixel P.sub.2 are
used for gap filling the empty pixels between pixels P.sub.1 and
P.sub.2. When the query block 230 returns a negative, the color and
depth values from the pixel P.sub.1 are used for gap filling the
empty pixels between pixels P.sub.1 and P.sub.2. The purpose of
block 230 is to avoid extending objects by additional pixels at
their edges.
[0053] It should be appreciated that the method 200 may be repeated
for the vertical gap filling as well.
[0054] Turning now to FIGS. 12-16, an example is illustrated of the
iteration steps for gap filling, such as provided by query block
116 (FIG. 8) for example. FIG. 12 illustrates the horizontal gap
filling that occurs in the first iteration, the blocks with the "x"
represent pixels containing a measured point. The number "1"
represents empty pixels that are filled in the first iteration,
such as by method 100 for example. Once the horizontal gap filling
is completed, the vertical gap filling is performed as shown in FIG
.13. The number "2" represents the pixels filled during the first
iteration vertical gap filling. It should be noted empty pixels are
only filled when they are bounded on opposite sides by either a
pixel having a measurement point X or a previously filled pixel,
for example pixel 300 is filled because it is bounded by the pixels
302, 304. It should further be noted that once the first iteration
is completed, there are a number pixels, such as pixels 306, 308
that have not been evaluated for gap filling because at least one
side of the row in which the pixel is located was unbounded at the
beginning of the first iteration vertical gap filling.
[0055] Referring now to FIG. 14 a second gap filling iteration is
initiated. The second iteration starts with a horizontal gap
filling. In this iteration, the pixel 306 is evaluated and filled
based on the bounding or end pixels 310, 312. The number "3"
represents pixels that were gap filled during the second iteration
horizontal gap filling. It should be noted in an embodiment the gap
filled pixels have an associated depth and color that was assigned
during blocks 228, 232 or 234 of FIG. 11. This assigned depth and
color allows the intermediate pixels to be evaluated and filled.
With the second iteration horizontal gap filling completed, the
second iteration vertical gap filling is performed as shown in FIG.
15. Here, the additional pixels are evaluated and filled (where
appropriate), based on filled pixels that include those pixels
filled during the second iteration horizontal gap filling. For
example, pixel 308 is evaluated and filled based on end pixels 306
(filled in the second iteration horizontal gap filling) and pixel
312 (filled in the first iteration gap filling).
[0056] In the illustrated embodiment, when the second iteration
vertical gap filling is completed, there remain no unevaluated
bounded pixels. Therefore, the query block 116 would return a
negative and the method 100 would proceed with displaying the image
on the display device 34. It should be appreciated that in other
embodiments, more or less iterations may be performed.
[0057] It should be appreciated that while embodiments herein
describe the creation of a displayed image by a gap-filling process
with respect to data acquired by a laser scanning device, this is
for exemplary purposes and the claims should not be so limited. In
other embodiments, the displayed image may be created using point
cloud data generated by any metrology device, such as a
triangulation-type laser scanner device, a triangulation-type
structured light scanner device, or a time-of-flight based scanner
device for example.
* * * * *