U.S. patent application number 15/060028 was filed with the patent office on 2017-03-02 for code domain power control for structured light.
The applicant listed for this patent is QUALCOMM Incorporated. Invention is credited to Kalin Mitkov Atanassov, Albrecht Johannes Lindner, James Wilson Nash.
Application Number | 20170061634 15/060028 |
Document ID | / |
Family ID | 58095794 |
Filed Date | 2017-03-02 |
United States Patent
Application |
20170061634 |
Kind Code |
A1 |
Nash; James Wilson ; et
al. |
March 2, 2017 |
CODE DOMAIN POWER CONTROL FOR STRUCTURED LIGHT
Abstract
Systems and methods for controlling structured light laser
systems are disclosed. One aspect is a structured light system. The
system includes a memory device configured to store a depth map.
The system further includes an image projecting device including a
laser system configured to project codewords. The system further
includes a receiver device including a sensor, the receiver device
configured to sense the projected codewords reflected from an
object. The system further includes a processing circuit configured
to retrieve a portion of the depth map and calculate expected
codewords from the depth map. The system further includes a
feedback system configured to control the output power of the laser
system based on the sensed codewords and the expected
codewords.
Inventors: |
Nash; James Wilson; (San
Diego, CA) ; Atanassov; Kalin Mitkov; (San Diego,
CA) ; Lindner; Albrecht Johannes; (La Jolla,
CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
QUALCOMM Incorporated |
San Diego |
CA |
US |
|
|
Family ID: |
58095794 |
Appl. No.: |
15/060028 |
Filed: |
March 3, 2016 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
62212507 |
Aug 31, 2015 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06T 7/50 20170101; G01B
11/22 20130101; G06K 9/4661 20130101; G06K 9/6267 20130101; G06T
7/521 20170101; G06K 9/6215 20130101; G01B 11/2513 20130101; G06K
9/18 20130101; G06K 9/52 20130101 |
International
Class: |
G06T 7/00 20060101
G06T007/00; G01B 11/22 20060101 G01B011/22; G06K 9/62 20060101
G06K009/62; G06K 9/52 20060101 G06K009/52; G06K 9/18 20060101
G06K009/18; G06K 9/46 20060101 G06K009/46 |
Claims
1. A structured light system, comprising: a memory device
configured to store a depth map; an image projecting device
including a laser system configured to project codewords; a
receiver device including a sensor, the receiver device configured
to sense the projected codewords reflected from an object; a
processing circuit configured to retrieve at least a portion of the
depth map stored in the memory device and calculate expected
codewords from the depth map; and a feedback system configured to
control an output power of the laser system based on the sensed
codewords and the expected codewords.
2. The structured light system of claim 1, wherein the processing
circuit is further configured to update the depth map based on the
sensed codewords, and wherein the memory device is further
configured to store the updated depth map.
3. The structured light system of claim 2, wherein the feedback
system is configured to determine a code domain statistic comparing
the sensed codewords with the expected codewords, and wherein the
feedback system controls the output power of the laser system based
at least in part on the determined code domain statistic.
4. The structured light system of claim 3, wherein the code domain
statistic quantifies symbol classification accuracy.
5. The structured light system of claim 4, wherein the code domain
statistic is the square of the difference in intensity means
divided by the sum of the intensity variances.
6. The structured light system of claim 5, wherein the processing
circuit is further configured to calculate expected symbols from
the depth map and/or previously received codewords; the processing
circuit further configured to assign each received intensity value
to the corresponding expected symbols, the processing circuit
further configured to calculate a mean intensity value for each
symbol, the processing circuit further configured to calculate a
variance intensity value for each symbol, and the processing
circuit further configured to calculate the code domain statistic
as the square of the difference in intensity means divided by the
sum of intensity variances.
7. The structured light system of claim 3, wherein the code domain
statistic quantifies codeword detection accuracy.
8. The structured light system of claim 7, wherein the code domain
statistic is the percent of received codewords that match expected
codewords.
9. The structured light system of claim 8, wherein the processing
circuit is further configured to compare received codewords to
expected codewords, the processing circuit is further configured to
calculate the percentage of correctly received codewords, wherein
correctly received codewords correspond to expected codewords.
10. The structured light system of claim 3, wherein the code domain
statistic quantifies basis function accuracy.
11. The structured light system of claim 10, wherein the code
domain statistic is the percent of received basis functions that
match expected basis functions.
12. The structured light system of claim 11, wherein the processing
circuit is further configured to calculate expected basis functions
from the depth map and/or previously received codewords; the
processing circuit further configured to compare received basis
functions to expected basis functions, and the processing circuit
further configured to calculate the percentage of correctly
received basis functions, wherein correctly received basis
functions correspond to expected basis functions.
13. The structured light system of claim 3, wherein the feedback
system is configured to control an output power of the laser system
iteratively to converge to a maximum value for the code domain
statistic.
14. A method of controlling laser power in a structured light
system, comprising: storing a depth map with a memory device;
projecting codewords with a laser system; sensing the projected
codewords reflected from an object with a receiver sensor;
retrieving a portion of the depth map from the memory device;
calculating expected codewords from the depth map; and controlling
output power of the laser system based on the sensed codewords and
the expected codewords.
15. The method of claim 14, further comprising: updating the depth
map based on the sensed codewords; and storing the updated depth
map with the memory device.
16. The method of claim 14, further comprising: determining a code
domain statistic comparing the sensed codewords with the expected
codewords; and controlling output power of the laser system based
at least in part on the determined code domain statistic.
17. The method of claim 16, wherein the code domain statistic
quantifies symbol classification accuracy.
18. The method of claim 17, wherein the code domain statistic is
the square of the difference in intensity means divided by the sum
of the intensity variances.
19. The method of claim 18, further comprising: calculating
expected symbols from the depth map and/or previously received
codewords; assigning each received intensity value to the
corresponding expected symbols; calculating a mean intensity value
for each symbol; calculating a variance intensity value for each
symbol; and calculating the code domain statistic as the square of
the difference in intensity means divided by the sum of the
intensity variances.
20. The method of claim 16, wherein the code domain statistic
quantifies codeword detection accuracy.
21. The method of claim 20, wherein the code domain statistic is
the percent of received codewords that match expected
codewords.
22. The method of claim 21, further comprising: comparing received
codewords to expected codewords; and calculating the percentage of
correctly received codewords, wherein correctly received codewords
correspond to expected codewords.
23. The method of claim 16, wherein the code domain statistic
quantifies basis function accuracy.
24. The method of claim 23, wherein the code domain statistic is
the percentage of received basis functions that match expected
basis functions.
25. The method of claim 24, further comprising: calculating
expected basis functions from the depth map and/or previously
received codewords; comparing received basis functions to expected
basis functions, and calculating the percentage of correctly
received basis functions, wherein correctly received basis
functions correspond to expected basis functions.
26. The method of claim 16, further comprising controlling an
output power of the laser system iteratively to converge to a
maximum value for the code domain statistic.
27. A structured light system, comprising: means for storing a
depth map; means for projecting codewords; means for sensing the
projected codewords reflected from an object; means for retrieving
a portion of the depth map from the means for storing a depth map;
means for calculating expected codewords from the depth map; and
means for controlling output power of the projecting means based on
a comparison between the sensed codewords and the expected
codewords.
28. The structured light system of claim 27, wherein the storing
means comprises a memory device, wherein the project means
comprises a laser system, wherein the sensing means comprises a
receiver sensor, wherein the retrieving means comprises a
processing circuit, wherein the calculating means comprising the
processing circuit, and wherein the controlling means comprises a
feedback system.
29. The structured light system of claim 27, further comprising:
means for determining a code domain statistic comparing the sensed
codewords with the expected codewords; and means for controlling
output power of the laser system based at least in part on the
determined code domain statistic.
30. A non-transitory computer-readable medium storing instructions
that when executed cause a processor to perform a method of
controlling output power of a laser system of a structured light
system, the method comprising:: storing a depth map with a memory
device; projecting codewords with the laser system; sensing the
projected codewords reflected from an object with a receiver
sensor; retrieving a portion of the depth map from the memory
device; calculating expected codewords from the depth map; and
controlling output power of the laser system based on the sensed
codewords and the expected codewords.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit of U.S. Provisional
Patent Application No. 62/212,507, filed on Aug. 31, 2015, entitled
"CODE DOMAIN POWER CONTROLS FOR STRUCTURED LIGHT," the content of
which is hereby incorporated by reference herein in its
entirety.
BACKGROUND
Field
[0002] Various features relate to active depth sensing, and more
specifically to controlling output power of a structured light
codeword transmitter using code domain statistics.
Description of the Related Art
[0003] Imaging devices that are structured light active sensing
systems include a transmitter and a receiver configured to transmit
and receive patterns corresponding to spatial codes (or
"codewords") to generate a depth map that indicates the distance of
one or more objects in a scene from the imaging device. The farther
away an object in a scene is from the transmitter and the receiver,
the closer a received codeword reflected from the object is from
its original position (compared to the transmitted codeword)
because a propagation path of the outgoing codeword and the
reflected incoming codeword are more parallel. Conversely, the
closer the object is to the transmitter and receiver, the farther
the received codeword is from its original position in the
transmitted codeword. Accordingly, the difference between the
position of a received codeword and the corresponding transmitted
codeword may be used to determine the depth of an object in a
scene. Structured light active sensing systems may use these
determined depths to generate a depth map of a scene, which may be
a three dimensional representation of the scene. Many applications
may benefit from determining a depth map of a scene, including
image quality enhancement and computer vision techniques.
[0004] Each codeword may be represented by rows and columns of
intensity values corresponding to symbols. For example, binary
spatial codes may use zeros (0's) and ones (1's), corresponding to
dark and bright intensity values, to represent a binary pattern.
Other spatial codes may use more than two different intensity
values corresponding to more than two symbols. Other spatial
representations also may be used.
[0005] Generating a depth map depends on detecting codewords. To
detect codewords made up of an array of symbols, decoding filters
may identify spatial boundaries for codewords and symbols, and
classify symbols as, for example, "0" or "1" based on their
intensity values. Decoding filters may use matched filters,
corresponding to the set of harmonic basis functions used to define
the set of possible codewords, to classify incoming basis
functions. Therefore, depth map accuracy depends on accurately
receiving symbols, codewords, and/or basis functions.
[0006] If the power level of a light source used to project a
pattern (for example, a laser) is too low, then the spots
corresponding to brighter symbols may be too dark to be
differentiated from darker symbols. If the power level of the light
source is too high, then the spots corresponding to brighter
symbols may become saturated and bleed into (blend in with)
neighboring spots. When this happens, it may be difficult to
accurately classify symbols, codewords, and basis functions.
Optimal power level ranges may depend at least partially on object
depth and surface reflectivity. Optimal power levels may vary both
within scenes and between scenes.
[0007] Existing methods and systems to control light source power
may not account for local variation, and may not be optimized to
maximize symbol, codeword, or basis function accuracy. Accordingly,
there is a need for methods and systems to control light source
power for structured light systems for more accurate depth map
generation.
SUMMARY
[0008] A summary of sample aspects of the disclosure follows. For
convenience, one or more aspects of the disclosure may be referred
to herein simply as "some aspects."
[0009] Methods and apparatuses or devices being disclosed herein
each have several aspects, no single one of which is solely
responsible for its desirable attributes. Without limiting the
scope of this disclosure, for example, as expressed by the claims
which follow, its more prominent features will now be discussed
briefly. After considering this discussion, and particularly after
reading the section entitled "Detailed Description" one will
understand how the features being described provide advantages that
include efficient ways to control output power of a structured
light codeword transmitter using code domain statistics resulting
in fewer decoding errors.
[0010] One innovation is a structured light system. The structured
light system may include a memory device configured to store a
depth map. The structured light system may further include an image
projecting device including a laser system configured to project
codewords. The structured light system may further include a
receiver device including a sensor, the receiver device configured
to sense the projected codewords reflected from an object. The
structured light system may further include a processing circuit
configured to retrieve at least a portion of a depth map stored in
the memory device and calculate expected codewords from the depth
map. The structured light system may further include a feedback
system configured to control the output power of the laser system
based on the sensed codewords and the expected codewords.
[0011] For some implementations, the processing circuit is further
configured to update the depth map based on the sensed codewords.
For some implementations, the memory device is further configured
to store the updated depth map.
[0012] For some implementations, the feedback system is configured
to determine a code domain statistic comparing the sensed codewords
with the expected codewords. For some implementations, the feedback
system controls the output of the laser system based at least in
part on the determined code domain statistic. For some
implementations, the code domain statistic quantifies symbol
classification accuracy. For some implementations, the code domain
statistic is the square of the difference in intensity means
divided by the sum of the intensity variances.
[0013] For some implementations, the processing circuit is further
configured to calculate expected received symbols from the depth
map and/or previously received codewords. For some implementations,
the processing circuit is further configured to assign each
received intensity value to a corresponding expected received
symbol. For some implementations, the processing circuit is further
configured to calculate a mean intensity value for each symbol. For
some implementations, the processing circuit is further configured
to calculate a variance intensity value for each symbol. For some
implementations, the processing circuit is further configured to
calculate the code domain statistic as the square of the difference
in intensity means divided by the sum of the intensity
variances.
[0014] For some implementations, the code domain statistic
quantifies codeword detection accuracy. For some implementations,
the code domain statistic is the percentage of received codewords
that match expected codewords.
[0015] For some implementations, the processing circuit is further
configured to compare received codewords to expected codewords. For
some implementations, the processing circuit is further configured
to calculate the percentage of correctly received codewords. For
some implementations, correctly received codewords correspond to
expected codewords.
[0016] For some implementations, the code domain statistic
quantifies basis function accuracy. For some implementations, the
code domain statistic is the percentage of received basis functions
that match expected basis functions.
[0017] For some implementations, the processing circuit is
configured to calculate expected basis functions from the depth map
and/or previously received codewords. For some implementations, the
processing circuit is configured to compare received basis
functions to expected basis functions. For some implementations,
the processing circuit is further configured to calculate the
percentage of correctly received basis functions. For some
implementations, correctly received basis functions correspond to
expected basis functions.
[0018] For some implementations, the feedback system controls the
output power of the laser system iteratively to converge to a
maximum value for the code domain statistic.
[0019] Another innovation is a method of controlling laser power in
a structured light system. The method may include storing a depth
map with a memory device. The method may include projecting
codewords with a laser system. The method may include sensing the
projected codewords reflected from an object with a receiver
sensor. The method may include retrieving a portion of the depth
map from the memory device. The method may include calculating
expected codewords from the depth map. The method may include
controlling output power of the laser system based on the sensed
codewords and the expected codewords.
[0020] In various embodiments, the method may further include
updating the depth map based on the sensed codewords. In various
embodiments, the method may further include storing the updated
depth map with the memory device.
[0021] In various embodiments, the method may further include
determining a code domain statistic comparing the sensed codewords
with the expected codewords. In various embodiments, the method may
further include controlling output power of the laser system based
at least in part on the determined code domain statistic.
[0022] In various embodiments, the method may further include
calculating expected received symbols from the depth map and/or
previously received codewords. In various embodiments, the method
may further include assigning each received intensity value to a
corresponding expected received symbol. In various embodiments, the
method may further include calculating a mean intensity value for
each symbol. In various embodiments, the method may further include
calculating a variance intensity value for each symbol. In various
embodiments, the method may further include calculating the code
domain statistic as the square of the difference in intensity means
divided by the sum of the intensity variances.
[0023] In various embodiments, the method may further include
comparing received codewords to expected codewords. In various
embodiments, the method may further include calculating the
percentage of correctly received codewords, wherein correctly
received codewords correspond to expected codewords.
[0024] In various embodiments, the method may further include
calculating expected basis functions from the depth map and/or
previously received codewords. In various embodiments, the method
may further include comparing received basis functions to expected
basis functions. In various embodiments, the method may further
include calculating the percentage of correctly received basis
functions, wherein correctly received basis functions correspond to
expected basis functions.
[0025] In various embodiments, the method may further include
controlling the output power of the laser system iteratively to
converge to a maximum value for the code domain statistic.
[0026] Another innovation is a structured light system. The
structured light system may include means for storing a depth map.
The structured light system may include means for projecting
codewords. The structured light system may include means for
sensing the projected codewords reflected from an object. The
structured light system may include means for retrieving a portion
of the depth map from the means for storing a depth map. The
structured light system may include means for calculating expected
codewords from the depth map. The structured light system may
include means for controlling output power of the projecting means
based on a comparison between the sensed codewords and the expected
codewords.
[0027] In various embodiments, the storing means may include a
memory device. In various embodiments, the project means may
include a laser system. In various embodiments, the sensing means
includes a receiver sensor. In various embodiments, the retrieving
means includes a processing circuit. In various embodiments, the
calculating means includes the processing circuit. In various
embodiments, the controlling means includes a feedback system.
[0028] In various embodiments, the structured light system further
includes means for determining a code domain statistic comparing
the sensed codewords with the expected codewords. In various
embodiments, the structured light system further includes means for
controlling output power of the laser system based at least in part
on the determined code domain statistic.
[0029] Another innovation is a non-transitory computer-readable
medium storing instructions that, when executed, cause a processor
to perform a method. The method may include storing a depth map
with a memory device. The method may include projecting codewords
with a laser system. The method may include sensing the projected
codewords reflected from an object with a receiver sensor. The
method may include retrieving a portion of the depth map from the
memory device. The method may include calculating expected
codewords from the depth map. The method may include controlling
output power of the laser system based on the sensed codewords and
the expected codewords.
BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Various features, aspects and advantages will become
apparent from the description herein and drawings appended hereto,
in which like reference symbols generally will identify
corresponding aspects or components illustrated in the drawings. As
a person of ordinary skill in the art will understand, aspects
described or illustrated for an embodiment may be included in one
or more other described or illustrated embodiments, if not
impractical for the implementation or function of such an
embodiment, unless otherwise stated.
[0031] FIG. 1 is a schematic illustrating an example of an active
sensing system where a known pattern is used to illuminate a scene
and obtain depth information with which to generate
three-dimensional (3D) information from two-dimensional (2D) images
and/or information.
[0032] FIG. 2 is a diagram illustrating another example of a system
for active sensing where a 3D scene is constructed from 2D images
or information.
[0033] FIG. 3 is a schematic illustrating how depth may be sensed
for an object or scene.
[0034] FIG. 4 is a block diagram illustrating an example of a
transmitter device that may be configured to generate a composite
code mask and/or project such composite code mask.
[0035] FIG. 5 is a block diagram illustrating an example of a
receiver device that may be configured to obtain depth information
from a composite code mask.
[0036] FIG. 6 is a block diagram of one embodiment of an apparatus
configured to perform one or more of the error correction methods
disclosed herein.
[0037] FIG. 7 is a picture illustrating an example of a code mask
with arrays of symbols corresponding to bright and dark spots.
[0038] FIG. 8 is a picture illustrating an image of a scene used to
generate a depth map, superimposed with codewords projected by a
laser through a code mask, such as the code mask of FIG. 7.
[0039] FIG. 9 illustrates an example of a depth map for the scene
of FIG. 8.
[0040] FIG. 10 illustrates an example of a codeword illuminated at
an optimal power level. The codeword includes a 4.times.4 array of
"0" or "1" symbols, corresponding to symbols encoded in the code
mask of FIG. 7, the symbols having well defined boundaries and
clear separation in intensity values between the "0" symbols and
the "1."
[0041] FIG. 11 shows an example of well separated probability
distribution functions of intensity values for "0" and "1" symbols
encoded in the code mask of FIG. 7, at optimum laser power, as
described in FIG. 10.
[0042] FIG. 12 illustrates an example of the codeword of FIG. 10
but illuminated at a lower power level than in FIG. 10, such that
the bright spots are not as bright as in FIG. 10.
[0043] FIG. 13 shows an example of overlapping probability
distribution functions of intensity values by symbol for "0" and
"1" symbols encoded in the code mask of FIG. 7, at a lower than
optimum power level, as described in reference to FIG. 12.
[0044] FIG. 14 illustrates an example of the codeword of FIG. 10
but illuminated at a higher power level than in FIG. 10, such that
the bright spots are saturated, bleed into each other, and cause
some dark spots to appear bright.
[0045] FIG. 15 shows an example of overlapping probability
distribution functions of intensity values by symbol for "0" and
"1" symbols encoded in the code mask of FIG. 7, at higher than
optimum laser power, as described in FIG. 14.
[0046] FIG. 16 illustrates an example of a graph of contrast or
separation between the two symbols as a function of power
level.
[0047] FIG. 17 illustrates structured light images taken at three
different power settings in an example embodiment.
[0048] FIG. 18 illustrates an example of a feedback control system
that can be used to control a laser in a structured light system to
project codewords that are neither too dark to be sensed and
distinguished, nor too bright to be saturated when sensed by a
receiver in the structured light system.
[0049] FIG. 19 illustrates an example of a process 1900 for
adjusting the power of a structured light transmitter using code
domain statistics based existing depth map information and sensed,
reflected, images of codewords.
[0050] FIG. 20 illustrates an example of a process 2000 for
controlling the power of a structured light transmitter using
information determined from received codewords that are reflected
from an object, including calculation of a code domain
statistic.
[0051] FIG. 21 illustrates an example of a process 2100 for
calculating the code domain statistic of process 2000, wherein the
code domain statistic is the contrast between symbols as measured
by the square of the difference in intensity means divided by the
sum of intensity variances.
[0052] FIG. 22 illustrates an example of a process 2200 for
calculating the code domain statistic of process 2000, wherein the
code domain statistic is the percentage of received codewords that
match the expected codewords.
[0053] FIG. 23 illustrates an example of a process 2300 for
calculating the code domain statistic of process 2000, wherein the
code domain statistic is the percentage of received basis functions
that match the expected basis functions.
DETAILED DESCRIPTION
[0054] The following detailed description is directed to certain
specific embodiments. However, the methods and systems disclosed
can be embodied in a multitude of different ways. It should be
apparent that aspects herein may be embodied in a wide variety of
forms and that any specific structure, function, or both being
disclosed herein is merely representative. Aspects disclosed herein
may be implemented independently of any other aspects. Two or more
of these aspects may be combined in various ways. For example, an
apparatus may be implemented, or a method may be practiced, using
any number of the aspects set forth herein. In addition, such an
apparatus may be implemented or such a method may be practiced
using other structure, functionality, or structure and
functionality in addition to or other than one or more of the
aspects set forth herein.
[0055] Further, the systems and methods described herein may be
implemented on a variety of different imaging systems and computing
devices and systems. They may use general purpose or special
purpose systems.
[0056] Structured light systems generate depth maps by decoding
received patterns of codewords and comparing them to the
transmitted patterns. If the received symbols and codewords have
well defined spatial boundaries and well separated intensity levels
for different symbol values, then it is possible to decode the
received patterns accurately and generate accurate depth maps. If
the symbol boundaries are not well defined, and/or the intensity
levels are not well separated, then detection accuracy goes down
and depth map accuracy suffers.
[0057] Structured light systems transmit the patterns of codewords
by emitting light at a controllable power level through a mask. In
some embodiments, the light source is a laser (although it may also
be another type of light source), and the mask is a diffractive
optical element. When the power level of the light source is too
low, the symbols may be too dark to be received accurately and
correctly decoded. At a higher power level of the light source, the
transmitted symbols can be more likely to be decoded because their
boundaries are well delineated and, with increasing power, well
separated by intensity. For example "0" symbols appear dark, and
"1" symbols appear light, and there is a large intensity gap
between the dark symbols and the light symbols. If the power level
of the light source is too high, then the symbols may appear to
bleed beyond the intended boundaries of the symbols into guard
bands and even into neighboring symbols. Therefore, when a power
level of the light source is too high, the symbol boundaries may be
unclear as symbols may blend with each other, and the received
symbols may appear significantly different than what was projected,
reducing detection accuracy. In any particular scene, objects at
different distances and/or having different surface characteristics
may require different laser power levels for accurate decoding.
[0058] The disclosed technology includes systems and methods to
control the light source power level, so that the received images
can be decoded accurately. Code domain statistics are used to
characterize how effectively received images can be decoded, by,
for example, quantifying contrast or separation among different
symbol values, quantifying codeword detection accuracy, or
quantifying basis function detection accuracy. These measures
directly characterize decoding accuracy, and enable control
convergence to an optimal laser power level by feeding back (for
example, via a negative feedback loop, or using an adaptive
algorithm) the code domain statistic to the laser controller. As a
result, the resulting depth map may have less decoding errors and
thus be more accurate.
[0059] FIG. 1 illustrates an example of an active sensing system
100 that generates three dimensional information, such as a depth
map 107, from two dimensional images. The active sensing system 100
includes a transmitter 102 and a receiver 108. The transmitter 102
projects light through a code mask to form a projected image 104. A
section 112 of projected image 104 includes a unique codeword 120
that is projected onto the scene 106. The surface of object(s) in
scene 106 is illuminated by spatial pattern 116, which forms part
of reflected image that is sensed by receiver 108. Receiver 108
senses a portion 118 (segment) of the reflected image 110,
including unique codeword 120, and compares the relative position
of unique codeword 120 to other unique codewords in the code mask
to determine depth information, for generating a depth map 107, of
the surface of object in scene 106, as described below with regard
to FIG. 3. The receiver 108 forms a depth map 107 based on depth
estimates over the surfaces of the objects in the scene, which
reflect other identifiable codewords from other segments of
reflected image 110. Each segment 118 that is captured may be
uniquely identifiable at the receiver 108 and its location relative
to other segments ascertained from the known pattern of the coded
mask. The receiver 108 may use pattern segmentation techniques to
address distortion, decoding techniques to identify codes, and
triangulation to ascertain orientation and/or depth.
[0060] FIG. 2 illustrates another example of a system for active
sensing to generate depth maps and display three dimensional
representations of scenes. An encoder/shape modulator 201 may
generate a code mask which is then projected by a transmitter
device 202 over a transmission channel 204. The code mask may be
projected onto a target (e.g., a scene) and the reflected light is
captured by a receiver sensor 205 as a projected code mask image.
The receiver sensor 205 (e.g., receiver 108 in FIG. 1), captures
the reflected image of the target, which segmentation/decoder 206
segments and decodes to determine depth information used to
generate depth map 208. The depth map 208 may then be used to
present, generate, and/or provide a 3D image version of a target,
for example, one of targets 210a-e.
[0061] Active sensing relies on being able to recognize (at the
receiver sensor 205 and/or segmentation/decoder 206) spatial codes
(e.g., codewords) from the code mask being projected by the
transmitter device 202 on a scene. If a scene is too close to the
transmitter and receiver, the surface of the scene may be angled or
curved, a baseline reference plane may be tilted, and the codes may
be modified under an unknown affine transformation (e.g., rotation,
skew, compression, elongation, etc.). One or more aspects or
features described herein may be implemented within the exemplary
environments of FIGS. 1 and 2
[0062] FIG. 3 illustrates an example of how depth may be sensed for
one or more objects in a scene. FIG. 3 shows a device 300 including
a transmitter 302 and a receiver 304. The device is illuminating
two objects 306 and 308 with structured light emitted from
transmitter 302 as codeword projection 310. The codeword projection
310 reflects from objects 306 and/or 308 and is received as a
reflected codeword 311 by receiver 304 on sensor plane 307.
[0063] As illustrated in FIG. 3, the transmitter 302 is on the same
reference plane as the receiver 304 (e.g., lens plane 305). The
transmitter 302 projects the codeword projection 310 onto the
objects 306 and 308 through an aperture 313.
[0064] The codeword projection 310 illuminates the object 306 as
projected segment 312', and illuminates the object 308 as projected
segment 312''. When the projected segments 312' and 312'' are
received by the receiver 304 through receiver aperture 315, the
reflected codeword 311 may show reflections generated from the
object 308 at a first distance d1 and reflections generated from
the object 306 at a second distance d2.
[0065] As shown by FIG. 3, since the object 306 is located closer
to the transmitter 302 (e.g., a first distance from the transmitter
device) the projected segment 312' appears at a distance d2 from
its initial location. In contrast, since the object 308 is located
further away (e.g., a second distance from the transmitter 302),
the projected segment 312'' appears at a distance d1 from its
initial location (where d1<d2). That is, the further away an
object is from the transmitter/receiver, the closer the received
projected segment/portion/window is from its original position at
the receiver 304 (e.g., the outgoing projection and incoming
projection are more parallel). Conversely, the closer an object is
from the transmitter/receiver, the further the received projected
segment/portion/window is from its original position at the
receiver 304. Thus, the difference between received and transmitted
codeword position may be used as an indicator of the depth of an
object. In one example, such depth (e.g., relative depth) may
provide a depth value for objects depicted by each pixel or grouped
pixels (e.g., regions of two or more pixels) in an image.
[0066] Various types of modulation and coding schemes may be used
to generate a codeword projection or code mask. These modulation
and coding schemes include, for example, temporal coding, spatial
coding, and direct codification.
[0067] In temporal coding, patterns are successively projected onto
the measuring surface over time. This technique has high accuracy
and resolution but is less suitable for dynamic scenes.
[0068] In spatial coding, information is encoded in a local
neighborhood based on shapes and patterns. Pseudorandom codes may
be based on De-Bruijn or M-arrays define the codebook of valid
codewords (e.g., m-ary intensity or color modulation). Pattern
segmentation may not be easily attained, for example, where the
shapes and patterns are distorted.
[0069] In direct codification, both horizontal and vertical pixel
coordinates are encoded. Modulation may be by a monotonic phase or
an intensity waveform. However, this scheme may utilize a codebook
that is larger than the codebook utilized for other methods. In
most methods, received codewords (sensed codewords) may be
correlated against a defined set of possible codewords (e.g., in a
codebook). Thus, use of a small set of codewords (e.g., small
codebook) may provide better performance than a larger codebook.
Also, since a larger codebook results in smaller distances between
codewords, additional errors may be experienced by implementations
using larger codebooks.
[0070] Structured light patterns may be projected onto a scene by
shining light through a codemask. Light projected through the
codemask may contain one or more tessellated codemask primitives.
Each codemask primitive may contain an array of spatial codes. A
codebook or data structure may include the set of codes. Spatial
codes, the codemask, and codemask primitives may be generated using
basis functions. The periodicities of the basis functions may be
chosen to meet the requirements for the aggregate pattern of
Hermitian symmetry (for eliminating ghost images and simplifying
manufacturing), minimum duty cycle (to ensure a minimum power per
codeword), perfect window property (for optimum contour resolution
and code packing for high resolution), and randomized shifting (for
improved detection on object boundaries). A receiver may make use
of the codebook and/or the attributes of the design intended to
conform to the constraints when demodulating, decoding, and
correcting errors in received patterns.
[0071] The size and corresponding resolution of the spatial codes
corresponds to a physical spatial extent of a spatial code on a
codemask. Size may correspond to the number of rows and columns in
a matrix that represents each codeword. The smaller a codeword, the
smaller an object that can be detected. For example, to detect and
determine a depth difference between a button on a shirt and the
shirt fabric, the codeword should be no larger than the size of the
button. In some embodiments, each spatial code may occupy four rows
and four columns. In some embodiments, the codes may occupy more or
fewer rows and columns (rows.times.columns), to occupy, for
example, 3.times.3, 4.times.4, 4.times.5, 5.times.5, 6.times.4, or
10.times.10 rows and columns.
[0072] The spatial representation of spatial codes corresponds to
how each codeword element is patterned on the codemask and then
projected onto a scene. For example, each codeword element may be
represented using one or more dots, one or more line segments, one
or more grids, some other shape, or some combination thereof.
[0073] The "duty cycle" of spatial codes corresponds to a ratio of
a number of asserted bits or portions (e.g., "1s") to a number of
un-asserted bits or portions (e.g., "0s") in the codeword. When a
coded light pattern including the codeword is projected onto a
scene, each bit or portion that has a value of "1" may have energy
(e.g., "light energy"), but each bit having a value of "0" may be
devoid of energy. For a codeword to be easily detectable, the
codeword should have sufficient energy.
[0074] The "contour resolution" or "perfect window" characteristic
of codes indicates that when a codeword is shifted by an amount,
for example, a one-bit rotation, the resulting data represents
another codeword.
[0075] FIG. 4 is a block diagram illustrating an example of a
transmitter device that may be configured to generate a composite
code mask and/or project such composite code mask. The transmitter
device 402 may include a processing circuit 404 coupled to a
memory/storage device 406 (memory device), an image projecting
device 408, and/or a tangible medium 409. In some aspects, the
transmitter device 402 may correspond to the transmitter 302
discussed above with respect to FIG. 3.
[0076] In a first example, the transmitter device 402 may be
coupled to include a tangible medium 409. The tangible medium may
define, include, and/or store a composite code mask 414. The
tangible medium may be a diffractive optical element (DOE) that
encodes the code mask, such that when light from a laser or other
light source is projected through the DOE at, for example, a near
infrared frequency, a codeword pattern image is projected from the
transmitter. The composite code mask may include a code layer
combined with a carrier layer. The code layer may include uniquely
identifiable spatially-coded codewords defined by a plurality of
symbols. The carrier layer may be independently ascertainable and
distinct from the code layer. The carrier layer may include a
plurality of reference objects that are robust to distortion upon
projection. At least one of the code layer and carrier layer may be
pre-shaped by a synthetic point spread function prior to
projection.
[0077] In a second example, the processing circuit (or processor)
404 may include a code layer generator/selector 416, a carrier
layer generator/selector 418, a composite code mask
generator/selector 420 and/or a pre-shaping circuit 422. The code
layer generator/selector 416 may select a pre-stored code layer 410
and/or may generate such code layer. The carrier layer
generator/selector 418 may select a pre-stored carrier layer 412
and/or may generate such carrier layer. The composite code mask
generator/selector may select a pre-stored composite code mask 414
and/or may combine the code layer 410 and carrier layer 412 to
generate the composite code mask 414. Optionally, the processing
circuit 404 may include a pre-shaping circuit 422 that pre-shapes
the composite code mask 414, the code layer 410, and/or the carrier
layer 412, to compensate for expected distortion in the channel
through which the composite code mask is to be projected.
[0078] In some implementations, a plurality of different code
layers and/or carrier layers may be available, where each such
carrier or code layers may be configured for different conditions
(e.g., for objects at different distances, or different
configurations between the transmitter device and receiver device).
For instance, for objects within a first distance or range, a
different combination of code and carrier layers may be used than
for objects at a second distance or range, where the second
distance is greater than the first distance. In another example,
different combination of code and carrier layers may be used
depending on the relative orientation of the transmitter device and
receiver device.
[0079] The image projecting device 408 may serve to project the
generated/selected composite code mask onto an object of interest.
For instance, a laser or other light source may be used to project
the composite code mask onto the object of interest (e.g., through
a projection channel). In one example, the composite code mask 414
may be projected in an infrared spectrum, so it may not be visible
to the naked eye. Instead, a receiver sensor in the infrared
spectrum range may be used to capture such projected composite code
mask.
[0080] FIG. 5 is a block diagram illustrating an example of a
receiver device 502 that is configured to receive a composite code
mask reflected from an object and to determine be depth information
from a composite code mask. The receiver device 502 may include a
processing circuit 504 coupled to a memory/storage device and a
receiver sensor 508 (e.g., an image capturing device 508). In some
aspects, the receiver device 502 illustrated in FIG. 5 may
correspond to the receiver 304 discussed above with respect to FIG.
3. In some embodiments, the receiver sensor 508 is an image capture
device, for example, a camera.
[0081] The receiver sensor 508 may be configured to obtain at least
a portion of a composite code mask projected on the surface of an
object. For instance, the receiver sensor may capture an image of
at least a portion of a composite code mask 414 projected on the
surface of a target object. The composite code mask 414 may be
defined by: (a) a code layer of uniquely identifiable
spatially-coded codewords defined by a plurality of symbols, and
(b) a carrier layer independently ascertainable and distinct from
the code layer and including a plurality of reference objects that
are robust to distortion upon projection. At least one of the code
layer and carrier layer may have been pre-shaped by a synthetic
point spread function prior to projection. In one example, the
receiver sensor 508 may capture (sense) the composite code mask in
the infrared spectrum.
[0082] Still referring to FIG. 5, in some embodiments, the code
layer may comprise n1 by n2 binary symbols, where n1 and n2 are
integers greater than two. In the composite code mask, each symbol
may be a line segment in one of two gray-scale shades distinct from
the reference objects. The symbols of the code layer may be
staggered in at least one dimension. The carrier layer reference
objects may comprise a plurality of equally spaced reference
stripes with a guard interval in between. The reference stripes and
the guard interval may be of different widths. The width of each
reference stripe relative to a guard interval width may be
determined by an expected optical spreading of a transmitter device
and/or a receiver device.
[0083] The processing circuit 504 may include a reference stripe
detector circuit/module 512, a distortion adjustment circuit/module
514, a codeword identifier circuit/module 516, a depth detection
circuit/module 518, and/or a depth map generation circuit/module
520.
[0084] The reference stripe detector circuit/module 512 may be
configured to detect reference stripes within the portion of the
composite code mask. The distortion adjustment circuit/module 514
may be configured to adjust a distortion of the portion of the
composite code mask based on an expected orientation of the
reference stripes relative to an actual orientation of the
reference stripes. The codeword identifier circuit/module 516 may
be configured to obtain a codeword from a window defined within the
portion of the composite code mask. The depth detection
circuit/module 518 may be configured to obtain depth information
for a surface portion of the target object corresponding to the
window based on: (a) a single projection of the composite code
mask, and (b) a displacement of the window relative to a known
reference code mask.
[0085] The depth map generation circuit/module 520 may be
configured to assemble a depth map for the object based on a
plurality of codewords detected as different overlapping windows
within the portion of the undistorted composite code mask.
[0086] FIG. 6 is a block diagram illustrating an embodiment of an
apparatus configured to perform one or more of the error correction
methods disclosed herein. Apparatus 600 includes a light emitter
602, a light receiving element 604, a processor 606, and a memory
608. The light emitter 602, light receiving element 604, processor
606, and the memory 608 are operably connected via a bus 610. In
some aspects, the light receiving element 604 may correspond to the
receiver device 502 discussed above with respect to FIG. 5. In some
aspects, the light emitter 602 may correspond to the transmitter
device 402 discussed above with respect to FIG. 4.
[0087] The memory 608 may store instructions that configure the
processor 606 to perform one or more functions of the methods
discussed herein. For example, instructions stored in the memory
may configure the processor 606 to control the light emitter 602 to
emit light that encodes structured light as codewords, in order to
illuminate a target object. Instructions stored in the memory 608
may further cause the processor 606 to control the light receiving
element 604 to receive light reflecting from the target object and
produce data encoded in the reflected light. Instructions stored in
the memory may further configure the processor to correct errors in
the data produced by the light receiving element according to the
method 1500 discussed below.
[0088] FIG. 7 is a picture of an example of a code mask 700 with
arrays of symbols corresponding to bright and dark spots. The
bright spots correspond to "1" symbols. They are aligned in rows
and columns, and separated by black guard intervals and guard bands
that give structure to the projected codes and make it possible to
determine spatial boundaries of individual symbols and codewords.
Codewords occupy a rectangular spatial area that includes rows and
columns of symbols. For example, a codeword may include sixteen
symbols in four rows and four columns. The "1" symbols with bright
spots are visible, but the "0" symbols with dark spots blend into
the guard intervals and guard bands.
[0089] FIG. 8 is a picture 800 of an image of a scene used to
generate a depth map superimposed with codewords projected by a
laser through a code mask, such as the code mask of FIG. 7. The
image of FIG. 8 includes a snowflake in the foreground, followed at
increasing depths by an open hand to the right of the snowflake, a
sculpted head in profile, a closed hand with a thumbs up, and
farthest back an open hand to the left of the hand with a thumbs
up. As the codewords may be received in a non-visible portion of
the frequency spectrum, the superimposed codeword projection of
FIG. 8 is printed in false color so that it is visible in FIG.
8.
[0090] FIG. 9 illustrates an example of a depth map 900 for the
scene of FIG. 8, determined using the structured light techniques
described above. The points of minimum and maximum depth are each
indicated in FIG. 9. In some embodiments, the depth map 900 is
continuously updated at video rate (for example, at 30 frames per
second). The existing depth map, generated from previously received
frames, provides the set of expected depths for each new image
frame. The depth at each location in an image corresponds to the
expected depth at each location, and may be used to determine the
expected codeword, symbol, and basis function at that location.
[0091] FIG. 10 illustrates an example of a codeword projected at an
optimal power level. The codeword includes a 4.times.4 array of
symbols 1000, corresponding to symbols encoded in the code mask of
FIG. 7. The symbols in FIG. 10 have well defined boundaries and
clear separation in intensity values between the "0" and "1"
symbols. Symbol 1010, as well as the other three symbols depicted
as a circle with a black border and no shading, correspond to dark
"0" symbol dots. Symbol 1020, as well as the other 11 symbols each
depicted as a circle with dark hatching, correspond to bright "1"
symbol dots. In the example of FIG. 10, the power level is optimal.
Each symbol in FIG. 10 has a clear boundary with no saturation or
bleeding over the edges. There is clear intensity separation
between the dark dots 1010 and bright dots 1020, as further
illustrated in FIG. 11.
[0092] FIG. 11 shows an example of well separated probability
distribution functions 1110 and 1120 of intensity values for the
"0" and "1" symbols, respectively, encoded in the code mask of FIG.
7, at an optimal laser power as described in FIG. 10. The
horizontal and vertical axes of FIG. 11 correspond to intensity
level and probability level, respectively. The probability
distribution function 1110 for the dark "0" symbols approximates a
Gaussian distribution with a peak at the mean intensity level of
.mu..sub.0 and a standard deviation of .sigma..sub.0. The
corresponding variance is .sigma..sub.0.sup.2. Similarly, the
probability distribution function 1120 for the bright "1" symbols
approximates a Gaussian distribution with mean .mu..sub.1 and a
standard deviation of .sigma..sub.1. In this example,
.mu..sub.1>.mu..sub.0 and .sigma..sub.1>.sigma..sub.0. The
two probability distribution functions 1110 and 1120 are well
separated but do overlap.
[0093] Received intensity levels to the left of the decision
boundary 1130 are more likely to be "0" symbols than "1" symbols.
The probability is equal at the decision boundary 1130 where the
two probability distribution functions 1110 and 1120 cross, with
equal probability values. Received intensity levels to the right of
the decision boundary 1130 are more likely to be "1" symbols than
"0" symbols. Therefore, "0" symbols with intensity values to the
left of the decision boundary 1130 will be correctly classified,
while those to the right of the decision boundary, corresponding to
the right tail 1150 will be incorrectly classified as "1"
symbols.
[0094] Similarly, "1" symbols with intensity values to the right of
the decision boundary 1130 will be correctly classified, while
those to the left of the decision boundary, corresponding to the
left tail 1140 will be incorrectly classified as "0" symbols.
Accordingly, less separation corresponds to fewer symbol
classification errors.
[0095] In the example of FIG. 11, the left tail 1140 and right tail
1150 are small because the difference in means of probability
distribution functions 1110 and 1120 is relatively large when
normalized by the sum of their variances. This relationship may be
quantified a code domain statistic that measures contrast, or
between cluster to within cluster variation, as defined in equation
(1), below. Better separation between symbol intensity levels
corresponds to higher values for .LAMBDA..
.LAMBDA. = ( .mu. 1 - .mu. 0 ) 2 .sigma. 0 2 + .sigma. 1 2 ( 1 )
##EQU00001##
[0096] FIG. 12 illustrates an example of the codeword of FIG. 10
but illuminated at lower power level, such that the bright spots
are not as bright as in FIG. 10, as shown by the lighter hatching
in symbol 1220. The symbols in FIG. 12 have well defined
boundaries, but there is less clear separation in intensity values
between the "0" and "1" symbols. Symbol 1210, as well as the other
three symbols depicted as a circle with a black border and no
shading, correspond to dark "0" symbol dots. Symbol 1220, as well
as the other 11 symbols each depicted as circle with light
hatching, correspond to bright "1" symbol dots. In the example of
FIG. 10, the power level lower than optimal, the bright "1" symbols
1220 are not as bright as the bright "1" symbols 1020, resulting in
less intensity separation between the dark dots and bright
dots.
[0097] FIG. 13 shows an example of overlapping probability
distribution functions of intensity values by symbol for "0" and
"1" symbols encoded in the code mask of FIG. 7, at lower than
optimum laser power as described in FIG. 12. When compared with
FIG. 10, the probability distribution functions 1310 and 1320 for
the dark "0" symbols and bright "1" symbols, respectively overlap
more because the bright spots are not as bright with the lower
laser power level. The decision boundary 1330 is at a lower
intensity value than the decision boundary 1130. Right tail 1350 of
probability distribution function 1310 is significantly larger in
area than right tail 1150. Similarly, Left tail 1340 is
significantly larger in area than left tail 1140. With the greater
degree of overlap between probability distribution functions 1310
and 1320, with less than optimal laser power levels, than was
apparent for probability distribution functions 1110 and 1120, the
contrast statistic .LAMBDA. is lower for less than optimal power
than it was for optimal power.
[0098] FIG. 14 illustrates an example of the codeword of FIG. 10
but illuminated at a higher power level, so that the bright spots
are saturated, bleed into each other, and cause some dark spots to
appear bright. The symbols in FIG. 12 no longer have well defined
boundaries that correspond to the transmitted boundaries because
the saturated bright "1" bits 1420 bleed into guard bands, guard
intervals, and may even overlap neighboring "0" symbols 1410. This
may result it average intensity values for "0" symbol values to
increase, the variance of dark "0" symbol values to increase,
resulting in less intensity separation between the bright dots and
the dark dots.
[0099] FIG. 15 shows an example of overlapping probability
distribution functions of intensity values by symbol for "0" and
"1" symbols encoded in the code mask of FIG. 7, at higher than
optimum laser power as described in FIG. 14. When compared with
FIG. 10, the probability distribution functions 1510 and 1520 for
the dark "0" symbols and bright "1" symbols, respectively overlap
more because the dark spots appear brighter due to bleeding or
blending in of neighboring bright spots, and an increase in the
variance of the intensity values for the dark spots, as shown in
the probability distribution function 1510. The decision boundary
1530 is at a higher intensity value than the decision boundary
1130. Right tail 1550 of probability distribution function 1510 is
significantly larger in area than right tail 1150. Similarly, Left
tail 1540 is significantly larger in area than left tail 1140. With
the greater degree of overlap between probability distribution
functions 1510 and 1520, with more than optimal laser power levels
than was apparent for probability distribution functions 1110 and
1120, the contrast statistic .LAMBDA. is lower for more than
optimal power than it was for optimal power.
[0100] FIG. 16 illustrates an example of a graph 1600 of contrast
(or separation) between the two symbols as a function of power
level of a light source. The horizontal axis corresponds to a light
source power level. The vertical axis corresponds to a calculated
contrast statistic .LAMBDA.. This "contrast by power" curve 1610
has a maximum contrast statistic value 1630 at optimal power level
1620. The "contrast by power" curve 1610 has a sharp tail to the
left of the optimal power level 1620 as power decreases to the
point where it is insufficient to illuminate the bright dots so
that they can be seen, and a long tail to the right of the optimal
power level 1620 to as power increases towards saturation.
[0101] FIG. 17 illustrates examples of structured light images
taken at three different power level settings of a light source. In
a first row of original images in FIG. 17, the corresponding images
produced from a code contrast statistic in a second row, and
corresponding depth maps in a third row. The first row of original
images include an image 1705 generated using a light source at a
nominal power level within an optimal range, an image 1710
generated using a light source at a 170% power level above the
optimal range, and an image 1715 generated using a light source at
a 200% power level well even further from the optimal range. The
second row in FIG. 17 includes code contrast statistic images,
specifically an image 1720 which corresponds to original image
1705, an image 1725 which corresponds to original image 1710, and
an image 1730 which corresponds to original image 1730. The third
row in FIG. 17 includes depth maps 1735, 1740 and 1745 generated by
the original structured light images 1705, 1710, and 1715,
respectively. FIG. 17 accordingly illustrates that as power of the
light source increases from 100% and optimal to 170% of optimal and
to 200% of optimal, the depth maps provide less accurate
information. For example, the circled snowflake depths are well
defined in depth map 1735, but are less well defined in depth map
1740, and even less well defined in depth map 1745.
[0102] FIG. 18 illustrates a feedback control system 1800 that can
be used to control a laser 1820 in a structured light system to
project codewords that are neither too dark to be sensed and
distinguished, nor too bright to be saturated, when sensed by a
receiver sensor 508 in the structured light system. The feedback
control system 1800 (feedback system) includes a controller 1810
coupled to a (light source) laser 1820, a composite code mask 414,
a receiver sensor 508, a processing circuit 504, a memory/storage
device 506 (memory device), and an adder 1830. These elements are
coupled to each other to form a negative feedback loop as
illustrated in FIG. 18 to iteratively control the output of the
laser 1820 (laser system). In some embodiments, the controller
1810, laser 1820, and composite code mask 414 may be elements of a
transmitter device 402 (FIG. 4). The controller 1810 and laser 1820
housed within image projecting device 408 (FIG. 4). In some
embodiments, the receiver sensor 508, processing circuit 504, and
memory/storage device 506 may be elements of receiver device 502
(FIG. 5). The adder 1830 may be incorporated within either the
transmitter device 402 or the receiver device 502 (FIG. 5). As
noted above, the transmitter device 402 and receiver device 502 may
be housed within a single device. Further, the controller 1810 may
include the adder 1830; and/or the controller 1810 and processing
circuit 404 may be combined within a single element. Processing
circuit 504, adder 1830, controller 1810, and laser 1820 are
coupled to, and in electronic communication with, each other. The
receiver sensor 508, processing circuit 504, and memory/storage
device are coupled to, and in electronic communication with, each
other.
[0103] Image projecting device 408 (FIG. 4) includes a laser 1820
controlled by controller 1810. The laser 1820 emits light at, for
example, a near infrared frequency that is not visible to the human
eye but may be sensed by receiver sensor 508. The output level of
the laser 1820 can be adjusted by controller 1810. Composite code
mask 414, receiver sensor 508, processing circuit 504, and
memory/storage device 506 are described above with respect to FIGS.
4 and 5.
[0104] FIG. 19 illustrates an example of a process 1900 for
adjusting the power of a structured light transmitter using code
domain statistics, using existing depth map information (for
example, previously determined depth map, or previously determined
codewords of a scene) and sensed images of codewords that are
received from (reflected from) a scene.
[0105] At block 1905, process 1900 initializes the laser power
level. This may be performed, for example, by the image projecting
device 408 (FIG. 4), controller 1810 (FIG. 18), or the light
emitter 602 (FIG. 6). The initial setting of the laser may be set,
for example, based on a previously optimal laser power level that
was previously stored in memory, and then retrieved from memory at
block 1905 and used to set the laser power level corresponding to
an output power of the laser. The stored laser power level may be,
for example, a predetermined "factory setting" value or it may have
been previously determined during a previous use of the laser and
stored in memory
[0106] At block 1910, process 1900 generates a depth map and stores
it in memory. This may be performed, for example, by processing
circuit 504 of FIG. 5 and FIG. 18, or processor 606 of FIG. 6.
Process 1900 may use structured light methods as described with
regard to FIG. 3, in which codeword displacements are used to
generate depth information. Process 1900 may generate depth map
information from a single structured light frame, or multiple
structured light frames. The depth map information may be stored in
memory/storage device 506 of FIG. 5 and FIG. 18, or in memory 608
of FIG. 6.
[0107] Circular flowchart element 1915 is the starting point for a
depth map update cycle. For each update cycle, process 1900
converges to an optimal laser power level as described in blocks
1920-1955, and feedback path 1960. Once converged, process 1900
updates the depth map and stores it in memory in block 1965. Once
updated, process 1900 returns to circular flowchart element 1915
via path 1970 for another depth map update cycle. In some
embodiments, the laser convergence and depth map update cycle may
occur at video rates, for example, 30 or more cycles per
second.
[0108] At block 1920, process 1900 retrieves depth map information
of a scene (or of a certain area or portion of a scene) from
memory. This may be performed by processing circuit 504 of FIG. 5
and FIG. 18, or processor 606 of FIG. 6. The depth map information
may be retrieved from memory/storage device 506 of FIG. 5 and FIG.
18, or from memory 608 of FIG. 6.
[0109] At block 1925, process 1900 generates expected symbols,
codewords, and/or basis functions for the area based on the
retrieved depth map information by calculating expected symbols,
calculating expected basis functions, and/or calculating expected
codewords. This may be performed by processing circuit 504 of FIG.
5 and FIG. 18, or processor 606 of FIG. 6. Depth map 1735 is a
pictorial representation of depths as a function of location.
[0110] Each row and column in the image has a depth value that
corresponds to an "expected" depth, or distance from to the surface
of an object in the scene. As described with respect to FIG. 3,
there is a one-to-one correspondence between codeword displacement
and depth for unique codewords within an area. At each codeword
location, process 1900 calculates the codeword displacement
corresponding to the depth in the retrieved depth map. Process 1900
then translates within the code mask by the codeword displacement
to determine the expected codeword at each codeword location. By
repeating this over the area, process 1900 determines an array of
expected codewords as a function of location.
[0111] Each codeword comprises a known array of symbols. By
associating each codeword with its symbols, process 1900 determines
the corresponding set of symbols at each symbol location.
[0112] Furthermore, the codewords at each portion of the codemask
map directly to the harmonic basis functions used to generate the
codemask. By associating codewords centered at each location with
the basis functions used to generate the codemask at the
corresponding (displaced) codemask location, process 1900
determines the corresponding set of basis functions at each basis
function location.
[0113] The expected codewords, expected symbols, and expected basis
functions correspond to the codewords, symbols, and basis functions
that process 1900 decodes if the laser power is at an optimal
level, and the depth map is accurate. Therefore, these values may
be used to help converge to an optimal laser power level.
[0114] In block 1930, process 1900 projects laser light through a
codemask to project codewords onto a scene. The codemask has the
same codewords, associated symbols, and are formed by the same
harmonic basis functions as the codemask described above with
respect to block 1925. This may be performed, for example, by the
image projecting device 408 (FIG. 4), laser 1820 (FIG. 18), or the
light emitter 602 (FIG. 6). The codewords are continuously
projected for time interval. The projected codewords may be
projected onto a scene, or objects in a scene.
[0115] At block 1935, process 1900 senses a reflected image of the
codewords. This may be performed by a receiver sensor 508 of FIG. 5
and FIG. 18, or a sensor integrated with a light source for
example, light receiving element 604 integrated with a light
emitter 602 of FIG. 6. The received codewords may be received in an
image of the scene or objects in the scene.
[0116] At block 1940, process 1900 determines intensity levels of
sensed symbols, codewords, and/or basis functions for the area
based on the sensed image. This may be performed by processing
circuit 504 of FIG. 5 and FIG. 18, or processor 606 of FIG. 6.
Process 1900 performs the functions described above with respect to
FIG. 5 to delineate and detect codewords using processing circuit
504 and modules 512, 514, 516, and 518. Once process 1900
determines the received codewords, it associates each codeword with
the known (e.g., stored, pre-existing) set of symbols corresponding
to each codeword to generate the set of received codewords. In
embodiments where the codemask and codewords are generated using
harmonic basis functions, process 1900 may determine received basis
functions by applying the incoming intensity values to matched
filters, one per harmonic basis function, and determining which
matched filter has the highest output. The matched filter with the
highest output corresponds to the most likely basis function at
that location.
[0117] At block 1950, process 1900 generates at least one code
domain statistic based on the expected and sensed symbol, codeword,
and/or basis functions. This may be performed by processing circuit
504 of FIG. 5 and FIG. 18, or processor 606 of FIG. 6. A first
example of a codeword statistic characterizes symbol separation
using contrast statistic .LAMBDA. as defined in equation (1),
above, to quantify how well the received codewords can be detected.
A second codeword statistic characterizes codeword detection
accuracy by calculating the percentage of received codewords that
match their corresponding expected codewords. A third codeword
statistic characterized basis function accuracy by calculating the
percentage of received basis functions that match their
corresponding expected basis functions. Examples of certain
processes to compute contrast, codeword detection accuracy, and
basis function detection accuracy statistics are described below
with respect to FIGS. 21, 22 and 23, respectively.
[0118] At block 1950, process 1900 adjusts the laser power in
response to the at least one code domain statistic This may be
performed by processing circuit 504 of FIG. 5 and FIG. 18, or
processor 606 of FIG. 6. With reference to FIG. 18, the code domain
statistic(s) can be combined with a reference value by adder 1830
to determine an error value that is used by controller 1810 to
determine whether the laser power level should be increased or
decreased. The controller 1810 then transmits a control signal to
laser 1820, thereby adjusting the laser power to improve the code
domain statistic. This feedback control system 1800 may operate
continuously and, in some embodiments, converge to an optimal power
level at video frame rates. FIG. 18 describes laser 1820 control as
a negative feedback loop. In some embodiments, laser 1820 may be
controlled using an adaptive algorithm or using non-linear search
techniques.
[0119] In block 1955, process 1900 determines whether the laser
power level converged. This may be performed by controller 1810 of
FIG. 18, processing circuit 504 of FIG. 5 and FIG. 18, or processor
606 of FIG. 6. Process 1900 may determine that the laser power
level converged if the power adjustment is less than a threshold
value. If the laser power level has not yet converged, and the
process 1900 proceeds along feedback path 1960 to block 1930.
[0120] If the laser power level has converged, process 1900
proceeds to block 1965. In block 1965, process 1900 updates the
depth map and stores it in memory. This may be performed by
processing circuit 504 of FIG. 5 and FIG. 18, or processor 606 of
FIG. 6. The updated depth map, or updated depth map information,
may be stored in memory/storage device 506 of FIG. 5 and FIG. 18,
or in memory 608 of FIG. 6. Process 1900 proceeds along path 1970
to circular flowchart element 1915 to start a new depth map update
cycle.
[0121] FIG. 20 illustrates an example of a process 2000 for
controlling the power of a structured light transmitter using
information determined from one or more received codewords that are
reflected from an object.
[0122] At block 2005, process 2000 projects patterns of codewords
onto one or more objects. This may be performed, for example, by
the image projecting device 408 (FIG. 4), laser 1820 (FIG. 18), or
the light emitter 602 (FIG. 6). The codewords are continuously
projected for time interval. The projected codewords may be
projected onto a scene, or objects in a scene.
[0123] At block 2010, process 2000 receives the codewords. This may
be performed by a receiver sensor 508 of FIG. 5 and FIG. 18, or a
sensor integrated with a light source for example, light receiving
element 604 integrated with a light emitter 602 of FIG. 6. The
received codewords may be received in an image of the scene or
objects in the scene.
[0124] At block 2015, process 2000 calculates a code domain
statistic from one or more of the received codewords. This may be
performed by processing circuit 504 of FIG. 5 and FIG. 18, or
processor 606 of FIG. 6. Code domain statistics quantify symbol
classification accuracy, codeword decoding accuracy, and/or basis
function decoding accuracy.
[0125] In a first example, symbol classification accuracy may
correlate with the contrast between symbols. The degree of contrast
may be quantified as described above with respect to FIG. 11, in
which means and standard deviations (and/or corresponding
variances) are estimated for the received intensity values for each
symbol, and equation (1) is used to determine code domain statistic
.LAMBDA.. FIG. 21, below, illustrates a process 2100 for
determining a code domain statistic. The determined code domain
statistic quantifies symbol classification accuracy.
[0126] In a second example, a code domain statistic may quantify
codeword detection accuracy by calculating the percentage of
decoded codewords that match expected codewords based on an
existing depth map or previously received codewords. FIG. 22,
below, illustrates a process 2200 for determining a code domain
statistic characterizing codeword detection accuracy.
[0127] In a third example, a code domain statistic may quantify
basis function coding accuracy by calculating the percentage of
correctly received basis function. FIG. 23, below, illustrates a
process 2300 for determining a code domain statistic. The
determined code domain statistic quantifies basis function
accuracy.
[0128] At block 2020, process 2000 adjusts power of the light
source based on the code domain statistic(s), and may loop back
through path 2025 to further project codewords at the adjusted
power setting of the light source. The process 2000 may adjust the
power of the light source in various implementations. One example
is a closed loop, negative feedback implementation as described
with respect to FIG. 18, in which a code domain statistic
determined by processing circuit 504 is compared to a reference
value by adder 1830 to generate a difference or error signal used
by controller 1810 to determine a control signal that adjusts the
power level of laser 1820. In a second example, an adaptive
algorithm is used to converge to an optimal laser power level. In a
third example, non-linear search techniques are used to identify an
optimum laser 1820 power level that maximizes one or more code
domain statistics.
[0129] FIG. 21 illustrates an example of a process 2100 for
calculating the code domain statistic of process 2000, wherein the
code domain statistic is the contrast between symbols as measured
by the square of the difference in intensity means divided by the
sum of the intensity variances.
[0130] At block 2105, process 2100 calculates corresponding
expected symbols from a depth map and/or previously received
codewords. The "expected" received symbols correspond to the most
likely (maximum likelihood) symbol. This may be performed by
processing circuit 504 of FIG. 5 and FIG. 18 or processor 606 of
FIG. 6. The depth map and/or previously received codewords are
stored in a memory or storage device. This may be performed by
memory/storage device 506 of FIG. 5 and FIG. 18 or memory 608 of
FIG. 6.
[0131] At block 2110, process 2100 assigns each received intensity
value to the expected symbol. This may be performed by processing
circuit 504 of FIG. 5 and FIG. 18 or processor 606 of FIG. 6. This
makes is possible to identify the received intensity values for
locations where "0" symbols are expected, as well as the intensity
values for locations where "1" symbols are expected based on the
depth map and/or previously received codewords. The received
intensity values may be labelled in a data structure by symbol, or
incorporated into respective symbol histograms.
[0132] At block 2115, process 2100 calculates the mean and variance
intensity values for each symbol. This may be performed by
processing circuit 504 of FIG. 5 and FIG. 18 or processor 606 of
FIG. 6. As discussed above with regard to FIG. 11, the probability
distribution functions of intensity values for each symbol may, if
considered to be normally distributed, be characterized by its mean
and variance (or corresponding standard deviation) of intensity
values.
[0133] At block 2120, process 2100 calculates a contrast statistic
based on the mean and variance intensity values for each symbol.
This may be performed by processing circuit 504 of FIG. 5 and FIG.
18 or processor 606 of FIG. 6. For example, contrast statistic
.LAMBDA. is the square of the distance in intensity means divided
by the sum of intensity variances, as defined in equation 1. Higher
contrast statistics correspond to greater symbol separation, less
overlap, and higher symbol detection accuracy.
[0134] FIG. 22 illustrates an example of a process 2200 for
calculating the code domain statistic of process 2000, wherein the
code domain statistic is the percentage of received codewords that
match the expected codewords. Blocks 2205, 2210, 2215, and 2220 of
process 2200 may each be performed by processing circuit 504 of
FIG. 5 and FIG. 18 or processor 606 of FIG. 6.
[0135] At block 2205, process 2200 calculates expected codewords
from a depth map and/or previously received codewords. The depth
map and/or previously received codewords are stored, for example,
in memory/storage device 506 of FIG. 5 and FIG. 18 or memory 608 of
FIG. 6.
[0136] At block 2210, process 2200 compares each received codeword
after error correction to its expected codeword. The expected
codeword is assumed to be correct in the absence of additional
information.
[0137] At block 2215, process 2200 calculates the percentage of
correctly received codewords. The percentage is the ratio of
received codewords that match the expected codewords. Higher
percentages correspond to greater codeword detection accuracy.
[0138] FIG. 23 illustrates an example of a process 2300 for
calculating the code domain statistic of process 2000, wherein the
code domain statistic is the percentage of received basis functions
that match the expected basis functions. Blocks 2305, 2310, and
2315 of process 2300 may each be described below, may each be
performed by processing circuit 504 of FIG. 5 and FIG. 18 or
processor 606 of FIG. 6.
[0139] At block 2305, process 2300 calculates expected basis
functions, as defined above with respect to FIG. 3, from a depth
map and/or previously received codewords. The depth map and/or
previously received codewords are stored, for example, in
memory/storage device 506 of FIG. 5 and FIG. 18 or memory 608 of
FIG. 6.
[0140] At block 2310, process 2300 compares each received basis
function to its expected basis function. The expected basis
function is assumed to be correct in the absence of additional
information.
[0141] At block 2315, process 2300 calculates the percentage of
correctly received basis functions. The percentage is the ratio of
received basis functions that match the expected basis functions.
Higher percentages correspond to greater basis function detection
accuracy. Higher basis function detection accuracy corresponds to
greater codeword detection accuracy.
[0142] It should be understood that any reference to an element
herein using a designation, for example, "first," "second," and so
forth does not generally limit the quantity or order of those
elements. Rather, these designations may be used herein as a
convenient method of distinguishing between two or more elements or
instances of an element. Thus, a reference to first and second
elements does not mean that only two elements may be employed there
or that the first element must precede the second element in some
manner. Also, unless stated otherwise a set of elements may
comprise one or more elements. In addition, terminology of the form
"at least one of: A, B, or C" used in the description or the claims
means "A or B or C or any combination of these elements."
[0143] As used herein, the term "determining" encompasses a wide
variety of actions. For example, "determining" may include
calculating, computing, processing, deriving, investigating,
looking up (e.g., looking up in a table for example a look-up
table, a database or another data structure), ascertaining and the
like. Also, "determining" may include receiving (e.g., receiving
information), accessing (e.g., accessing data in a memory) and the
like. Also, "determining" may include resolving, selecting,
choosing, establishing and the like.
[0144] As used herein, a phrase referring to "at least one of" a
list of items refers to any combination of those items, including
single members. As an example, "at least one of: a, b, or c" is
intended to cover: a, b, c, a-b, a-c, b-c, and a-b-c.
[0145] The various operations of methods described above may be
performed by any suitable means capable of performing the
operations, for example, various hardware and/or software
component(s), circuits, and/or module(s). Generally, any operations
illustrated in the Figures may be performed by corresponding
functional means capable of performing the operations.
[0146] The various illustrative logical blocks, modules and
circuits described in connection with the present disclosure may be
implemented or performed with a general purpose processor, a
digital signal processor (DSP), an application specific integrated
circuit (ASIC), a field programmable gate array (FPGA) or other
programmable logic device (PLD), discrete gate or transistor logic,
discrete hardware components or any combination thereof designed to
perform the functions described herein. A general purpose processor
may be a microprocessor, but in the alternative, the processor may
be any commercially available processor, controller,
microcontroller or state machine. A processor may also be
implemented as a combination of computing devices, e.g., a
combination of a DSP and a microprocessor, a plurality of
microprocessors, one or more microprocessors in conjunction with a
DSP core, or any other such configuration.
[0147] In one or more aspects, the functions described may be
implemented in hardware, software, firmware, or any combination
thereof. If implemented in software, the functions may be stored on
or transmitted over as one or more instructions or code on a
computer-readable medium. Computer-readable media includes both
computer storage media and communication media including any medium
that facilitates transfer of a computer program from one place to
another. A storage media may be any available media that can be
accessed by a computer. By way of example, and not limitation, such
computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or
other optical disk storage, magnetic disk storage or other magnetic
storage devices, or any other medium that can be used to carry or
store desired program code in the form of instructions or data
structures and that can be accessed by a computer. Disk and disc,
as used herein, includes compact disc (CD), laser disc, optical
disc, digital versatile disc (DVD), floppy disk and Blu-ray disc
where disks usually reproduce data magnetically, while discs
reproduce data optically with lasers. Thus, in some aspects
computer readable medium may comprise non-transitory computer
readable medium (e.g., tangible media).
[0148] The methods disclosed herein comprise one or more steps or
actions for achieving the described method. The method steps and/or
actions may be interchanged with one another without departing from
the scope of the claims. In other words, unless a specific order of
steps or actions is specified, the order and/or use of specific
steps and/or actions may be modified without departing from the
scope of the claims.
[0149] Further, it should be appreciated that modules and/or other
appropriate means for performing the methods and techniques
described herein can be downloaded and/or otherwise obtained by a
user terminal and/or base station as applicable. For example, such
a device can be coupled to a server to facilitate the transfer of
means for performing the methods described herein. Alternatively,
various methods described herein can be provided via storage means
(e.g., RAM, ROM, a physical storage medium, for example, a compact
disc (CD) or floppy disk, etc.), such that a user terminal and/or
base station can obtain the various methods upon coupling or
providing the storage means to the device. Moreover, any other
suitable technique for providing the methods and techniques
described herein to a device can be utilized.
[0150] It is to be understood that the claims are not limited to
the precise configuration and components illustrated above. Various
modifications, changes and variations may be made in the
arrangement, operation and details of the methods and apparatus
described above without departing from the scope of the claims.
* * * * *