U.S. patent application number 15/257166 was filed with the patent office on 2017-03-09 for method and apparatus for generating a 3-d image.
The applicant listed for this patent is INUITIVE LTD.. Invention is credited to Chagai ENSENBERG, Lev GOLDENTOUCH.
Application Number | 20170070726 15/257166 |
Document ID | / |
Family ID | 58190722 |
Filed Date | 2017-03-09 |
United States Patent
Application |
20170070726 |
Kind Code |
A1 |
GOLDENTOUCH; Lev ; et
al. |
March 9, 2017 |
METHOD AND APPARATUS FOR GENERATING A 3-D IMAGE
Abstract
A method and apparatus are provided for generating a three
dimensional image. The method comprises: illuminating a target;
capturing an image that comprises an object present at the
illuminated target; converting the captured image into data and
processing it to determine depth of the object; and generating a
three dimensional image of the object whose depth has been
determined. Preferably, the illumination intensity applied while
illuminating the target, is determined based on the current
intensity of ambient light at the target.
Inventors: |
GOLDENTOUCH; Lev; (Rishon
Lezion, IL) ; ENSENBERG; Chagai; (Zur-Yigal,
IL) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
INUITIVE LTD. |
Ra'anana |
|
IL |
|
|
Family ID: |
58190722 |
Appl. No.: |
15/257166 |
Filed: |
September 6, 2016 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
62213639 |
Sep 3, 2015 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
H04N 13/257 20180501;
G06T 7/521 20170101; H04N 13/25 20180501; H01S 5/423 20130101; G06T
7/593 20170101; H01S 5/042 20130101; H04N 13/254 20180501 |
International
Class: |
H04N 13/02 20060101
H04N013/02; H04N 9/04 20060101 H04N009/04; G06K 9/62 20060101
G06K009/62; G06T 7/00 20060101 G06T007/00; G06K 9/20 20060101
G06K009/20 |
Claims
1. A method for generating a three dimensional image, comprising
the steps of: (i) illuminating a target; (ii) capturing an image
that comprises one or more objects present at the illuminated
target; (iii) converting the captured image into data; (iv)
processing the data received from the conversion of the captured
image; (v) determining depth of at least one of the one or more
objects present at the illuminated target; and (vi) generating a
three dimensional image of the at least one of the one or more
objects whose depth has been determined.
2. The method of claim 1, wherein the step of illuminating a target
comprises projecting light at a pre-defined pattern onto the
target.
3. The method of claim 2, wherein the projected light is emitted
from a VSCEL array and its pattern is formed by at least one
diffractive optical element.
4. The method of claim 1, wherein the illumination intensity
applied while illuminating the target, is determined based on
current intensity of ambient light at the target.
5. The method of claim 1, wherein the image of the one or more
objects comprised in the illuminated target is captured by at least
one depth capturing sensor and at least one RGB sensor, and wherein
the step of processing comprises processing data retrieved from the
at least two sensors.
6. The method of claim 1, further comprising a step of capturing at
least one other image of the one or more objects associated with
the preceding three dimensional image, and determining whether a
change had occurred to said one or more objects within a period of
time extending between the time at which the first image was
captured and the time at which the at least one other image was
captured.
7. The method of claim 1, further comprising a step of retrieving
audio information emanating at the target vicinity.
8. The method of claim 7, further comprising repeating steps (ii)
to (iv) and determining a movement of at least one of the one or
more objects whose depth has been determined, by combining
information derived from at least two captured images and said
audio information.
9. An apparatus configured to provide a three dimensional image of
at least one object, the apparatus comprising: a projector
configured to illuminate a target where the at least one object is
present; at least one image capturing sensor, configured to capture
an image that comprises the at least one object, by retrieving
light reflected from the at least one object; and at least one
processor configured to process data retrieved from the image
captured by the at least one image capturing sensor and to generate
a three dimensional image that comprises the at least one
object.
10. The apparatus of claim 9, wherein said projector comprises a
VCSEL array.
11. The apparatus of claim 9, wherein said apparatus further
comprises a diffractive optical element for providing a pre-defined
pattern of light emitted from the projector.
12. The apparatus of claim 9, wherein the pre-defined pattern is a
pattern that enables carrying out a three dimensional imaging of
the at least one object.
13. The apparatus of claim 9, wherein the projector illumination
intensity for illuminating the target is determined based on
current intensity of ambient light at the target.
14. The apparatus of claim 9, wherein the apparatus comprises at
least one depth capturing sensor and at least one RGB sensor, and
wherein the processor is configured to process data retrieved from
the at least two sensors for generating the three dimensional
image.
15. The apparatus of claim 9, further comprising a microphone
configured to capture audio information emanating at the target
vicinity.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims priority of U.S. Provisional Patent
Application No. 62/213,639, filed Sep. 3, 2015, the disclosure of
which is incorporated by reference herein in its entirety.
TECHNICAL FIELD
[0002] The present disclosure generally relates to methods for
using optical devices, and more particularly, to methods for
determining depth for generating a three dimensional image of an
object.
BACKGROUND
[0003] The present disclosure relates to active systems having
global shutter sensor, for example a CCD sensor having a global
shutter. The term "CCD" stands for Charge Coupled Device, and the
pixels in a CCD store their charge until it has been depleted. A
camera that has a CCD sensor almost always has also a shutter
system, which can be electronic or mechanical. While this shutter
system is "closed" (or "off"), the pixels can still be read because
they store their charges. However, when the shutter is open, the
sensor collects light, and after the shutter closes, the AFE
("Analog Front End") reads the pixel charges one by one, dumps any
excess charge, and gets the pixels ready for the next frame. In
other words, the CCD captures the entire image at the same time and
then reads the information after the capture is completed, rather
than reading top to bottom during the exposure. Because it captures
everything at once, the shutter is considered to be a "global" one.
The result is an image with no motion artifacts. CCD sensors are
double layered and are adapted to transfer their charges from one
layer to the next before the read out is conducted.
[0004] An active system that determines depth from stereoscopic
images, uses projected pattern generated by projector light source
(such as a laser, a LED, a VCSEL or any similar device). In
addition to the projected pattern, a measurement is conducted of
observable light emitted by other sources. For example, ambient
light derived from the sun or from room illumination, stray light
generated as a side effect of the projector light source operation,
backlight generated by a secondary controlled light source such as
LED, and the like.
[0005] In order to take ambient light into consideration, it may be
measured and the power of the projector/LED be modified
accordingly. Alternatively, the exposure value may be set up by a
user who determines an illumination profile.
[0006] The projector light source is efficient when operating under
a certain range of currents, and dictates the voltage value,
optical power and efficiency. Typically, the operation of a light
source is efficient when the light source operates under relatively
high surge current for short pulses, or when the light source
operates under lower current for longer pulses, due to heating.
[0007] The projector light source may behave differently under
different illumination conditions. Under intense ambient light
conditions, it would make sense to generate short high power
pulses, so that the projected pattern is visible at the object. On
the other hand, under moderate ambient light conditions it would
make sense to generate longer and weaker pulses, in order to
maximize the light source efficiency as well as the available
ambient light. Yet, in the absence of ambient light, it would make
sense to utilize still longer pulses and a secondary light source,
in order to obtain scattered light upon the object. In addition, it
should be noted that the projected pattern contrast, may shift the
working points.
[0008] The power provided to the projector light source needs
preferably to be such that it is able to meet several requirements,
among which: pulse shape and power, energy efficiency, nominal
input voltage, maximal input current, low physical size and
cost.
[0009] Two standard methods of providing projector power are known
in the art. The first, using DC to DC converter circuits and the
second, using switch capacitor voltage and current sources. In both
these methods, the size and cost of the circuitry involved in order
to drive effectively the projected pattern and the secondary light
source, are quite limiting factors.
SUMMARY OF THE DISCLOSURE
[0010] The disclosure may be summarized by referring to the
appended claims.
[0011] It is an object of the present disclosure to provide a
method and an apparatus for generating a three dimensional image of
an object which is present at a target area.
[0012] It is another object of the present disclosure to provide a
method and an apparatus for generating a three dimensional image of
an object by illuminating the target area at which the object is
present.
[0013] It is another object of the present disclosure to provide a
method and an apparatus for generating a three dimensional image of
an object by illuminating the target area at which the object is
present, and wherein the illumination is carried out by taking into
account the ambient light conditions at the target area.
[0014] Other objects of the present invention will become apparent
from the following description.
[0015] According to a first embodiment of the disclosure, there is
provided a method for generating a three dimensional image,
comprising the steps of: [0016] (i) illuminating a target (i.e. at
least one part of the target area); [0017] (ii) capturing an image
that comprises one or more objects present at the illuminated
target; [0018] (iii) converting the captured image into data;
[0019] (iv) processing the data received from the conversion of the
captured image; [0020] (v) determining depth of at least one of the
one or more objects present at the illuminated target; and [0021]
(vi) generating a three dimensional image of the at least one of
the one or more objects whose depth has been determined.
[0022] It should be noted that the terms "target" and "target area"
are both used herein throughout the description and claims to
denote an area which is illuminated in accordance with certain
embodiments of the present disclosure.
[0023] According to another embodiment, the step of illuminating a
target comprises projecting light at a pre-defined pattern onto the
target.
[0024] In accordance with another embodiment, the projected light
is emitted from a VSCEL array and its pattern is formed by at least
one diffractive optical element.
[0025] According to yet another embodiment, the illumination
intensity applied while illuminating the target, is determined
based on current intensity of ambient light at the target.
[0026] By still another embodiment, the step of capturing the image
of the one or more objects comprised in the illuminated target,
comprises retrieving the image by at least one depth capturing
sensor (e.g. a camera) and at least one RGB sensor (e.g. a camera),
and wherein the step of processing the retrieved data relates to
data retrieved from the at least two sensors.
[0027] The term an "RGB sensor" is used herein throughout the
specification and claims to denote a sensor (e.g. a camera) that
delivers the three basic color components (red, green, and blue) on
three different wires. This type of camera often uses three
independent CCD sensors to acquire the three color signals. RGB
cameras are typically used for very accurate color image
acquisitions.
[0028] According to another embodiment, the method further
comprising a step of capturing at least one other image (a second
image) of the at least one object associated with the preceding
three dimensional image (a first image) that was generated, and
determining whether a change had occurred to that at least one
object within a period of time extending between the time at which
the first image was captured and the time at which the at least one
other (second) image was captured.
[0029] In accordance with another embodiment, the method further
comprising a step of retrieving audio information emanating at the
target vicinity.
[0030] By yet another embodiment, the method further comprising
repeating steps (ii) to (iv) and determining a movement of the at
least one of the one or more objects whose depth has been
determined, by combining information derived from at least two
captured images and the retrieved audio information emanating at
the target vicinity.
[0031] According to another aspect of the disclosure, an apparatus
is provided, wherein the apparatus is configured to provide a three
dimensional image of at least one object, and comprising:
[0032] a projector configured to illuminate a target area where the
at least one object is present;
[0033] at least one image capturing sensor, configured to capture
an image that comprises the at least one object, by receiving light
reflected from the at least one object; and
[0034] at least one processor configured to process data retrieved
from the image captured by the at least one image capturing sensor
and to generate a three dimensional image that comprises the at
least one object.
[0035] According to still another embodiment of this aspect, the
projector comprises a VCSEL array.
[0036] By yet another embodiment, the apparatus further comprises a
diffractive optical element for providing a pre-defined pattern of
the light emitted from the projector.
[0037] In accordance with another embodiment, the pre-defined
pattern is a pattern that enables a three dimensional imaging of
the at least one object, e.g. by implementing a structured light
technique. A structured light technique is a process of projecting
a known pattern (often grids or horizontal bars) onto a target. The
way that this pattern is deformed when light strikes a surface,
allows vision systems to calculate the depth and surface
information of the objects at the target.
[0038] Invisible structured light technique uses structured light
without interfering with other computer vision tasks for which the
projected pattern will be confusing. For example, by using infrared
light or of extremely high frame rates alternating between two
exact opposite patterns.
[0039] According to another embodiment, the illumination intensity
of the projector applied while illuminating the target, is
determined based on current intensity of ambient light at the
target area. This functionality, may be obtained for example by
using a simple capacitor-based circuit configured to shape the
illumination profile of light emitted by the projector for various
ambient light values based on exposure setting. Optionally, the
apparatus is adapted to enable switching between several light
sources based on the image properties (and if a motion detection is
required).
[0040] According to still another embodiment, the apparatus
comprises at least one depth capturing sensor and at least one RGB
sensor, and wherein the processor is configured to process data
retrieved from the at least two sensors, for generating the three
dimensional image.
[0041] In accordance with yet another embodiment, the apparatus
further comprising a microphone configured to capture audio
information emanating at the target vicinity.
BRIEF DESCRIPTION OF THE DRAWING
[0042] For a more complete understanding of the present invention,
reference is now made to the following detailed description taken
in conjunction with the accompanying drawing wherein:
[0043] FIG. 1--illustrates an exemplary system construed according
to an embodiment of the present invention;
[0044] FIG. 2--exemplifies a block diagram of an imaging
module;
[0045] FIG. 3--illustrates a process of controlling a LED driver
according to an embodiment of the invention;
[0046] FIG. 4--exemplifies current output of the controlled LED
driver of FIG. 2;
[0047] FIG. 5--illustrates a series of curves of current profiles
for both the pattern projector and the secondary light source;
[0048] FIG. 6--represents a simple implementation of the controlled
LED driver which utilizes a capacitor discharge to drive the power
profile shown in FIG. 4;
[0049] FIG. 7--exemplifies a method for implementing an exposure
control by depth processing pipeline, according to an embodiment of
the invention;
[0050] FIG. 8--illustrates a method for controlling an analog gain
implemented according to an embodiment of the invention;
[0051] FIG. 9--illustrates a method for controlling the exposure of
a secondary light source, implemented according to an embodiment of
the invention; and
[0052] FIG. 10--demonstrates steps of a method for controlling
motion in imaging mode, implemented according to an embodiment of
the invention.
DETAILED DESCRIPTION
[0053] In this disclosure, the term "comprising" is intended to
have an open-ended meaning so that when a first element is stated
as comprising a second element, the first element may also include
one or more other elements that are not necessarily identified or
described herein, or recited in the claims.
[0054] In the following description, for the purposes of
explanation, numerous specific details are set forth in order to
provide a better understanding of the present invention by way of
examples. It should be apparent, however, that the present
invention may be practiced without these specific details.
[0055] In addition, in the following description the term
"reference sensor" should be understood as a sensor to which the
disparity image is attributed, e.g., the right sensor or the left
sensor. However, it should be understood by those skilled in the
art, that a reference sensor may alternatively be a virtual
reference sensor, generated by interpolating or extrapolating
disparities that are visible by the left and right sensors, while
filling in missing occlusion data.
[0056] FIG. 1 illustrates an exemplary system construed according
to an embodiment of the present invention.
[0057] The system comprises cameras 101 and 102 which capture the
same image with some displacement caused by a baseline shift
between the images. A depth image is then generated by combining
the stereo images captured by these two cameras.
[0058] Optionally, an additional camera 103 (such as a webcam) may
be used to provide an RGB image at a higher resolution. This RGB
image needs to be combined with the depth image obtained by
combining the two stereo images, in order to obtain a 3D
display.
[0059] Processing data in order to determine the applicable depth
from the stereo images is carried out by a processor (referred to
herein as depth processing pipeline 110), which may be implemented
by using a Very-Large-Scale Integration ("VLSI") circuit or a
Field-Programmable Gate Array ("FPGA"). It may generate
disparity/or and depth related data as well as RGB data retrieved
from the cameras' inputs. The depth processing pipeline is
controlled by using registers and shadow registers set by the host
device (or firmware) 120.
[0060] Stereo image acquisition block 111 may be implemented as a
hardware pipeline performing any one or more of the following
functions: sensor control, sensor synchronization, geometric
distortion correction, illumination correction as well as other
processing required for generating a reliable 3D stereo image out
of the data retrieved from cameras 101 and 102.
[0061] Disparity from stereo block 112 may be implemented by a
hardware pipeline performing operations required to generate
disparity from a stereo image as will be further described in
connection with FIG. 2.
[0062] Post processing block 123 may be implemented as a hardware
for improving the disparity image and producing depth output. In
some embodiments it may further include RGB related data received
from the cameras.
[0063] Host device (or firmware) 120 is operative to control the
depth processing pipeline 110 and to generate an output as required
by the end user.
[0064] Application block 121 may execute various algorithms
requested by the user, including face detection, skeleton tracking,
hands tracking, gaze tracking, 3D object stitching, and the
like.
[0065] Image analysis block 122 is operative to carry out one or
more of the following functions: image segmentation, object
detection and motion analysis.
[0066] Flow control block 123 is configured to carry out control of
complex flows, using multiple heterogeneous frame acquisition.
[0067] Parameter tuning block 124 may include generation of
specific registers and shadow registers values, as well as
modifying software and firmware parameters for each consecutive
frame.
[0068] FIG. 2 exemplifies a block diagram of imaging module 201.
According to this example it is operative to receive data retrieved
from the left (101) and right (102) cameras and the additional
camera 103 (e.g. an RGB camera) depicted in FIG. 1. The imaging
module may comprise a pattern projector 211, at least one second
light source 212, where the latter may include a flood light LED or
an additional pattern projector, or any other applicable cameras
and light sources.
[0069] Pattern projector 211 and the at least one second light
source 212 may receive electric power from a controlled LED driver
202. The depth processing pipeline 110 described in FIG. 1, may be
configured to control the controlled LED driver 202 and the imaging
module 201, based on images retrieved from cameras 101, 102 (and
optionally 103).
[0070] Ambient light depicted as 213 in this FIG. 2, may be the
outcome of solar radiation, lamp(s) illumination or of any other
applicable external light source, as long as it is not controllable
by the system of the present invention.
[0071] Power source 203 is used to supply power to all modules. The
power source may be one which is configured to supply a pre-defined
constant voltage, and/or has a maximal surge power and/or complies
with a pre-defined average power threshold.
[0072] FIG. 3 illustrates steps in the process of controlling the
LED driver according to an embodiment of the invention, wherein the
controlled LED driver 202 shown in FOG. 2, may change states during
each frame acquisition. The steps according to this example
are:
[0073] Step 301 (the charging step) takes place preferably when the
light sources are not active and the internal capacitors are then
charged. As will be appreciated by those skilled in the art, this
charging step may be the longest step from among the steps depicted
in FIG. 3.
[0074] Step 302 (designated in FIG. 3 as "strong light" step),
takes place when the light emitted by the pattern projector 211 of
FIG. 2 is of higher intensity (e.g. brighter) than the ambient
light. The system may be less efficient during this step. The
pattern may be used for stereo detection of walls and other large
objects, where a pattern may be useful.
[0075] Step 303, efficient illumination step, takes place if the
system is configured for obtaining a maximal efficiency in the
presence of moderate or weak ambient light. This step may be
carried out for example by modifying (increasing) exposure, the
analog gain and the digital gain of the sensors.
[0076] In the secondary illumination step (step 304) of this
example, the main pattern projector 211 is turned off, and a
secondary light source 212 is applied. For example, when operating
under high definition mode, a secondary pattern may project finer
grid onto the target. In the absence of ambient light, a flood LED
may be used to generate the flood light. The flood light may be
used to enable distinguishing between bright and dark objects,
between objects and shadows, to illuminate larger field of view
than the field of view illuminated by pattern projector 211, or for
any other applicable purpose. In some embodiments, the pattern
projector may operate within the infrared range, while the
secondary light source may be utilized to provide visible light
(e.g. for the additional camera(s) 103).
[0077] FIG. 4 exemplifies current output of the controlled LED
driver (202), for the different steps described in FIG. 3, which in
a way may be considered as being similar to a capacitor
discharge.
[0078] The maximal current output of the controlled LED driver 202
should not be higher than the surge/saturation current of the
pattern projector 211. During step 302 where the light emitted by
the pattern projector 211 of FIG. 2 is of higher intensity than the
ambient light, the light output may fall fast from the saturation
current value where the light output of the pattern projector 211
has a maximal value to the nominal operation current characterized
by high energy efficiency of the pattern projector 211.
[0079] In the efficient illumination step, 303, the current output
of the controlled LED driver 202 is around the nominal current
value and should not be below the cutoff current of the pattern
projector 211.
[0080] In some embodiments, secondary light source 212 has a
different value for the nominal current, a value which may be below
the nominal current value of the pattern projector 211.
[0081] FIG. 5 illustrates a series of curves of current profiles
for both the pattern projector and the secondary light source.
[0082] The pattern projector 211 may have cutoff current value 501
being the threshold value below which it is not efficient, nominal
current 502 which is the current used for a long-term operation,
saturation current 503 which is used for an efficient intense light
operation, while the value of the surge current threshold, 504, is
a value beyond which eye safety hazards or damage of certain
components, might occur.
[0083] In a somewhat similar way, the secondary light source 212
may have a cutoff current value 505, a nominal current 506, a
saturation current 507 and a surge current 508. It should be noted
that the currents characterizing the operation of the secondary
light source 212, may be in a different range than the currents
characterizing the operation of the pattern projector 211.
[0084] Turning now to FIG. 6 which represents a simple
implementation of the controlled LED driver 202 which utilizes a
capacitor discharge to drive the power profile shown in FIG. 4. The
actual implementation of the electric circuit may include
additional elements such as an NPN transistor to drive the light
sources, an operational amplifier to close feedback loops or other,
more complex electric components.
[0085] A current limited source 601, such as USB connection or a DC
power supply, may be used to charge a capacitor 602 via an NTC
resistor 603. An NTC resistor may be selected so as to limit the
current surge when the circuit is turned on. Capacitor 602 may
comprise several capacitor elements with complementing capacity and
ESR (equivalent series resistance). Low ESR large capacity elements
may be expensive, so they may be replaced for example by using
multiple high ESR large capacity and low ESR low capacity
elements.
[0086] A signal from the depth processing pipeline (110) may be
used to activate the pattern projector 211, the second light source
212 and cameras 101, 102 and 103 (if used). Preferably, the
operation of the light sources and the cameras are synchronized in
order to achieve an improved efficiency.
[0087] Once a light source is activated, capacitor 602 begins its
discharge while powering the respective light source. If pattern
projector 211, implemented for example by using an array of
vertical-cavity surface-emitting laser (VCSEL) 605, is activated,
the capacitor 602 discharges via inductive element 604 and a low
value resistor 607. The resistance of resistor 607 may be tuned to
the ESR of VCSEL 605, while taking into account the required
discharge time of capacitor 602. The inducting element 604 may be
used to limit the current of the capacitor discharge below the
surge current of the VCSEL array 605.
[0088] When the secondary light source 212, implemented for example
by LED 606, is activated, the capacitor 602 discharges through a
different resistor (possibly one having a larger resistance) 608,
providing currents that are suitable for the LED 606 operation.
[0089] In cases where both VCSEL array 605 and LED 606 are both
activated, the LED 606 preferably emits its light only after the
VCSEL array 605 has ceased to emit light, e.g. when current cutoff
conditions occurred.
[0090] FIG. 7 exemplifies a series of steps for implementing an
exposure control technique by depth processing pipeline 110,
according to an embodiment of the invention.
[0091] In step 701, the system measures the ambient light, such a
measurement may be done for example by turning off light sources
211 and 212 during a dedicated frame. The average grayscale levels
detected at the image taken by a camera when light sources 211 and
212 are turned off, may be used as indicators for the intensity of
the ambient light.
[0092] Next, in step 702, ROIs (regions of interest) are selected
on the grayscale image, in order to perform the intensity
evaluation. In some embodiments, the FOV (field of view) of light
sources 211 and 212 may be different from each other and the
illumination intensity may vary within the respective FOVs.
Therefore, instead of taking the full image into account, the
exposure control mechanism may be activated for certain selected
areas of the image.
[0093] In step 703, histograms of the grayscale values within the
ROI, are generated. The histograms preferably provide the median,
the top percentile and the bottom percentile of the grayscale
levels as well as other suitable information.
[0094] The term "percentile" is used herein denotes a measure used
in statistics indicating the value below which a given percentage
of observations in a group of observations fall. For example, the
tenth percentile is the value (or score) below which 10 percent of
the observations may be found.
[0095] In step 704, the change in exposure required to optimize
median grayscale level, is calculated. Optionally, the desired
median grayscale level is roughly at the middle of the dynamic
range of the system.
[0096] Next, the exposure level is reduced (step 705) in order to
improve the system power consumption and to provide light which is
more intense than the ambient light. Preferably, the system
performance (depth accuracy, noise, resolution, etc.) does not
change much within a large part of the dynamic range, so that the
exposure may be reduced accordingly. The lower percentile of the
gray levels may be observed in order to minimize degradation of the
system performance.
[0097] In some embodiments, object proximity and eye safety
considerations may also be taken into account when determining the
reduction in the exposure.
[0098] In step 706 the top gray levels percentile is monitored and
the exposure may be further reduced in order to avoid reaching a
saturation state.
[0099] FIG. 8 illustrates steps of a method for controlling an
analog gain implemented according to an embodiment of the invention
by depth processing pipeline 110.
[0100] In step 801, the exposure is checked. Optionally, the analog
gain control is activated only when the exposure control reaches a
maximal exposure level.
[0101] In step 802, dark and bright gray level percentile are
evaluated, in order to enable calculating the required modification
of the analog gain.
[0102] Next, the analog gain is modified (step 803) in order to
provide improved (preferably the best) gray levels for the depth
processing pipeline 110.
[0103] Noise resulting from new (modified) analog gain is then
calculated in step 804, and the algorithmic parameters are updated
(step 805) to enable achieving the best depth processing available
under the given noise conditions. For example, using larger areas
for aggregation computation.
[0104] FIG. 9 illustrates steps of a method for controlling the
exposure of a secondary light source, implemented according to an
embodiment of the invention by depth processing pipeline 110.
[0105] First, imaging mode is checked in order to verify whether
the exposure signal is conveyed to the pattern projector 211, or to
the secondary light source 212, or to both (step 901).
[0106] Then, the system resolution is checked (step 902) and a
secondary, finer pattern is activated (step 903) if the mode of
high resolution is configured. Ambient light is then checked (step
904) and the flood light of the LED or the RGB illumination is
activated in case it has been determined that the ambient light is
low.
[0107] FIG. 10 demonstrates steps of a method for controlling
motion in imaging mode, implemented according to an embodiment of
the invention by depth processing pipeline 110. The method
comprises the steps of:
[0108] Dividing the image into a plurality of blocks (step
1001).
[0109] For each of the blocks, calculating block statistics and
determining if the block statistics matches the block statistics of
previous frame(s) (step 1002).
[0110] Detecting if a motion took place between frames, by
analyzing discrepancies in block statistics (step 1003).
[0111] Activating the illumination mode, based on the determination
made on the system motion (step 1004).
[0112] It should be noted that the motion detection mechanism
demonstrated above may be performed by using data retrieved from
left camera 101 or from the right camera 102. In addition or in the
alternative, it may be performed by using data retrieved from the
additional camera(s) 103.
[0113] The following description provides further elaborations on
certain aspects of the examples described hereinbefore.
[0114] According to an embodiment of the system provided by the
present invention several algorithms may be activated independently
of each other and the results obtained from the different
algorithms may then be processed in order to improve the end
result.
[0115] Following are some exemplified algorithms that may be
activated independently of each other: [0116] 1. Histogram
computation over disparity, left image or right image in a set of
regions of interest; [0117] 2. Exposure control based on statistics
of left image or the right image. [0118] 3. Selected statistics
computations over separate tiles of an image, including mean
values, standard deviations, minimum values, maximum values, median
value, etc. [0119] 4. A/B mode switching between a mode with
projected pattern and depth computation and an ambient-driven mode
with a dedicated computer vision processing. Let us now consider in
more details some of the modes described:
Ambient-Driven Mode:
[0120] 1. In ambient-driven mode no pattern is projected. 2. Image
statistics computation includes ROI histogram computation and
selected tile statistics for the left and right images. 3. The
current image statistics is compared with the statistics of the
preceding frame and the statistics of the first frame after
deactivation of projector-driven mode. If the change in statistics
between frames or the aggregated statistics drift is above a
pre-defined threshold, the projector-driven mode would be
activated. 4. Optionally, if a change has been detected, an
enhanced ambient-driven mode may be activated. For example, image
resolution and frame rate may be increased without turning on the
projector.
Projector-Driven Mode:
[0121] In this mode, a pattern is projected and depth processing is
performed.
[0122] Statistical data may be collected for determining disparity
and/or the depth map.
[0123] A comparison may be conducted between depth statistics or
the statistics of left and right images, of different frames. If
the statistical data is not significantly different between frames,
a statistical data aggregation algorithm may be activated. The
statistical data aggregation algorithm compares the statistics of
each frame not only to the current frame but also to the first
frame for which the aggregation algorithm has begun collecting the
statistical data.
[0124] If a substantial change has been noted in the statistical
data collected or in the aggregated statistical data, the
statistical aggregation algorithm is deactivated. However, if there
has been no substantial change in the statistical data collected or
in the aggregated statistical data for a pre-defined number of
frames, the ambient-driven mode may be activated (or re-activated)
while the projector-driven mode is deactivated.
[0125] When the projector-driven mode is deactivated, the user may
receive the last valid depth frame.
[0126] Optionally, activation or deactivation of the projector mode
is region-of-interest specific. In other words, the depth of the
projected image is calculated over a certain region of
interest.
[0127] Also, the system of the present invention may be configured
to use a reduced projector-driven mode, i.e. a mode characterized
in a lower frame rate, a lower resolution or a lower exposure.
Alternating Imaging Mode:
[0128] While operating under a full alternating imaging mode, both
projector mode image and ambient-driven image are collected with
certain time sharing between them. The ambient-driven imaging may
switch to an advanced mode having higher capabilities which is
based on motion detection. Projector-driven imaging may on the
other hand, switch to a reduced mode with lower capabilities based
on motion detection.
[0129] In the description and claims of the present application,
each of the verbs, "comprise" "include" and "have", and conjugates
thereof, are used to indicate that the object or objects of the
verb are not necessarily a complete listing of members, components,
elements or parts of the subject or subjects of the verb.
[0130] The present invention has been described using detailed
descriptions of embodiments thereof that are provided by way of
example and are not intended to limit the scope of the invention in
any way. For example, the apparatus may include a cameras' array
that has two or more cameras, such as, for example, video cameras
to capture two or more video streams of the target. The described
embodiments comprise different features, not all of which are
required in all embodiments of the invention. Some embodiments of
the present invention utilize only some of the features or possible
combinations of the features. Variations of embodiments of the
present invention that are described and embodiments of the present
invention comprising different combinations of features noted in
the described embodiments will occur to persons of the art. The
scope of the invention is limited only by the following claims.
* * * * *