U.S. patent application number 17/365985 was filed with the patent office on 2021-11-04 for method and system for determining optimal exposure time and number of exposures in structured light-based 3d camera.
This patent application is currently assigned to RESEARCH & BUSINESS FOUNDATION SUNGKYUNKWAN UNIVERSITY. The applicant listed for this patent is RESEARCH & BUSINESS FOUNDATION SUNGKYUNKWAN UNIVERSITY. Invention is credited to Yun-su CHOI, Suk-han LEE, In-sub SHIN.
Application Number | 20210344892 17/365985 |
Document ID | / |
Family ID | 1000005724857 |
Filed Date | 2021-11-04 |
United States Patent
Application |
20210344892 |
Kind Code |
A1 |
LEE; Suk-han ; et
al. |
November 4, 2021 |
METHOD AND SYSTEM FOR DETERMINING OPTIMAL EXPOSURE TIME AND NUMBER
OF EXPOSURES IN STRUCTURED LIGHT-BASED 3D CAMERA
Abstract
Disclosed is a method for determining the optimal exposure time
and the number of exposures in a structured light-based 3D camera
system. The method includes capturing a reference image onto which
first and second patterns are projected, calculating a slope
related to brightness and an exposure time for a pixel of the
reference image and categorizing a state of the slope, creating an
upper loss pixel distribution chart and an accumulation pixel
distribution chart by calculating the number of upper loss pixels
and the number of accumulation pixels for the exposure time of the
structured light-based 3D camera system, and determining the
optimal exposure time and the number of exposures of the structured
light-based 3D camera system using the upper loss pixel
distribution chart and the accumulation pixel distribution
chart.
Inventors: |
LEE; Suk-han; (Yongin-si,
KR) ; CHOI; Yun-su; (Seongnam-si, KR) ; SHIN;
In-sub; (Bucheon-si, KR) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
RESEARCH & BUSINESS FOUNDATION SUNGKYUNKWAN UNIVERSITY |
Suwon-si |
|
KR |
|
|
Assignee: |
RESEARCH & BUSINESS FOUNDATION
SUNGKYUNKWAN UNIVERSITY
Suwon-si
KR
|
Family ID: |
1000005724857 |
Appl. No.: |
17/365985 |
Filed: |
July 1, 2021 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
16558151 |
Sep 1, 2019 |
11070788 |
|
|
17365985 |
|
|
|
|
15334314 |
Oct 26, 2016 |
10404968 |
|
|
16558151 |
|
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
H04N 5/2351 20130101;
H04N 13/254 20180501; H04N 5/2353 20130101; H04N 5/2356
20130101 |
International
Class: |
H04N 13/254 20060101
H04N013/254; H04N 5/235 20060101 H04N005/235 |
Foreign Application Data
Date |
Code |
Application Number |
Oct 27, 2015 |
KR |
10-2015-0149156 |
Claims
1. A method for determining an exposure parameter for a structured
light-based 3D camera system, the method comprising: capturing a
first reference image and a second reference image, the first
reference image including a reflection of a first pattern from a
target object and the second reference image including a refection
of a second pattern from the target object; calculating a pixel
distribution from the first reference image and the second
reference image; and determining the exposure parameter using the
pixel distribution.
2. The method of claim 1, wherein the calculating the pixel
distribution includes calculating a plurality of slopes related to
a brightness and an exposure time for selected pixels of the first
reference image and the second reference image, and categorizing
the plurality of slopes.
3. The method of claim 1, wherein the calculating the pixel
distribution further includes calculating an upper loss pixel
distribution and an accumulation pixel distribution from a number
of upper loss pixels and a number of accumulation pixels.
4. The method of claim 1, wherein the exposure parameter includes
at least one of an exposure time and a number of exposures.
5. The method of claim 1, wherein each of the first pattern and the
second pattern includes at least one of an all-black pattern and an
all-white pattern.
6. The method of claim 3, wherein the calculating the upper loss
pixel distribution and the accumulation pixel distribution includes
excluding states in which a brightness value of a pixel associated
with the first reference image and a brightness value of a pixel
associated with the second reference image are less than a
predetermined threshold brightness value.
7. The method of claim 1, wherein the first pattern and the second
pattern are the same.
8. The method of claim 1, wherein the first pattern and the second
pattern are different.
9. A structured light-based 3D camera system, comprising: a
projection unit for projecting a first pattern and a second pattern
onto a target object; a capturing unit for capturing a first
reference image and a second reference image, the first reference
image including a reflection of the first pattern from the target
object and the second reference image including a reflection of the
second pattern from the target object; and a processing unit for
calculating a pixel distribution from the first reference image and
the second reference image, and determining an exposure parameter
for the structured light-based 3D camera system using the pixel
distribution.
10. The structured light-based 3D camera system of claim 9, wherein
each of the first pattern and the second pattern includes at least
one of an all-black pattern and an all-white pattern.
11. The structured light-based 3D camera system of claim 9, wherein
the exposure parameter includes at least one of an exposure time
and a number of exposures.
12. The structured light-based 3D camera system of claim 9, wherein
the processing unit calculates the pixel distribution by
calculating a plurality of slopes related to a brightness and an
exposure time for selected pixels of the first reference image and
the second reference image, and categorizing the plurality of
slopes.
13. The structured light-based 3D camera system of claim 9, wherein
the processing unit calculates the pixel distribution by
calculating an upper loss pixel distribution and an accumulation
pixel distribution from a number of upper loss pixels and a number
of accumulation pixels.
14. The structured light-based 3D camera system of claim 13,
wherein the calculating the upper loss pixel distribution and the
accumulation pixel distribution comprise excluding states in which
a brightness value of a pixel associated with the first reference
image and a brightness value of a pixel associated with the second
reference image are less than a predetermined threshold brightness
value.
15. The structured light-based 3D camera system of claim 9, wherein
the first pattern and the second pattern are the same.
16. The structured light-based 3D camera system of claim 9, wherein
the first pattern and the second pattern are different.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] The present application is a continuation of U.S. patent
application Ser. No. 16/558,151, entitled "Method and System for
Determining Optimal Exposure Time and Number of Exposures in
Structured Light-Based 3D Camera", filed on Sep. 1, 2019, which is
a continuation of U.S. patent application Ser. No. 15/334,314,
entitled "Method and System for Determining Optimal Exposure Time
and Number of Exposures in Structured Light-Based 3D Camera", filed
on Oct. 26, 2016, now U.S. Pat. No. 10,404,968, which claims
priority under 35 U.S.C. .sctn. 119 to Korean Patent Application
No. 10-2015-0149156 (Filed on Oct. 27, 2015), the entirety of all
two of which are expressly incorporated herein by reference.
BACKGROUND OF THE INVENTION
Field of the Invention
[0002] The present invention relates to a method for determining an
exposure of a structured light-based 3D camera.
Description of the Related Art
[0003] Generally, technologies for reconstructing a 3D image of an
object using one or more cameras may be largely classified as
either being active or passive techniques. Examples of active
techniques include laser triangulation and structured light. An
example of a passive technique is stereovision. Active techniques
are typically preferred for research or industrial purposes because
they may achieve higher precision scanning than passive
techniques.
[0004] A 3D camera system for use with structured light can be
considered as a modification of a stereo camera system. In
particular, unlike a stereo camera system, in which two identical
cameras are used, a 3D camera system used for structured light is
configured such that one camera is replaced by a projection device
such, as a beam projector. Accordingly, in a 3D camera system used
for structured light scanning, one or more patterns are projected
onto an object using a projection device, an image of the object
onto which the pattern is projected is captured using an
image-capturing device, such as a camera, and 3D information about
the object is acquired by analyzing the captured image.
[0005] In other words, because a stereo camera system passively
uses features extracted from an image but a structured light-based
camera system actively projects one or more patterns onto an object
using a projection device and uses the projected pattern to
distinguish features of the object, a structured light-based camera
system has a fast processing speed and high spatial resolution.
Thanks to these advantages, the structured light-based camera
system is widely used in various fields, such as object modeling
and recognition, 3D measurement, industrial inspection, reverse
engineering, and the like.
[0006] However, structured light-based camera systems have the
following general problems.
[0007] Generally, in the real world, objects have various
reflection coefficients. Accordingly, when patterns are projected
from a projection device onto an object having a low reflection
coefficient (for example, a black ball), it is difficult to
correctly acquire the projected patterns using a capturing device.
Conversely, when patterns are projected onto an object having a
high reflection coefficient (for example, a white ball having a
glossy surface), because the gloss causes saturation of patterns
(seen as blurred patterns) in the image acquired using a capturing
device, it is difficult to acquire accurate patterns. Also, light
conditions of a real environment may affect the image acquisition
of patterns. Additionally, it may be difficult to acquire accurate
patterns in a real environment that is too bright or too dark.
SUMMARY OF THE INVENTION
[0008] In order to solve the above problems, it is necessary to
adjust the exposure of a camera by controlling the aperture of a
camera depending on the environment.
[0009] With the introduction of 3D printers, there is an increase
in the use of structured light-based cameras by laypersons.
However, because it is not easy to adjust the parameters of such
cameras to adapt in real time to changing environmental conditions,
it is difficult for laypersons, who are not experts, to use such
cameras. Also, with the development of structured light-based 3D
cameras, there is an increase in the demand for acquiring accurate
3D data in industrial fields. Here, in order to construct an
automated system for use in industry, a structured light-based
camera is fixed to an industrial robot, whereby accurate 3D data
may be acquired. However, because most objects to be recognized in
industrial fields are made of metal, which has a low reflection
coefficient, the construction of an automated system for use in
industrial fields is impeded.
[0010] Therefore, it is necessary to automatically adjust the
number of exposures and the exposure time of a 3D camera depending
on the environment.
[0011] The present invention provides a method and system for
determining the optimal exposure time and the number of exposures
by which, when an object having a low reflection coefficient and an
object having a high reflection coefficient are mixed together in
an environment, the optimal exposure time of a structured
light-based 3D camera is determined, and by which, when it is
difficult to reconstruct 3D data using only a single exposure, two
exposure times having the optimal lengths and the optimal number of
the exposures are determined. An object of the present invention is
to improve reliability and to increase the extent to which 3D data
may be reconstructed using such a method and system.
[0012] According to one aspect of the present invention, there is
provided a method for determining an optimal exposure time and a
number of exposures in a structured light-based 3D camera system.
The method includes capturing a reference image onto which a first
pattern and a second pattern are projected, calculating a slope
related to brightness and an exposure time for a pixel of the
reference image and categorizing a state of the slope, creating an
upper loss pixel distribution chart and an accumulation pixel
distribution chart by calculating a number of upper loss pixels and
a number of accumulation pixels for an exposure time of the
structured light-based 3D camera system, and determining the
optimal exposure time and the number of exposures of the structured
light-based 3D camera system using the upper loss pixel
distribution chart and the accumulation pixel distribution
chart.
[0013] The first pattern and the second pattern may include at
least one of an all-black pattern and an all-white pattern.
[0014] The capturing the reference image may include determining a
first exposure time so as to prevent a brightness value of an
object having a high reflection coefficient from being greater than
250, and determining a second exposure time so as to prevent a
brightness value of an object having a low reflection coefficient
from being less than 10.
[0015] The calculating the slope and categorizing the state of the
slope may include defining a state of brightness depending on a
brightness value of a pixel in the reference image that is captured
with the first exposure time and the second exposure time.
[0016] The calculating the slope and categorizing the state of the
slope may further include defining the state of brightness as a
case in which the brightness value of the pixel in the reference
image captured with the first exposure time and the second exposure
time is less than 10, a case in which the brightness value of the
pixel in the reference image captured with the first exposure time
and the second exposure time is greater than 250, and a case in
which the brightness value is not less than 10 and is not greater
than 250.
[0017] The calculating the slope and categorizing the state of the
slope may further include calculating the slope related to the
brightness and the exposure time for each of the first pattern and
the second pattern depending on the state of brightness, and
categorizing the state of the slope.
[0018] The creating the upper loss pixel distribution chart and the
accumulation pixel distribution chart may include excluding states
in which the brightness value of the pixel is less than a
predetermined threshold brightness value depending on the state of
the slope for each of the first pattern and the second pattern when
predicting a pixel to be reconstructed.
[0019] The determining the optimal exposure time and the number of
exposures may include searching for an exposure time that first
satisfies a predetermined first criterion in the accumulation pixel
distribution chart, and determining whether a percentage of upper
loss pixels at the found exposure time satisfies a predetermined
second criterion.
[0020] The determining the optimal exposure time and the number of
exposures may further include a first step in which, when the
percentage of the upper loss pixels does not satisfy the
predetermined second criterion, whether the corresponding exposure
time is a maximum exposure time of the structured light-based 3D
camera system is determined; a second step in which, when the
corresponding exposure time is not the maximum exposure time of the
structured light-based 3D camera system, the corresponding exposure
time is increased by a predetermined amount; a third step in which
whether the percentage of the upper loss pixels at the increased
exposure time satisfies the predetermined second criterion is
determined again; and repeating the first, second, and third steps
until the percentage of the upper loss pixels satisfies the
predetermined second criterion or until the corresponding exposure
time becomes the maximum exposure time.
[0021] The determining the optimal exposure time and the number of
exposures may further include setting the number of exposures to a
single exposure when the percentage of the upper loss pixels
satisfies the predetermined second criterion.
[0022] The determining the optimal exposure time and the number of
exposures may further include, when the corresponding exposure time
is the maximum exposure time of the structured light-based 3D
camera system, setting the exposure time that first satisfies the
predetermined first criterion in the accumulation pixel
distribution chart as a long exposure time; and setting the
exposure time at which the percentage of the upper loss pixels
first satisfies the predetermined second criterion in the upper
loss pixel distribution chart as a short exposure time.
[0023] According to another aspect of the present invention, there
is provided a structured light-based 3D camera system. The system
includes a projection unit for projecting a pattern onto a target
object, a capturing unit for capturing an image that includes the
pattern projected from the projection unit, and a processing unit
for determining an optimal exposure time and a number of exposures
using a reference image acquired by the capturing unit, wherein the
processing unit includes an image acquisition unit for acquiring an
image onto which a pattern is projected; a slope calculation and
state categorization unit for calculating a slope related to
brightness and an exposure time for a pixel of the reference image
acquired from the image acquisition unit and for categorizing a
state of the slope; a distribution chart creation unit for creating
an upper loss pixel distribution chart and an accumulation pixel
distribution chart by calculating a number of upper loss pixels and
a number of accumulation pixels for the exposure time of the
structured light-based 3D camera system; and an optimal exposure
time and frequency determination unit for determining the optimal
exposure time and the number of exposures of the structured
light-based 3D camera system using the upper loss pixel
distribution chart and the accumulation pixel distribution
chart.
BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The above and other objects, features and other advantages
of the present invention will be more clearly understood from the
following detailed description when taken in conjunction with the
accompanying drawings, in which:
[0025] FIG. 1 is a flowchart of a method for determining the
optimal exposure time;
[0026] FIG. 2A and FIG. 2B are detailed flowcharts of the process
of calculating a slope having high reliability in a captured
reference image and of categorizing the state of the slope;
[0027] FIG. 3 is an example of a slope1/slope0 distribution
chart;
[0028] FIG. 4A is an example of an accumulation pixel distribution
chart;
[0029] FIG. 4B is an example of an upper loss pixel distribution
chart;
[0030] FIG. 5 is a flowchart of the process of determining an
exposure time and the number of exposures;
[0031] FIG. 6 is a schematic block diagram of a structured
light-based 3D camera system in which a method for determining the
optimal exposure according to the present invention is used;
[0032] FIG. 7 shows an experimental environment for comparing the
case in which a single exposure is set according to an existing
method for determining the optimal exposure with the case in which
two exposures are set according to the method proposed by the
present invention;
[0033] FIG. 8 is a table that shows an exposure time and the number
of reconstructed pixels when the scene of FIG. 7 is captured using
a structured light-based 3D camera system; and
[0034] FIGS. 9A, 9B and 9C show the result of reconstruction of 3D
data when the scene of FIG. 7 is captured using a structured
light-based 3D camera system.
DETAILED DESCRIPTION OF THE INVENTION
[0035] The present invention relates to methods and apparatus for
determining an optimal number of exposures and a duration thereof
in a structured light-based 3D camera system and a structured
light-based 3D camera system using such methods and apparatus.
[0036] The present invention may be variously changed, and may have
various embodiments, and specific embodiments will be described in
detail below with reference to the attached drawings. However, it
should be understood that those embodiments are not intended to
limit the present invention to specific disclosure forms and that
they include all changes, equivalents or modifications included in
the spirit and scope of the present invention.
[0037] In the present specification, a singular expression includes
a plural expression unless a description to the contrary is
specifically pointed out in context. In the present specification,
it should be understood that terms such as "include" or "have" are
merely intended to indicate that features, numbers, steps,
operations, components, parts, or combinations thereof are present,
and are not intended to exclude the possibility that one or more
other features, numbers, steps, operations, components, parts, or
combinations thereof will be present or added.
[0038] Hereinafter, embodiments of the present invention will be
described in detail with reference to the accompanying drawings. In
the following description of the present invention, the same
reference numerals are used to designate the same or similar
elements throughout the drawings, and repeated descriptions of the
same components will be omitted.
[0039] In the present invention, an exposure of a structured
light-based 3D camera means the amount of light that is introduced
when an image is captured using the camera. Generally, the exposure
of a camera is adjusted using an aperture and/or shutter speed of
the camera. This specification describes a method and system for
determining the optimal exposure (by adjusting the exposure time
and/or the number of exposures) of a structured light-based 3D
camera.
[0040] FIG. 1 is a flowchart of a method for determining the
optimal exposure time. Referring to FIG. 1, the method for
determining the optimal exposure of a structured light-based 3D
camera includes capturing a reference image at step S100,
calculating a slope and categorizing the state of the slope at step
S200, creating a distribution chart at step S300, and determining
the optimal exposure time and the number of exposures at step
S400.
[0041] In the capturing the reference image at step S100, images of
an object onto which pattern 0 and pattern 1 are projected,
respectively, are captured. Here, pattern 0 means an all-black
pattern (when a projection device is turned off), and pattern 1
means an all-white pattern. Here, in order to improve reliability
when a slope related to brightness and an exposure time is
calculated at step S200, two reference images are captured for each
of pattern 0 and pattern 1. Here, if only a single reference image
is used, reliability may be reduced because the brightness value of
a pixel becomes 255 even when the actual brightness value of a
pixel is greater than 255 (because the maximum brightness value of
a pixel in an 8-bit image is 255, the brightness beyond that is
also represented as 255). Also, if the brightness value of a pixel
is less than 10, because the percentage of noise in a
Signal-to-Noise Ratio (SNR) may be equal to or greater than 10%,
reliability may be reduced when a slope is calculated. Accordingly,
in order to calculate a reliable slope, the exposure time for
capturing a reference image is set as described below. First, in
order to prevent the brightness value of an object having a high
reflection coefficient from being greater than 250, the shutter
speed is set fast (that is, a short exposure time is set).
Conversely, in order to prevent the brightness value of an object
having a low reflection coefficient from being less than 10, the
shutter speed is set slow (that is, a long exposure time is
set).
[0042] In the calculating a slope and categorizing the state of the
slope at step S200, a reference image for calculating a reliable
slope is selected from among the captured reference images, a slope
is calculated for the selected reference image, and the state
thereof is categorized. In other words, a slope for pattern 0 and a
slope for pattern 1 may be calculated for all the pixels in the
reference image according to the processes of the flowchart in FIG.
2A and FIG. 2B.
[0043] In FIG. 2A and FIG. 2B, E.sub.1 means a short exposure time,
which is set in a camera when a reference image is captured, and
E.sub.2 means a long exposure time, which is set in the camera when
the reference image is captured. Also, I.sub.1 means the brightness
value of a pixel in the reference image, which is captured when
E.sub.1 is set, and I.sub.2 means the brightness value of a pixel
in the reference image, which is captured when E.sub.2 is set.
Because there is a linear relationship between I and E, the value
of E corresponding to the desired exposure time may be estimated
using the slope of I/E. Here, in order to acquire a reliable slope,
the state of brightness of a pixel is defined as follows. When
I<10, the state is defined as `under`, and when I>250, the
state is defined as `over`. When the state is neither `under` nor
`over`, the state is defined as `normal`. Depending on the values
of I.sub.1 and I.sub.2, there may be seven cases, as shown in FIG.
2A and FIG. 2B.
[0044] S201 is the case in which both the value of I.sub.1 and the
value of I.sub.2 are in the `normal` state, in which case the mean
of the two slopes for the brightness and the exposure time in the
`normal` state is set as the value of a slope.
[0045] S202 and S203 are the cases in which either the value of
I.sub.1 or the value of I.sub.2 is in the `normal` state, in which
case the slope for the brightness and the exposure time in the
`normal` state is set as the value of a slope.
[0046] S204 and S205 are the cases in which the value of I.sub.1
and the value of I.sub.2 are respectively in the `under` and `over`
states or in the `over` and `under` states. Here, the value in the
`over` state indicates the case in which the actual brightness
value cannot be estimated, and the value in the `under` state
indicates the case in which the percentage of noise is high but the
value is more reliable than the value in the `over` state.
Accordingly, the slope for the brightness and the exposure time in
the `under` state is set as the value of a slope.
[0047] S206 is the case in which both the value of I.sub.1 and the
value of I.sub.2 are in the `under` state. Here, because there is
less noise as the brightness value of a pixel is greater, the
greater brightness value is selected, and the slope for the
brightness and the exposure time corresponding to the selected
brightness value is set as the value of a slope.
[0048] S207 is the case in which both the value of I.sub.1 and the
value of I.sub.2 are in the `over` state, in which case the slope
for the brightness and the exposure time corresponding to the lower
brightness value is set as the value of a slope.
[0049] Next, in the creating a distribution chart at step S300,
using the difference between the slope that is calculated when
pattern 1 is projected and the slope that is calculated when
pattern 0 is projected, pixels that can be reconstructed and pixels
that cannot be reconstructed at each exposure time in the
structured light-based camera system are predicted (that is, the
number of upper loss pixels and the number of accumulation pixels
are calculated), whereby an upper loss pixel distribution chart and
an accumulation pixel distribution chart are acquired.
[0050] The structured light-based 3D camera may calculate 3D data
only when a patterned region, which includes patterns projected
from a projection device, is distinguishable from a non-patterned
region. Therefore, the difference between the brightness in the
patterned region and the brightness in the non-patterned region
must be greater than a threshold value I.sub.th.
[0051] In order to predict pixels that can be reconstructed and
pixels that cannot be reconstructed in the structured light-based
3D camera system, the Slope.sub.1/Slope.sub.0 distribution chart
shown in FIG. 3 is used.
[0052] In FIG. 3, Slope.sub.0 means the slope that is calculated
when pattern 0 is projected, and Slope.sub.1 is the slope that is
calculated when pattern 1 is projected.
[0053] p.sub.0 is determined using the following Equation (1):
p.sub.0=(255-I.sub.th)/E (1)
[0054] p.sub.1 is determined using the following Equation (2):
p.sub.1=255/E (2)
[0055] p.sub.2 is determined using the following Equation (3):
p.sub.2=(I.sub.th/E)+Slope.sub.0 (3)
[0056] In the Slope.sub.1/Slope.sub.0 distribution chart
illustrated in FIG. 3, a pixel at which Slope.sub.0 is less than
p.sub.0 and Slope.sub.1 is less than p.sub.1 and greater than
p.sub.2 is estimated as a pixel from which 3D data may be
reconstructed. In FIG. 3, a pixel at which Slope.sub.0 is greater
than p.sub.0 and Slope.sub.1 is greater than p.sub.1 is referred to
as an upper loss pixel, and a pixel at which Slope.sub.1 is greater
than p.sub.2 is referred to as an accumulation pixel.
[0057] When the maximum exposure time allowable in the structured
light-based 3D camera system is referred to as E.sub.M, the number
of pixels that satisfy the condition in which Slope.sub.1 is
greater than p.sub.2, which is calculated after E.sub.M is
substituted for E in Equation (3), that is, the number of pixels
that satisfy the condition of
Slope.sub.1>p.sub.2=(I.sub.th/E.sub.M)+Slope.sub.0, is the
maximum number of accumulation pixels. Here, the maximum number of
accumulation pixels is referred to as N.sub.M. Also, for all
exposure times (from E.sub.1 to E.sub.M) allowable in the
structured light-based 3D camera system, the number of accumulation
pixels at each exposure time may be calculated using Equation (3).
Here, the number of accumulation pixels at each exposure time
E.sub.n is referred to as N.sub.n. Accordingly, the percentage of
accumulation pixels to the maximum number of accumulation pixels is
calculated as
N n N M .times. 1 .times. 0 .times. 0 , ##EQU00001##
whereby the distribution chart shown in FIG. 4A may be
acquired.
[0058] Meanwhile, the number of pixels that satisfy the condition
of Slope.sub.0.gtoreq.(255-I.sub.th)/E and Slope.sub.1>255/E is
the number of upper loss pixels. Here, the number of upper loss
pixels calculated at each of all exposure times (from E.sub.1 to
E.sub.M) is referred to as L.sub.n. Accordingly, the percentage of
upper loss pixels to the maximum number of accumulation pixels is
calculated as
L n N M .times. 100 , ##EQU00002##
whereby the distribution chart shown in FIG. 4B may be
acquired.
[0059] Here, when the calculated Slope.sub.1 is in the `over` state
and the calculated Slope.sub.0 is in the `normal` state, the value
of (Slope.sub.1-Slope.sub.0)E may be greater than the actual value
thereof. In this case, because the condition of
(Slope.sub.1-Slope.sub.0)E>I.sub.th is satisfied, the
corresponding pixel may be assumed to be a pixel that can be
reconstructed. However, when Slope.sub.1 is in the `under` state
and Slope.sub.0 is in the `over` state, the calculated value of
(Slope.sub.1-Slope.sub.0)E may be less than the actual value
thereof, and the value may be less than I.sub.th. In this case, an
erroneous determination that a pixel can be reconstructed may be
made. Therefore, the case in which the value of
(Slope.sub.1-Slope.sub.0)E may be less than I.sub.th because the
value (Slope.sub.1-Slope.sub.0)E is less than the actual value
thereof may be excluded when a pixel that can be reconstructed is
predicted. Such a case is as follows.
[0060] (1) the case in which Slope.sub.1 is in the `over` state and
Slope.sub.0 is in the `over` state
[0061] (2) the case in which Slope.sub.1 is in the `normal` state
and Slope.sub.0 is in the `over` state
[0062] (3) the case in which Slope.sub.1 is in the `under` state
and Slope.sub.0 is in the `over` state
[0063] After the accumulation pixel distribution chart and the
upper loss pixel distribution chart, as shown in FIG. 4A and FIG.
4B, are acquired, the optimal exposure time and the number of
exposures are determined at step S400, as shown in the flowchart of
FIG. 5.
[0064] Referring to FIG. 5, S501 is the step in which the number of
exposures is set to a single exposure and the optimal duration
thereof is the exposure time at which the percentage of pixels lost
in the upper and lower regions is less than 1%. Here, the
percentage of lower loss pixels is
1 - ( N n N M .times. 1 .times. 0 .times. 0 ) . ##EQU00003##
Specifically, the exposure time at which the percentage of
accumulation pixels is 99.5% is searched for in the accumulation
pixel distribution chart at step S51, whether the percentage of
upper loss pixels is equal to or less than 0.5% is determined at
step S52, and whether the number of exposures is set to a single
exposure is determined at step S53.
[0065] Also, S502 is the step in which the number of exposures is
set to two exposures. Here, step S502 includes steps S54, S55, and
S56. At step S54, when it has been determined at step S52 that the
percentage of upper loss pixels is greater than 0.5%, whether the
exposure time found at step S51 is the maximum exposure time of the
camera is determined. At step S55, the exposure time at which the
percentage of accumulation pixels first becomes 99.5% is searched
for in the accumulation pixel distribution chart, and the found
exposure time is set as a long exposure time. Also, at step S56,
the exposure time at which the percentage of upper loss pixels is
equal to or less than 0.5% is searched for, and the found exposure
time and the long exposure time are set as the duration of the two
exposures. Specifically, if the exposure time found at step S51 is
not the maximum exposure time of the camera, the exposure time is
increased by 1 ms and the process returns to step S52. If the
exposure time found at step S51 is the maximum exposure time of the
camera, the exposure time found at step S51 and the newly found
exposure time at step S56 are set as the exposure times of the two
exposures.
[0066] In the flowchart of FIG. 5, if 99% of pixels can be
reconstructed through a single exposure, the number of exposures is
set to a single exposure, but if not, the number of exposures is
set to two exposures in order to enable an object having a low
reflection coefficient to be reconstructed at a long exposure time
and an object having a high reflection coefficient to be
reconstructed at a short exposure time, whereby 99% of pixels can
be reconstructed.
[0067] 3D Camera System Based on Structured Light
[0068] As an embodiment of the present invention, a structured
light-based 3D camera system 10 that uses the above-described
method for determining the optimal number of exposures and the
duration thereof may be provided.
[0069] Referring to FIG. 6, the structured light-based 3D camera
system 10 includes a projection unit 11, a capturing unit 12, and a
processing unit 13.
[0070] The projection unit 11 may project various patterns onto a
target object.
[0071] The capturing unit 12 may capture an image that includes the
pattern projected from the projection unit 11.
[0072] The processing unit 13 includes an image acquisition unit
131, a slope calculation and state categorization unit 132, a
distribution chart creation unit 133 and an optimal exposure time
and frequency determination unit 134.
[0073] The image acquisition unit 131 may acquire an image onto
which patterns, such as pattern 0 (an all-black pattern), pattern 1
(an all-white pattern) or the like, are projected.
[0074] The slope calculation and state categorization unit 132
calculates a slope related to brightness and an exposure time for
all the pixels in the reference image, onto which pattern 0 and
pattern 1 are projected, and categorizes the state of the slope,
the reference image being acquired from the image acquisition
unit.
[0075] The distribution chart creation unit 133 may acquire a
distribution chart by predicting pixels that can be reconstructed
and pixels that cannot be reconstructed (that is, by calculating
the number of upper loss pixels and the number of accumulation
pixels) at each exposure time using the difference between the
slope calculated when pattern 1 is projected and the slope
calculated when pattern 0 is projected.
[0076] The optimal exposure time and frequency determination unit
134 may determine the optimal exposure time and the optimal number
of exposures using the above-described method that uses the upper
loss pixel distribution chart and the accumulation pixel
distribution chart.
Experimental Example
[0077] FIG. 7 shows an experimental example for comparing the case
in which a single exposure is set according to an existing method
for determining the optimal exposure with the case in which two
exposures are set according to the method proposed by the present
invention. This example is configured to include an object having a
high reflection coefficient (the white object at the left side) and
an object having a low reflection coefficient (the black object in
the box).
[0078] FIG. 8 is a table that shows the exposure time and the
number of reconstructed pixels when the scene of FIG. 7 is captured
using a structured light-based 3D camera system. Here, the number
of exposures is set to a single exposure, the duration thereof is
set to a range from 1 ms to 100 ms, having an interval of 3 ms, and
the number of pixels that are actually reconstructed into 3D data
is measured. When the exposure duration is set to 28 ms, the
maximum number of pixels are reconstructed, in which case the
number of reconstructed pixels is 237,536.
[0079] FIGS. 9A, 9B and 9C show the result of reconstruction of 3D
data when the scene of FIG. 7 is captured using a structured
light-based 3D camera system. When the method proposed in the
present invention is applied to the scene of FIG. 7, the necessary
number of exposures is set to two exposures, and the durations
thereof are set to 3 ms and 84 ms, respectively. FIG. 9A is the
result of capturing an image when the exposure time is set to 3 ms,
in which case only a region that has a high reflection coefficient
is reconstructed. FIG. 9B is the result of capturing an image when
the exposure time is set to 84 ms, in which case only a region that
has a low reflection coefficient is reconstructed. FIG. 9C is the
result of a combination of the reconstructed pixels of FIG. 9A and
the reconstructed pixels of FIG. 9B, with a total of 251,900
reconstructed pixels. Consequently, it is confirmed that the
percentage of reconstructed pixels is increased by 6% when the
number of exposures is set to two exposures compared to when the
number of exposures is set to one exposure.
[0080] In a general 3D camera system based on structured light, it
is difficult to accurately detect patterns when the patterns are
projected onto an object having a reflection coefficient that is
too high or too low. This problem may be solved by adjusting the
number of exposures and/or the exposure time of a camera, whereby
reliable 3D data may be reconstructed. Therefore, the structured
light-based 3D camera system of the present invention may more
accurately reconstruct 3D data of an object having a low or high
reflection coefficient in industrial fields.
[0081] Also, the structured light-based 3D camera system of the
present invention sets two exposures having the optimal duration
when it is difficult to reconstruct 3D data only through a single
exposure in an environment in which an object having a low
reflection coefficient and an object having a high reflection
coefficient are both present, whereby reliability may be improved
and the extent to which 3D data are reconstructed may be
increased.
[0082] In the case of existing structured light-based 3D camera
systems, it is difficult for laypersons to adjust the aperture of
the camera because the adjustment thereof is not simple. However,
in the structured light-based 3D camera system of the present
invention, because the exposure time of the camera is automatically
determined, laypersons may easily use the camera.
[0083] The above embodiments have been described for illustrative
purposes and should not be construed as limiting the scope of the
present invention, which is defined by the accompanying claims.
Accordingly, other embodiments may fall into the range of technical
spirit defined by the accompanying claims. For example, those
skilled in the art may make various modifications without departing
from the scope of the present invention. Additionally, because some
of the above-described steps are independent from the order in
which the steps are performed, they may be performed in a different
order from the order in the above description. Also, some of the
steps may be omitted or other steps may be further included.
* * * * *