U.S. patent application number 10/637420 was filed with the patent office on 2004-05-20 for roi setting method and apparatus, electronic camera apparatus, program, and recording medium.
Invention is credited to Ikebe, Keiichi, Inoue, Takao, Kodama, Taku, Maki, Takashi, Sakuyama, Hiroyuki, Takahashi, Akira, Yamashiro, Ikuko, Yano, Takanori.
Application Number | 20040095477 10/637420 |
Document ID | / |
Family ID | 32017864 |
Filed Date | 2004-05-20 |
United States Patent
Application |
20040095477 |
Kind Code |
A1 |
Maki, Takashi ; et
al. |
May 20, 2004 |
ROI setting method and apparatus, electronic camera apparatus,
program, and recording medium
Abstract
An ROI setting apparatus including an ROI recognition unit and
an ROI control unit is provided. In one embodiment, the ROI
recognition unit contains multiple ROI recognition modules for
recognizing an ROI of image data according to various methods to
obtain a recognition result. The ROI control unit selects one ROI
recognition module out of the ROI recognition modules and sets ROI
information based on the recognition result. The ROI recognition
module may be selected according to an instruction from a user
input via an operation unit, or a scene type selected by a scene
selection switch of an image capture unit. The ROI control unit may
perform operations such as selecting, enlarging, or reducing the
ROI recognized by the ROI recognition module, or changing the ROI
recognition conditions according to the respective instructions
from the user input via the operation unit.
Inventors: |
Maki, Takashi; (Kanagawa,
JP) ; Ikebe, Keiichi; (Kanagawa, JP) ;
Sakuyama, Hiroyuki; (Tokyo, JP) ; Kodama, Taku;
(Kanagawa, JP) ; Yamashiro, Ikuko; (Kanagawa,
JP) ; Yano, Takanori; (Kanagawa, JP) ;
Takahashi, Akira; (Kanagawa, JP) ; Inoue, Takao;
(Kanagawa, JP) |
Correspondence
Address: |
Michael J. Mallie
BLAKELY, SOKOLOFF, TAYLOR & ZAFMAN LLP
Seventh Floor
12400 Wilshire Boulevard
Los Angeles
CA
90025
US
|
Family ID: |
32017864 |
Appl. No.: |
10/637420 |
Filed: |
August 8, 2003 |
Current U.S.
Class: |
348/222.1 ;
375/E7.161; 375/E7.162; 375/E7.172; 375/E7.182; 375/E7.226 |
Current CPC
Class: |
H04N 19/136 20141101;
G06K 2009/366 20130101; G06K 9/3233 20130101; H04N 19/14 20141101;
H04N 19/17 20141101; H04N 19/60 20141101; G06V 10/25 20220101; H04N
19/162 20141101; G06V 10/248 20220101 |
Class at
Publication: |
348/222.1 |
International
Class: |
H04N 005/228 |
Foreign Application Data
Date |
Code |
Application Number |
Aug 9, 2002 |
JP |
2002-232475 |
Claims
What is claimed is:
1. An ROI setting apparatus comprising: a plurality of ROI
recognition modules each for recognizing an ROI of image data
according to a predetermined method and to obtain a recognition
result; and an ROI control unit adapted to select an ROI
recognition module out of the plurality of ROI recognition modules
and set ROI information based on the recognition result obtained by
the selected ROI recognition module.
2. The ROI setting apparatus as claimed in claim 1, wherein the ROI
control unit selects the ROI recognition module according to an
instruction from a user.
3. The ROI setting apparatus as claimed in claim 1, wherein the ROI
control unit selects the ROI recognition module according to a
scene type selected by a user.
4. The ROI setting apparatus as claimed in claim 1, wherein the ROI
control unit sets the ROI information based on a region selected
out of a plurality of regions recognized by the selected ROI
recognition module, the region being selected according to an
instruction from a user.
5. The ROI setting apparatus as claimed in claim 1, wherein the ROI
control unit changes a size of a region recognized by the selected
ROI recognition module according to an instruction from a user.
6. The ROI setting apparatus as claimed in claim 1, wherein the
plurality of ROI recognition modules each include a recognition
condition and the ROI control unit changes the recognition
condition of the selected ROI recognition module according to an
instruction from a user.
7. The ROI setting apparatus as claimed in claim 1, wherein at
least one of the plurality of ROI recognition modules to detect a
region of the image data containing a face and using the region as
a base region to recognize the ROI.
8. The ROI setting apparatus as claimed in claim 1, wherein at
least one of the plurality of ROI recognition modules to detect a
region of the image data in which high frequency components are
concentrated and using the region as a base region to recognize the
ROI.
9. The ROI setting apparatus as claimed in claim 1, wherein at
least one of the plurality of ROI recognition modules to detect a
region of the image data in which patterns having a striking
contrast are concentrated and using the region as a base region to
recognize the ROI.
10. The ROI setting apparatus as claimed in claim 1, wherein at
least one of the plurality of ROI recognition module to detect a
region at a center portion of the image data containing an object
and using the region as a base region to recognize the ROI.
11. The ROI setting apparatus as claimed in claim 1, wherein at
least one of the plurality of ROI recognition modules to use an AF
evaluation region used in an image capturing operation of the image
data as a base image to recognize the ROI.
12. The ROI setting apparatus as claimed in claim 1, wherein at
least one of the plurality of ROI recognition modules to use an AE
evaluation region used in an image capturing operation of the image
data as a base image to recognize the ROI.
13. The ROI setting apparatus as claimed in claim 1, wherein at
least one of the plurality of ROI recognition modules to detect a
region of the image data in which movement occurs and using the
region as a base region to recognize the ROI.
14. The ROI setting apparatus as claimed in claim 1, wherein at
least one of the plurality of ROI recognition modules to extract a
region of the image data that is significantly brighter than a
corresponding region of monitoring image data obtained right before
an image capturing operation and using the region as a base region
to recognize the ROI.
15. The ROI setting apparatus as claimed in claim 1, wherein at
least one of the plurality of ROI recognition modules to detect a
high brightness region of the image data and using the region as a
base region to recognize the ROI.
16. The ROI setting apparatus as claimed in claim 1, wherein at
least one of the plurality of ROI recognition modules to detect a
high brightness region of the image data, subtracting from the
region a high brightness region portion at a predetermined portion
of the image data, and using the resulting region as a base region
to recognize the ROI.
17. The ROI setting apparatus as claimed in claim 1, wherein at
least one of the plurality of ROI recognition modules to recognize
the ROI based on a packet size of compressed data of the image
data.
18. An electronic camera apparatus comprising: an ROI setting
apparatus having a plurality of ROI recognition modules each for
recognizing an ROI of image data according to a predetermined
method to obtain a recognition result, and an ROI control unit to
select an ROI recognition module out of the plurality of ROI
recognition modules and set ROI information based on the
recognition result obtained by the selected ROI recognition
module.
19. A computer readable recording medium adapted to store a program
administering a computer to function as a plurality of ROI
recognition modules to recognize an ROI of image data according to
a predetermined ROI recognition method to obtain a recognition
result, and an ROI control unit for selecting one ROI recognition
module out of the plurality of ROI recognition modules and setting
ROI information based on the recognition result obtained by the
selected ROI recognition module, the ROI recognition module and the
ROI control unit being implemented in an ROI setting apparatus.
20. An ROI setting method comprising: selecting an ROI recognition
method out of a plurality of ROI recognition methods; recognizing
an ROI of image data using the ROI recognition method to obtain a
recognition result; and setting ROI information based on the
recognition result.
21. The ROI setting method as claimed in claim 20, wherein
selecting the ROI recognition method includes selecting the ROI
recognition method according to an instruction from a user.
22. The ROI setting method as claimed in claim 20, wherein
selecting the ROI recognition method includes selecting an ROI
recognition method according to a scene type selected by a
user.
23. The ROI setting method as claimed in claim 20, wherein setting
ROI information includes setting the ROI information based on a
region selected out of a plurality of regions recognized as the
ROI, the region being selected according to an instruction from a
user
24. The ROI setting method as claimed in claim 20, wherein
recognizing the ROI includes changing a size of a region recognized
as the ROI according to an instruction from a user.
25. The ROI setting method as claimed in claim 20, wherein each of
the plurality of ROI recognition methods includes a recognition
condition, and further wherein recognizing the ROI includes
changing a recognition condition according to an instruction from a
user.
Description
[0001] The present application claims priority to the corresponding
Japanese Application No. 2002-232475 filed on Aug. 9, 2002, the
entire contents of which are hereby incorporated by reference.
BACKGROUND OF THE INVENTION
[0002] 1. Field of the Invention
[0003] The present invention relates generally to the field of
image processing, and particularly to a technique for setting a
region of interest (ROI) of an image in an electronic processing
apparatus such as an electronic camera.
[0004] 2. Description of the Related Art
[0005] In an electronic apparatus such as a digital camera, a
captured image is usually compressed before it is recorded on a
recording medium.
[0006] Presently, JPEG is widely used for compressing image data;
however, a significant amount of research is being done to promote
the use of a new image compression technique, JPEG 2000 (ISO/IEC
FCD 15444-1). JPEG 2000 has a selective region image quality
improvement function (ROI function) for reducing the compression
rate (improving the image quality) of a region of interest (ROI) of
an image without reducing the compression rate of the overall
image.
[0007] This ROI function is particularly appealing to an electronic
camera. By setting an important region of a captured image as the
ROI, high image quality can be maintained in the important region
while the amount of data of the overall image other than the
important region can be significantly reduced.
[0008] To use the ROI function, it is necessary to specify an ROI
in each individual captured image. A publicly known technique for
specifying the ROI is disclosed in Japanese Patent Laid-Open
Publication No.2001-230947, for example. In this Japanese
application, an electronic camera having an EVF (electronic view
finder) screen that displays a guidance frame or a template image
is disclosed. The guidance frame or template image is used as a
base image for framing and capturing an image. Then, a region
within the guidance frame or a region other than the template image
is handled as the ROI in the compression process of the captured
image. Also, the ROI information is recorded on a header of a
corresponding image file, and thus the ROI can be displayed on the
rest of the reproduced image in the image reproduction process.
[0009] Also, Japanese Patent Laid-Open Publication No.2001-119625
discloses a digital camera that determines a range of a main object
of a captured image based on a user operation, handles this range
as an ROI, compresses the corresponding range at a lower
compression rate than that for the rest of the regions, and
describes the ROI information on a header of a corresponding
compressed image file. This Japanese patent application mentions
that the range of the main object may be determined automatically
based on edge or histogram information of the image. However, there
appears to be no mention of JPEG 2000 in this Japanese patent
application.
[0010] The ROI setting method using a guidance frame or a template
image as disclosed in Japanese Patent Laid-Open Publication
No.2001-230947 limits the flexibility of the framing and scene
capturing operation.
[0011] Also, with the method of manually setting the range of the
main object as the ROI as disclosed in Japanese Patent Laid-Open
Publication No.2001-119625, there is a problem in that the manual
operation can be quite troublesome and the method may not be used
for successive capturing of still images or capturing of moving
images. Although mention is made of automatically determining the
range of the main object based on the edge or histogram of the
image, a specific method for determining the ROI does not appear to
be disclosed in this Japanese patent application. The legitimacy of
the determined ROI depends upon such factor as this determination
method.
[0012] Also, generally speaking, a wide variety of scene capturing
operations need to be considered in automating the ROI setting
operation. Further, since user preferences may vary considerably,
it may be more practical to enable a user to intervene in the
capturing operation to a certain degree.
SUMMARY OF THE INVENTION
[0013] An ROI setting technique is disclosed. In one embodiment, an
ROI setting apparatus comprises ROI recognition modules each for
recognizing an ROI of image data according to a predetermined
method and to obtain a recognition result, and an ROI control unit
adapted to select an ROI recognition module out of the plurality of
ROI recognition modules and set ROI information based on the
recognition result obtained by the selected ROI recognition
module.
BRIEF DESCRIPTION OF THE DRAWINGS
[0014] FIG. 1 is a schematic block diagram illustrating an
embodiment of the present invention;
[0015] FIG. 2 is a schematic block diagram illustrating JPEG 2000
image compression/decompression algorithms;
[0016] FIGS. 3A-3D are diagrams illustrating two-dimensional
wavelet conversion processes;
[0017] FIG. 4 is a diagram showing a configuration of compressed
image data (code stream) generated according to the JPEG 2000;
[0018] FIG. 5 is a flowchart of an ROI setting operation;
[0019] FIG. 6 is a flowchart of an operation shown in FIG. 5;
[0020] FIG. 7 is a flowchart of the operation of an ROI recognition
module (1);
[0021] FIG. 8 is a flowchart of the operation of an alternative ROI
recognition module (2);
[0022] FIG. 9 is a flowchart of the operation of another
alternative ROI recognition module (3);
[0023] FIG. 10 is a flowchart of the operation of another
alternative ROI recognition module (4);
[0024] FIG. 11 is a flowchart of the operation of another
alternative ROI recognition module (5);
[0025] FIG. 12 is a flowchart of the operation of another
alternative ROI recognition module (6);
[0026] FIG. 13 is a flowchart of the operation of another
alternative ROI recognition module (7);
[0027] FIG. 14 is a flowchart of the operation of another
alternative ROI recognition module (8);
[0028] FIG. 15 is a flowchart of the operation of another
alternative ROI recognition module (9);
DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0029] An ROI setting apparatus and method that can include a more
practical ROI setting function in an image processing apparatus
such as an electronic camera apparatus are described. One
embodiment of the present invention provides an electronic camera
apparatus such as a digital camera with a more practical ROI
setting function.
[0030] In one embodiment of the present invention, an ROI setting
apparatus includes: multiple ROI recognition modules each adapted
for recognizing an ROI of image data according to a predetermined
method and thereby obtaining a recognition result; and an ROI
control unit adapted to select an ROI recognition module out of the
ROI recognition modules and set ROI information based on the
recognition result obtained by the selected ROI recognition
modules. The term "module," for purposes herein, may comprise
hardware (e.g., circuitry, dedicated logic, etc.), software (such
as is run on a general purpose computer system or a dedicated
machine), or a combination of both. The terms "module" and "unit"
may be used interchangeably.
[0031] Since multiple different ROI recognition modules
corresponding to ROI recognition methods can be used, a suitable
ROI setting operation can be performed for various image data of
various captured scenes. Notably, the ROI setting apparatus may be
used in a successive capturing of still images or a capturing of
moving images.
[0032] Further, in one embodiment, the ROI control unit may select
the ROI recognition module according to an instruction from a user.
The user is able to change the ROI recognition method as desired so
that a desired ROI setting operation may be performed.
[0033] In an embodiment of the present invention, the ROI control
unit may select the ROI recognition module according to a scene
type selected by a user. The user can select a scene type and the
ROI can be set according to a higher quality ROI recognition method
for this scene type.
[0034] Further, in an embodiment of the present invention, the ROI
control unit may set the ROI information based on a region selected
out of a plurality of regions recognized by the selected ROI
recognition module, the region being selected, for example,
according to an instruction from a user. The user is able to make
corrections on the automatically recognized ROI to set an ROI
suiting the preference of the user.
[0035] Additionally, in an embodiment of the present invention, the
ROI control unit may enlarge or reduce a region recognized by the
selected ROI recognition module according to an instruction from a
user. The user is able to make corrections on the automatically
recognized ROI to set an ROI suiting the preference of the
user.
[0036] Also, in an embodiment of the present invention, the ROI
recognition module may each include a recognition condition, and
the ROI control unit may change the recognition condition of the
selected ROI recognition module according to an instruction from a
user. The user is able to adjust the ROI recognition condition to
set an ROI suiting the preference of the user.
[0037] Further, in an embodiment of the present invention, at least
one of the ROI recognition modules may detect a region of the image
data containing a face and using the region as a base region to
recognize the ROI. A face portion of a portrait image, for example,
can be set as the ROI.
[0038] Additionally, in an embodiment of the present invention, at
least one of the ROI recognition modules may detect a region of the
image data in which high frequency components are concentrated and
using the region as a base region to recognize the ROI. An
intricate portion of a landscape image, for example, can be set as
the ROI.
[0039] Also, in an embodiment of the present invention, at least
one of the ROI recognition modules may detect a region of the image
data in which patterns having a striking contrast are concentrated
and using the region as a base region to recognize the ROI. A
region of the image containing a barcode or letters can be set as
the ROI.
[0040] Additionally, in an embodiment of the present invention, at
least one of the ROI recognition modules may detect a region at a
center portion of the image data containing an object and using the
region as a base region to recognize the ROI. A region containing
an object, which is oftentimes at the center portion of the image,
can be set as the ROI.
[0041] Also, in an embodiment of the present invention, at least
one of the ROI recognition modules may use an AF evaluation region
used in an image capturing operation of the image data as a base
image to recognize the ROI. A region containing an object on which
the focus is directed can be set as the ROI.
[0042] Additionally, in an embodiment of the present invention, at
least one of the ROI recognition modules may use an AE evaluation
region used in an image capturing operation of the image data as a
base image to recognize the ROI. A region containing an object on
which the exposure is directed can be set as the ROI.
[0043] Also, in an embodiment of the present invention, at least
one of the ROI recognition modules may detect a region of the image
data in which movement is perceived and using the region as a base
region to recognize the ROI. A region containing the moving object
in a captured image of a moving object can be set as the ROI.
[0044] Additionally, in an embodiment of the present invention, at
least one of the ROI recognition modules may extract a region of
the image data that is significantly brighter than a corresponding
region of monitoring image data obtained right before an image
capturing operation and using the region as a base region to
recognize the ROI. A portion of a captured flash image containing
the object, which is positioned close by, can be set as the
ROI.
[0045] Also, in an embodiment of the present invention, at least
one of the ROI recognition modules may detect a high brightness
region of the image data and using the region as a base region to
recognize the ROI. A bright portion of the image containing the
object can be set as the ROI.
[0046] Additionally, in an embodiment of the present invention, at
least one of the ROI recognition modules may detect a high
brightness region of the image data, subtracting from the region a
high brightness region portion at a predetermined portion of the
image data, and using the resulting region as a base region to
recognize the ROI. A bright portion of the image except for upper
portion of the image representing the sky can be set as the
ROI.
[0047] Also, in an embodiment of the present invention, at least
one of the ROI recognition modules may recognize the ROI based on a
packet size of compressed data of the image data. As is described
below, according to JPEG 2000, tile data are packets of respective
regions called precincts, and a precinct with a large packet size
(amount of codes) is likely to be an important portion of the
image. The ROI can be set based on the packet size of each
precinct.
[0048] In another embodiment, an electronic camera apparatus
includes an ROI setting apparatus as described herein. In one
embodiment, an optimum ROI can be set for a variety of captured
scenes and the ROI can also be set according to the preference of
the user
[0049] In one embodiment, a program administers a computer to
function as multiple ROI recognition modules and an ROI control
unit of the ROI setting apparatus. Further, in another embodiment,
a computer readable recording medium stores a program according to
the present invention. By using such program and/or recording
medium, the ROI setting apparatus of the present invention can be
realized by a computer such as a personal computer or a
microcomputer.
[0050] In another embodiment of the present invention, an ROI
setting method includes: selecting an ROI recognition method out of
multiple ROI recognition methods; recognizing an ROI of image data
according to the selected ROI recognition method to obtain a
recognition result; and setting ROI information based on the
obtained recognition result
[0051] In embodiments of the present invention described below,
algorithms conforming to JPEG 2000 are used in the compression
process of an image. Image compression algorithms according to JPEG
2000 are descried in detail in Next Generation Image Encoding
Technique JPEG 2000, Yasuyuki Nomizu, Triceps, Co. Ltd., for
example. In the following, a brief description of the algorithms is
given with reference to FIGS. 2 through 4.
[0052] FIG. 2 is a schematic block diagram illustrating image
compression/decompression processes according to JPEG 2000. Herein,
image data being subjected to the compression process (e.g., image
data of each frame in a case where moving images are handled) are
divided into rectangular regions called tiles that do not overlap
with one another. In a DC level/color space
conversion/de-conversion unit 200, a level shift operation and a
color space conversion process are performed in order to raise the
compression rate. The level shift operation involves subtracting
half a dynamic range from an integral value such as an RGB value
that has no codes. The level shift operation is not performed on an
integral value having a code, and thus, if the input image data
corresponds to YCrCb data, for example, the Cr and Cb components,
which have attached codes, do not go through the level shift
operation. The color space conversion process involves converting
input image data into YCrCb data when the input image data
correspond to RGB data or CMY data. It is noted that in an
alternative embodiment, this DC level/color space
conversion/de-conversion unit 200 may be omitted.
[0053] Then, in a two-dimensional wavelet conversion/de-conversion
unit 202, a two-dimensional wavelet conversion (discrete wavelet
transform: DWT) is performed on the image data of each component of
each tile that has been processed by the DC level/color space
conversion/de-conversion unit 200.
[0054] FIGS. 3A-3D are diagrams illustrating an exemplary
two-dimensional wavelength conversion process in a case where the
decomposition level is 3. Specifically, with the two-dimensional
wavelet conversion process, a tile image shown in FIG. 3A is
divided into sub bands 1LL, 1HL, 1LH, and 1HH as shown in FIG. 3B.
Then, by performing a further two-dimensional wavelet conversion on
the coefficients of the sub band 1LL, the sub band 1LL is divided
into sub bands 2LL, 2HL, 2LH, and 2HH as shown in FIG. 3C. Then, by
performing a further two-dimensional wavelet conversion on the
coefficients of the sub band 2LL, the sub band 2LL is divided into
sub bands 3LL, 3HL, 3LH, and 3HH, as shown in FIG. 3D.
[0055] Referring back to FIG. 2, each of the sub band coefficients
obtained from a recursive division (octave division) of the low
frequency components (coefficients of sub band LL) are quantized by
a quantization/de-quantization unit 204, after which the data is
encoded by an entropy coding/decoding unit 206. In JPEG 2000, both
reversible compression (lossless compression) and irreversible
compression (lossy compression) are possible. In the case of a
reversible compression, the quantization step width is 1, and
quantization is not actually performed at this stage.
[0056] The entropy coding/decoding unit 206 performs entropy coding
on wavelet coefficients. In the above entropy coding, a block-base
bit plane coding scheme called EBCOT (Embedded Block Coding with
Optimized Truncation) including block division, coefficient
modeling, and binary arithmetic coding is used. The wavelet
coefficients being subjected to coding are either positive or
negative integers, and while a scanning operation of these
coefficients is performed in a designated order, the coding is
performed on the coefficients represented in absolute values one
bit plane at a time from a higher bit plane to a lower bit
plane.
[0057] A code sequence generated at the entropy coding/decoding
unit 206 is sent to the tag processing unit 208 where the code
sequence is arranged into a code stream with tag information
attached thereto, after which the code stream is output.
[0058] FIG. 4 is a diagram illustrating a configuration of the code
stream (compressed image data). As is shown in the drawing, the
code stream begins with an SOC marker, which is followed by a main
header describing a parameter for the coding operation or a
parameter for the quantization process, for example. Then, the code
stream of each tile follows. The code stream of each tile starts
with an SOT marker, followed by a tile header, then an SOD marker,
and then tile data. Further, an EOC marker indicating the end of
the code stream is placed after the last tile data set in the code
stream. Each set of tile data corresponds to a packet of the codes
of the coefficients of the sub bands in each region called a
precinct.
[0059] Herein, it is noted that the algorithms according to JPEG
2000 have many features other than that of realizing excellent
image quality at a high compression rate (low bit rate).
[0060] One feature of the JPEG 2000 algorithms is the
above-described selective region image quality improvement function
(ROI function). According to the basic method of JPEG 2000 JPEG
2000 Part 1), an ROI method of shifting the wavelet coefficient
values of a selected region to a higher bit plane and shifting the
wavelet coefficients of the rest of the regions to a lower bit
plane before encoding the wavelet coefficients (max-shift method)
is used. Alternatively, the ROI function may be realized in the
quantization process by quantizing the wavelet coefficient values
of the ROI using more intricate quantization steps than those for
the rest of the regions. In either case, the ROI method can realize
a reduction in the overall amount of codes without degrading the
image quality of the ROI of the image.
[0061] Referring back to FIG. 2, the decompression process of the
code stream is the reverse process of the compression process.
Namely, tag information of an input code stream is read and broken
down into a code sequence of the respective components of the
respective tiles by use of the tag processing unit 208. Then, the
resulting sequence is decoded into wavelet coefficients in the
order according to the tag information by use of the entropy
coding/decoding unit 206. The decoded wavelet coefficients are
de-quantized by the quantization/de-quantization unit 204, after
which a two-dimensional wavelet de-conversion is performed on the
de-quantized data by use of the two-dimensional wavelet
conversion/de-conversion unit 202 so that the image data of the
respective components of the respective tiles are reproduced. Then
the DC level/color space conversion/de-conversion unit 200 performs
the reverse conversion of the conversion performed in the
compression process on the image data of the respective components
of the respective tiles, after which the image is output.
[0062] FIG. 1 is a schematic block diagram illustrating an
embodiment of the present invention. An electronic camera apparatus
shown in this drawing has an image capture unit 100, a storage unit
102, a display unit 104, a compression/decompression unit 106, a
write/read unit 108, an ROI setting apparatus that includes an ROI
control unit 120 and an ROI recognition unit 122, and an operation
unit 124 for a user to input instructions relating to the ROI
setting operation.
[0063] The image capture unit 100 captures an image of an object
with an image sensor via an optical lens and outputs the captured
image data like a conventional image capture unit of a digital
camera. Further, the image capture unit 100 has a still image
capture mode and a moving image capture mode. Also, the image
capture unit 100 may have an auto-focus (AF) function, an
auto-exposure (AE) function, and a flash function, for example. In
the present embodiment, it is assumed that the electronic camera
apparatus uses the AF scheme and the AE scheme to determine the
focusing condition or the exposure condition based on a plurality
of evaluation regions of an image.
[0064] The storage unit 102 temporarily stores the image data
output by the image capture unit 100 or image data decompressed by
the compression/decompression unit 106. The display unit 104
displays the image data stored in the storage unit 102 and other
information on an LCD panel, for example, and is also used as an
EVF (electronic view-finder). The compression/decompression unit
106 performs the compression process on the image data and the
decompression process on the code stream using the algorithms
according to JPEG 2000 as described above. The write/read unit 108
writes the code stream output from the compression/decompression
unit 106 on a recording medium 110 as an image file, or reading the
image file from the recording medium 110.
[0065] The ROI recognition unit 122 is arranged to automatically
recognize the ROI of an image, and includes ten various ROI
recognition modules (1) through (10) each conforming to an ROI
recognition method differing from one another. The ROI recognition
method of each of these recognition modules will be described in
detail later on. The ROI control unit 120 definitively sets the ROI
information based on information relating to a selection of the ROI
recognition module of the ROI recognition unit 122, a control of
the selected ROI recognition module, and the ROI (candidate)
recognized by the ROI recognition module. The ROI control unit 120
also stores ROI recognition conditions (parameter values).
[0066] Instructions from a user relating to the ROI setting
operation are input from the operation unit 124 to the ROI control
unit 120. In the following, examples of the instructions input from
the operation unit 124 are given.
[0067] (a) instruction to perform an ROI setting operation
involving the user
[0068] (b) instruction, given during an ROI setting operation
involving the user, that instructs the selection of a particular
region in a case where a plurality of ROIs (candidates) have been
automatically recognized
[0069] (c) instruction, given during an ROI setting operation
involving the user, that instructs the enlargement/reduction of the
ROI (candidate) that has been automatically recognized
[0070] (d) instruction, given during an ROI setting operation
involving the user, that instructs the change of a parameter
relating to the ROI recognition process
[0071] (e) instruction to select an ROI recognition method
(module)
[0072] (f) instruction to automatically set the ROI of a still
image
[0073] (g) instruction to automatically set the ROI upon capturing
a moving image
[0074] Also, the image capture unit 100 supplies image capture
condition information such as image capture mode information
(information distinguishing between the still image capture mode
and the moving image capture mode), flash information (information
indicating the flash mode), AF information (information on the AF
evaluation region used in the focus evaluation for the image
capture operation), AE information (information on the AE
evaluation region used in the optimum exposure evaluation for the
image capturing operation), and scene selection information
(information indicating the type of scene selected by a scene
selection switch) to the ROI control unit 120.
[0075] FIG. 5 is a flowchart illustrating an ROI setting operation
involving the user that is performed, for example, each time a
still image is captured. In this case, in the still image capture
mode, the instruction (a) to perform the ROI setting operation
involving the user may be given during a monitoring operation (or
during display of through images) in which images captured by the
image capture unit 100 (monitoring images or through images) are
successively displayed on the display unit 104.
[0076] As is shown in FIG. 5, in step S1, the ROI control unit 120
selects one of the ROI recognition modules implemented in the ROI
recognition unit 122. In a case where a particular scene type is
selected by the scene selection switch of the image capture unit
100, an ROI recognition module for the selected scene type
according to the scene selection information is selected. In a case
where the scene type is not selected, a default ROI recognition
module or an ROI recognition module predesignated by the user is
selected.
[0077] The image capture unit 100 performs scene monitoring until a
release switch is pushed. Herein, the image capture unit 100
successively outputs data of the monitoring images, and this data
is stored in the storage unit 102. Then, the monitoring images are
displayed on the display unit 104. It is noted that a monitoring
image is usually an image in which some of the scanning lines are
left out.
[0078] When the release switch is pushed, the image capture unit
100 captures a still image. The data of the captured still image
without the scanning lines being left out are stored in the storage
unit 102, after which this captured image is displayed on the
display unit 104. Herein, data of at least one monitoring image
obtained right before the image capturing operation are also stored
in the storage unit 102.
[0079] In step S2, the ROI control unit 120 administers the
selected ROI recognition module to perform an ROI recognition
process on this captured still image data, and also administers the
display unit 104 to display the recognize ROI on the captured still
image. At this point, it is possible to enlarge/reduce or select
the automatically recognized ROI, or to make changes in the
parameter values determining the ROI recognition condition, for
example. The detailed descriptions of such operations will be
described later on.
[0080] In step S3, if the user inputs an instruction to confirm the
ROI via the operation unit 124, then in step S5, the ROI control
unit 120 confirms the ROI information based on the current ROI, and
sets this information to the compression/decompression unit 106 and
the write/read unit 108. Also, the ROI control unit 120 stores the
parameter values used for this ROI recognition process. Then, in
step S6, the captured still image data stored in the storage unit
102 are compressed by the compression/decompression unit 106, and
the resulting code stream is written on the recording medium 110 as
an image file by use of the write/read unit 108. In this
compression process, the ROI is compressed at a compression rate
lower than that for the rest of the regions in accordance with the
set ROI information. Also, the ROI information is described in the
header of the image file.
[0081] Further, if the user inputs the instruction (e) to select
the ROI recognition module after step S2, the ROI recognition
module can be altered. In response to his instruction, the ROI
control unit 120 selects an ROI recognition module other than the
currently selected ROI recognition module in step S4, and
administers the newly selected ROI recognition module to perform
the ROI recognition process over again and display the recognized
ROI on the display unit 104.
[0082] FIG. 6 is a flowchart illustrating detailed process steps
for the operation of step 2.
[0083] First, in step S11, the ROI control unit 120 sets the
parameter values (described below) for controlling the recognition
conditions for the selected ROI recognition module (method). The
parameter values may be either default values or values previously
used and stored. Then, in step S12, the ROI control unit 120
administers the ROI recognition module to perform ROI recognition
and also administers the display unit 104 to display the recognized
ROI (still an ROI candidate at this point) on the captured
image.
[0084] Thus, the user can see the displayed ROI candidate, and can
then input instructions (b), (c), and/or (d) as necessary or
desired. More specifically, when a plurality of ROI candidates are
recognized, the user may, for example, use a cursor on the screen
of the display unit 104 to specify the necessary or desired region
and the unnecessary region (instruction (b)). Also, the user may
use an enlargement/reduction indicator displayed on the screen of
the display unit 104 to instruct the enlargement/reduction of the
required region from the recognized ROI candidate, or instruct the
enlargement/reduction of each individual ROI candidate using a
cursor (instruction (c)). Also, the user may use a parameter value
indicator displayed on the display unit 104 to instruct an increase
or decrease of a parameter value controlling the ROI recognition
condition (instruction (d)).
[0085] In step S14, the ROI control unit 120 determines the type of
instruction input from the operation unit 124, and performs the
appropriate operation according to the determined instruction.
[0086] In a case where the region selection instruction (b) is
input, the process moves on to step S15 where the ROI control unit
120 invalidates the region(s) selected by the user from among the
plurality of recognized ROI candidates. Thus, the user is able to
validate the preferred ROI candidate when a plurality of ROI
candidates are recognized. This process may be performed
repeatedly, and after each time this process is performed, the
display is rearranged in step S12.
[0087] In a case where the region enlargement/reduction instruction
(c) is input, the process moves on to step S16 where the ROI
control unit 120 enlarges or reduces the ROI candidate according to
this instruction. Thus, the user is able to enlarge or reduce the
automatically recognized ROI candidate as necessary or desired.
This process may be performed repeatedly, and after each time this
process is performed, the display is rearranged in step S12.
[0088] In a case where the parameter alteration instruction (d) is
input, the process moves on to step S17 where the ROI control unit
120 changes the parameter value, and administers the ROI
recognition module to perform a recognition process once more so
that the newly recognized ROI candidate is displayed on the display
unit 104 in step S12.
[0089] In the following, each of the ROI recognition modules
implemented in the ROI recognition unit 122 and the respective
parameters relating to each of the ROI recognition module will be
described.
[0090] Referring to FIG. 7, according to ROI recognition module
(1), a face portion recognition process is performed on the image
data in step S21 so that the eyes, nose, mouth and other facial
features of a person are particularly taken into consideration and
a center region of a person's face is detected from the image. Then
in step S22, a range having significant contrast and including the
center region (base region) is detected, and in step S23, the
detected range is slightly enlarged and this enlarged range is
recognized as an ROI candidate. Herein, a significant contrast
determination threshold value and/or an extent of enlargement of
the range correspond to the parameter values of this ROI
recognition module (1). Further, this ROI recognition module (1) is
selected when a `person` is selected as the scene type by the scene
selection switch. Also, the range detected in step S22 may be used
as the ROI candidate without being enlarged.
[0091] Referring to FIG. 8, according to ROI recognition module
(2), a region of the image in which high frequency components are
concentrated is detected in step S31. Then in step S32, a range
having significant contrast and including this region (base region)
is detected, and in step S33, the detected range is slightly
enlarged and this enlarged range is recognized as an ROI candidate.
Herein, a high frequency component detection threshold value, the
significant contrast determination threshold value, and/or the
extent of enlargement of the range correspond to the parameter
values of this ROI recognition module (2). Further, the range
detected in step S32 maybe used as the ROI candidate without being
enlarged.
[0092] Referring to FIG. 9, according to ROI recognition module
(3), a region of the image in which patterns with striking contrast
such as a bar code or letters are concentrated is detected in step
S41. Then in step S42, a range having significant contrast and
including this region (base region) is detected, and in step S43,
the detected range is slightly enlarged and the enlarged range is
recognized as an ROI candidate. Herein, a striking contrast
determination threshold value, the significant contrast
determination threshold value, and/or the extent of enlargement of
the range correspond to the parameter values of this ROI
recognition module (3). Further, the range detected in step S42 may
be used as the ROI candidate without being enlarged.
[0093] Referring to FIG. 10, according to ROI recognition module
(4), an object in a center portion of the image is recognized in
step S51. Then, in step S52, a range having significant contrast
and including the recognized object region (base region) is
detected. Then, in step S53, the detected range is slightly
enlarged and this enlarged range is recognized as an ROI candidate.
Herein, the significant contrast determination threshold value
and/or the extent of enlargement of the range correspond to the
parameter values of this ROI recognition module (4). Further, the
range detected in step S52 can be used as the ROI candidate without
being enlarged.
[0094] Referring to FIG. 11, according to ROI recognition module
(5), a range having significant contrast and including a final AF
evaluation region used in a focus determination (provided by the
image capture unit 100 as AF information) is detected in step S61.
Then in step S62, the detected range is slightly enlarged and this
enlarged range is recognized as an ROI candidate. Herein, the
significant contrast determination threshold value and/or the
extent of enlargement of the range correspond to the parameter
values of this ROI recognition module (5). Further, the range
detected in step S61 may be used as the ROI candidate without being
enlarged. Also, in order to use the ROI recognition module (5) on
an image recorded in the recording medium 110, the AF information
obtained upon capturing this image may be recorded on the header of
the corresponding image file.
[0095] Referring to FIG. 12, according to ROI recognition module
(6), a range having significant contrast and including a final AE
evaluation region (provided by the image capture unit 100 as AE
information) used in an optimum exposure determination is detected
in step S71. Then, in step S72, the detected range is slightly
enlarged and this enlarged range is recognized as an ROI candidate.
Herein, the significant contrast determination threshold value
and/or the extent of enlargement of the range correspond to the
parameter values of this ROI recognition module (6). Further, the
range detected in step S71 may be used as the ROI candidate without
being enlarged. Also, in order to use the ROI recognition module
(6) on an image recorded in a recording medium 110, the AE
information obtained upon capturing this image may be recorded on
the header of the corresponding image file.
[0096] Referring to FIG. 13, according to ROI recognition module
(7), a region in which movement can be detected in a subjected
image based on a comparison between the subjected image and an idle
image immediately preceding the image capture operation, for
example, is detected in step S81. Then in step S82, a range having
significant contrast and including this region (base region) is
detected. Then in step S83, the detected range is slightly enlarged
and this enlarged region is recognized as an ROI candidate. Herein,
a movement determination threshold value, the significant contrast
determination threshold value, and/or the extent of enlargement of
the range correspond to the parameter values of this ROI
recognition module (7). Further, the range detected in step S82 may
be used as the ROI candidate without being enlarged. This ROI
recognition module is selected when a `moving object` is selected
as the scene type by the scene selection switch. Also, in order to
use the ROI recognition module (7) on a still image recorded in a
recording medium 110, the range in which movement occurs may be
detected in the image capturing stage and this information may be
recorded on the header of the corresponding image file.
[0097] Referring to FIG. 14, according to ROI recognition module
(8), which is used in a flash image capture operation, a region of
a subjected image (e.g., person or some other object) that is
significantly brighter than the corresponding region in an idle
image right before the flash is turned on is detected in step 91.
Then in step S92, a range having significant contrast and including
this region (base region) is detected, and in step S93, the
detected range is slightly enlarged and the enlarged range is
recognized as an ROI candidate. Herein, a threshold value for
determining a region to be brighter in the subjected image than in
the idle image, the significant contrast determination threshold
value, and/or the extent of enlargement of the range correspond to
the parameter values of this ROI recognition module (8). Further,
the range detected in step S92 may be used as the ROI candidate
without being enlarged.
[0098] Referring to FIG. 15, according to ROI recognition module
(9), a high brightness region of the image is detected in step
S101, and then in step S102, a range having significant contrast
and including this region (base region) is detected. Then, in step
S103, the detected range is slightly enlarged and this enlarged
range is recognized as an ROI candidate. Herein, a high brightness
determination threshold value, the significant contrast
determination threshold value, and/or the extent of enlargement of
the range correspond to the parameter values of this ROI
recognition module (9). Further, the range detected in step S102
may be used as the ROI candidate without being enlarged.
[0099] ROI recognition module (10) recognizes a region in a manner
identical to that of the ROI recognition module (9). However, the
ROI recognition module (10) further subtracts a high brightness
region portion having an area exceeding a predetermined value and
being situated in a predetermined region (e.g., an upper region) of
the image (e.g., a region corresponding to the sky) from the high
brightness region detected by the ROI recognition module (9), and
recognizes the resulting region as the ROI candidate.
[0100] Now, referring back to FIG. 5, an operation performed upon
giving the instruction (f) for automatically setting the ROI in the
still image capture mode will be described.
[0101] In this case, in step S1, the ROI control unit 120 selects
the ROI recognition module predesignated by the user or the ROI
recognition module corresponding to the scene type selected by the
scene selection switch. In step S2, an ROI candidate is recognized
by the ROI recognition module for each captured image. As for the
parameter values for the ROI recognition process, if parameter
values for the selected ROI recognition module are stored, the
stored parameter values are used, and if there are no parameter
values stored for the selected recognition module, default
parameter values are set to the ROI recognition module. Then, in
step S5, without waiting for the confirmation instruction from the
user, the recognized ROI candidate is confirmed as the ROI, and
this ROI information is set to the compression/decompression unit
106 and the write/read unit 108. Further, in this case, although a
change in the ROI recognition module may be made (step S4), a
selection of the region (FIG. 6, step S15), an
enlargement/reduction of the region (FIG. 6, step S16), and a
change in the parameter values (FIG. 6, step S17) may not be
performed.
[0102] On the other hand, when the instruction (g) for
automatically setting the ROI in the moving image capture mode is
given, an operation identical to that performed in response to the
instruction (f) for automatically setting the ROI in the still
image capture mode is performed, except for the fact that the ROI
is automatically set for each of the images of each of the captured
frames corresponding to the captured moving image. Further, for
moving image data, the image of each captured frame is
independently compressed as a still image, after which the image
data of all the frames are recorded on the recording medium 110 in
a motion JPEG 2000 (ISO/IEC FCD 15444-1) file format.
[0103] When the instruction (a) to perform ROI setting involving
the user is given in an image playback mode, the user may intervene
in setting the ROI of the still image or each of the frame images
of the moving image that is compressed through lossless compression
or low rate compression and recorded on the recording medium 110.
The operation performed in this case is descried below with
reference to FIG. 5.
[0104] In an exemplary image playback mode, thumbnail images or
lists of a plurality of images already recorded on the recording
medium 110 may be displayed on the display unit 104, and the user
is able to select a desired image.
[0105] When a particular image is selected, the corresponding image
file is read out by the write/read unit 108 and decompressed by the
compression/decompression unit 106. Then the decompressed image
data is stored in the storage unit 102, and the image is displayed
on the display unit 104. Then, in step S2, the ROI is recognized by
use of the selected ROI recognition module. The rest of the
operation is identical to the ROI setting operation of a direct
continuation of the image capture operation. However, the ROI
recognition module (5) and/or ROI recognition module (6) may not be
used unless AF information and/or AE information is described in
the header of the corresponding image file. Also, in the case of
replaying a still image, the ROI recognition module (7) and ROI
recognition module (8) may not be used. In the case of replaying a
moving image, the ROI recognition module (7) may be used since
movement can be detected between consecutive frame images, but the
ROI recognition module (8) may not be used.
[0106] If the ROI is already set in the replayed image, this ROI
information will be described in the header of the corresponding
image file. Thus, the ROI control unit 120 administers the display
unit 104 to display the ROI on the image according to this ROI
information.
[0107] In the above-described exemplary operations, the ROI
recognition unit 122 is used. Alternatively, an ROI recognition
method without using the ROI recognition unit 122 can also be
realized. An example of such operation is described below with
reference to FIG. 5.
[0108] When the above-described ROI recognition method without
using the ROI recognition unit 122 is selected in step S1 or S4,
the ROI control unit 120 administers the compression/decompression
unit 106 to perform a lossless or a low rate compression process on
the image data in step S2. The compression process may be performed
on the wavelet coefficients at decomposition level 1, for example.
Then, the size (amount of codes) of each packet in each tile data
is obtained by the compression/decompressio- n unit 106, and this
is compared with the determination threshold value. Then, a region
(precinct) corresponding to a packet having a size greater than the
determination threshold value is recognized as the ROI. In other
words, according to this operation, the ROI control unit 120
co-operates with the compression/decompression unit 106 to function
as the ROI recognition module. Alternatively, the method for
performing the ROI recognition may be set independently from the
ROI control unit 120. In step S6, the ROI information is set to the
compression/decompression unit 106 so that the compression process
is performed. Also, the ROI information is set to the write/read
unit 108 so that the information is described in the header of the
image file that is to be written on the recording medium 110.
[0109] Further, ROI recognition involving a user is also possible
in this ROI recognition method. In this case, first the recognized
ROI is displayed on the display unit 104. Herein, the user is able
to instruct selection of a region (corresponding to FIG. 6 step
S15), enlargement/reduction of a region (corresponding to FIG. 6
step S16), and/or change of a packet size determination threshold
value (corresponding to FIG. 6 step S17), for example. Then, when a
completion instruction is input by the user, the ROI control unit
120 confirms the ROI and sets the ROI information (FIG. 5 step
S5).
[0110] Further the compression/decompression unit 106 may
alternatively be arranged to perform the compression process
without taking the ROI into consideration, and the header of a
packet having a size exceeding the packet size determination
threshold value may be rewritten so that a precinct corresponding
to this packet is arranged to be the ROI.
[0111] As is evident from the above descriptions, the ROI control
unit 120, the ROI recognition unit 122, and a part or all of the
compression/decompression unit 106 shown in FIG. 1 (and/or their
functions and/or processes) may also be realized by programs
implemented in a computer such as a personal computer or a
microcomputer. Such programs and various recording (storage)
mediums on which the programs are recorded are also included in the
scope of the present invention. Also, the ROI setting apparatus and
method according to the present invention is not limited to
implementation in an electronic camera apparatus such as a digital
camera and its method; rather, the present invention may be applied
to various other types of image processing apparatuses that can
record an image through image compression.
[0112] According to the present invention, an image processing
apparatus such as an electronic camera apparatus is able to reduce
limitations in the framing of an image while also reducing the
trouble arising from a completely manual operation. Further, the
present invention may be adapted for various scenes and user
preferences, and is able to realize a practical ROI setting
function that is applicable to various operations such as a
successive capturing of still images or a capturing of a moving
image.
[0113] The present application is based on and claims the benefit
of the earlier filing date of Japanese priority application
No.2002-232475 filed on Aug. 9, 2002, the entire contents of which
are hereby incorporated by reference.
* * * * *