U.S. patent application number 12/301630 was filed with the patent office on 2009-04-30 for image coding device, image coding method, and image coding integrated circuit.
Invention is credited to Hiroshi Amano, Masayasu Iguchi, Masaki Maeda, Youji Shibahara, Takeshi Tanaka, Kenjiro Tsuda.
Application Number | 20090110077 12/301630 |
Document ID | / |
Family ID | 38723398 |
Filed Date | 2009-04-30 |
United States Patent
Application |
20090110077 |
Kind Code |
A1 |
Amano; Hiroshi ; et
al. |
April 30, 2009 |
IMAGE CODING DEVICE, IMAGE CODING METHOD, AND IMAGE CODING
INTEGRATED CIRCUIT
Abstract
When compression encoding processing of an image is performed in
units of macroblocks using pipeline structure, application of the
skip mode or the like according to MPEG4AVC to compression encode
an encoding target block requires motion vectors and the like of
adjacent blocks of the encoding target block. However, depending on
the structure of the pipeline stages, the motion vectors and the
like may not be determined. In such cases, the skip mode cannot be
applied to compression encode the encoding target block. The
present invention aims to solve this problem and (i) calculates all
motion information candidates, of the encoding target block,
corresponding to all motion information selectable by a previous
block of the encoding target block, and (ii) selects, as the motion
information of the encoding target block in the skip mode, the
motion information corresponding to the motion information
determined for the previous block.
Inventors: |
Amano; Hiroshi; (Osaka,
JP) ; Tanaka; Takeshi; (Osaka, JP) ; Maeda;
Masaki; (Osaka, JP) ; Tsuda; Kenjiro; (Osaka,
JP) ; Iguchi; Masayasu; (Hyogo, JP) ;
Shibahara; Youji; (Osaka, JP) |
Correspondence
Address: |
WENDEROTH, LIND & PONACK L.L.P.
1030 15th Street, N.W., Suite 400 East
Washington
DC
20005-1503
US
|
Family ID: |
38723398 |
Appl. No.: |
12/301630 |
Filed: |
May 23, 2007 |
PCT Filed: |
May 23, 2007 |
PCT NO: |
PCT/JP2007/060516 |
371 Date: |
November 20, 2008 |
Current U.S.
Class: |
375/240.16 ;
375/240.12; 375/E7.125 |
Current CPC
Class: |
H04N 19/105 20141101;
H04N 19/51 20141101; H04N 19/43 20141101; H04N 19/436 20141101;
H04N 19/61 20141101; H04N 19/137 20141101; H04N 19/52 20141101;
H04N 19/176 20141101; H04N 19/139 20141101; H04N 19/109
20141101 |
Class at
Publication: |
375/240.16 ;
375/240.12; 375/E07.125 |
International
Class: |
H04N 7/32 20060101
H04N007/32; H04N 7/26 20060101 H04N007/26 |
Foreign Application Data
Date |
Code |
Application Number |
May 24, 2006 |
JP |
2006-143840 |
Claims
1. An image encoding apparatus which compression encodes an image
in units of blocks of a predetermined size, the image encoding
apparatus comprising: a first processing unit operable to
sequentially generate candidate designation information
corresponding to each of the blocks, based on candidate designation
information corresponding to a prior block which has been processed
prior to the block, each candidate designation information
designating a plurality of prediction images as candidates for a
prediction image of the corresponding block, and a second
processing unit operable to, in parallel with generation of the
candidate designation information corresponding to the block by the
first processing unit, (i) select, as a prediction image of one of
the blocks which has been processed by the first processing unit, a
prediction image from among the plurality of prediction images
indicated by the candidate designation information corresponding to
the one of the blocks, based on prediction image designation
information designating a prediction image of a block processed
prior to the one of the blocks, (ii) outputs a difference signal
showing a difference which is between the selected prediction image
and an original image of the one of the blocks, and is to be
reflected in a result of compression encoding of the one of the
blocks, and (iii) generates prediction image designation
information designating the selected prediction image.
2. The image encoding apparatus of claim 1 compression encoding a
plurality of images which are input sequentially, wherein each
candidate designation information includes designation information
designating, as one of the candidates for the prediction image of
the corresponding block, one of the plurality of prediction images
which is generated based on a motion vector detected by searching
one of the plurality of images that is other than the image being
compression encoded.
3. The image encoding apparatus of claim 1, wherein each candidate
designation information includes designation information
designating, as a plurality of candidates among the candidates for
the prediction image of the corresponding block, out of the
plurality of prediction images, two or more prediction images
generated based on motion vectors of a plurality of adjacent blocks
including the prior block of the corresponding block, the plurality
of adjacent blocks being located in predetermined positions with
respect to the corresponding block.
4. The image encoding apparatus of claim 1 further comprising: a
third processing unit operable to cause each candidate designation
information to include designation information designating, as one
of the candidates for the prediction image of the corresponding
block, one of the plurality of prediction images which is generated
based on one or more pixel values of one or more pixels located in
predetermined positions with respect to the corresponding block,
wherein the second processing unit selects the prediction image for
the prior block of the corresponding block, from among the
plurality of prediction images indicated by the candidate
designation information caused by the third processing unit to have
the designation information.
5. The image encoding apparatus of claim 1, wherein each candidate
designation information is generated also based on prediction image
designation information of one of the blocks which has been
processed by the second processing unit.
6. The image encoding apparatus of claim 1, wherein each candidate
designation information is generated based on information
designating only some of the plurality of prediction images which
are candidates for the prediction image of the prior block of the
corresponding block.
7. The image encoding apparatus of claim 1 performing processing in
a pipeline structure composed of two stages that are a first stage
and a second stage, wherein the first processing unit processes the
first stage and includes: a motion detection unit operable to
generate, for each of the blocks, the candidate designation
information including designation information designating, as one
of the candidates for the prediction image of the block, one of the
plurality of prediction images which is generated based on a motion
vector detected by searching an image other than the image being
compression encoded; and a motion vector speculative calculation
unit operable to cause the candidate designation information of the
block to include designation information designating, as a
plurality of candidates among the candidates for the prediction
image of the block, out of the plurality of prediction images, two
or more prediction images generated based on motion vectors of a
plurality of adjacent blocks including the prior block, the
plurality of adjacent blocks being located in predetermined
positions with respect to the block, and the second processing unit
processes the second stage and includes: a motion vector
determination unit operable to, based on the prediction image
designation information of the block prior to the one of the
blocks, select a prediction image out of the one or more prediction
images designated by the designation information caused by the
motion vector speculative calculation unit to be included in the
candidate designation information of the one of the blocks; and a
mode judgement unit operable to (i) select the prediction image for
the one of the blocks, from among the prediction image selected by
the motion vector determination unit and the prediction image
indicated by the designation information generated by the motion
detection unit, (ii) output the difference signal, and (iii)
generate the prediction image designation information designating
the selected prediction image.
8. An image encoding method which compression encodes an image in
units of blocks of a predetermined size, the image encoding method
comprising: a first processing step of sequentially generating
candidate designation information corresponding to each of the
blocks, based on candidate designation information corresponding to
a prior block which has been processed prior to the block, each
candidate designation information designating a plurality of
prediction images as candidates for a prediction image of the
corresponding block, and a second processing method of, in parallel
with generation of the candidate designation information
corresponding to the block in the first processing step, (i)
selecting, as a prediction image of one of the blocks which has
been processed by the first processing unit, a prediction image
from among the plurality of prediction images indicated by the
candidate designation information corresponding to the one of the
blocks, based on prediction image designation information
designating a prediction image of a block processed prior to the
one of the blocks, (ii) outputting a difference signal showing a
difference which is between the selected prediction image and an
original image of the one of the blocks, and is to be reflected in
a result of compression encoding of the one of the blocks, and
(iii) generating prediction image designation information
designating the selected prediction image.
9. A computer program for causing a computer to perform compression
encoding processing which compression encodes an image in units of
blocks of a predetermined size, the compression encoding processing
comprising: a first processing step of sequentially generating
candidate designation information corresponding to each of the
blocks, based on candidate designation information corresponding to
a prior block which has been processed prior to the block, each
candidate designation information designating a plurality of
prediction images as candidates for a prediction image of the
corresponding block, and a second processing method of, in parallel
with generation of the candidate designation information
corresponding to the block in the first processing step, (i)
selecting, as a prediction image of one of the blocks which has
been processed by the first processing unit, a prediction image
from among the plurality of prediction images indicated by the
candidate designation information corresponding to the one of the
blocks, based on prediction image designation information
designating a prediction image of a block processed prior to the
one of the blocks, (ii) outputting a difference signal showing a
difference which is between the selected prediction image and an
original image of the one of the blocks, and is to be reflected in
a result of compression encoding of the one of the blocks, and
(iii) generating prediction image designation information
designating the selected prediction image.
10. An integrated circuit used for image encoding, the integrated
circuit compression encoding an image in units of blocks of a
predetermined size and comprising: a first processing unit operable
to sequentially generate candidate designation information
corresponding to each of the blocks, based on candidate designation
information corresponding to a prior block which has been processed
prior to the block, each candidate designation information
designating a plurality of prediction images as candidates for a
prediction image of the corresponding block, and a second
processing unit operable to, in parallel with generation of the
candidate designation information corresponding to the block by the
first processing unit, (i) select, as a prediction image of one of
the blocks which has been processed by the first processing unit, a
prediction image from among the plurality of prediction images
indicated by the candidate designation information corresponding to
the one of the blocks, based on prediction image designation
information designating a prediction image of a block processed
prior to the one of the blocks, (ii) outputs a difference signal
showing a difference which is between the selected prediction image
and an original image of the one of the blocks, and is to be
reflected in a result of compression encoding of the one of the
blocks, and (iii) generates prediction image designation
information designating the selected prediction image.
Description
TECHNICAL FIELD
[0001] The present invention relates to an image encoding apparatus
which compression encodes images, and in particular to speeding-up
of compression encoding processing.
BACKGROUND ART
[0002] MPEG4AVC (Motion Pictures Experts Group 4 Advanced Video
Coding) standard, which is a standard for compression encoding of
images, defines a skip mode and a spatial direct mode as encoding
modes.
[0003] When compression encoding (hereinafter, also referred to as
simply "encoding") a target image (hereinafter, referred to as
"encoding target image") in units of macroblocks of a predetermined
size, for example, of 16*16 pixels, both of these encoding modes
calculate the motion vector of the macroblock being encoded
(hereinafter, referred to as "encoding target block") and the
reference image number (hereinafter, a set of a motion vector and a
reference image number are referred to as "motion information")
identifying the reference image, based on the motion information of
a plurality of macroblocks (hereinafter, referred to as "adjacent
blocks") adjacent to the encoding target block, the reference image
being equivalent in size with the encoding target block and used to
predict an image of the encoding target block. After that, these
encoding modes encode the difference information which is
information showing the difference between the prediction image, of
the encoding target block, predicted based on the calculated motion
information, and the original image of the encoding target
block.
[0004] In other words, because the motion information of the
macroblock encoded in the skip mode or the spatial direct mode
(hereinafter, also referred to as "skip mode or the like") can be
calculated using the motion information of the adjacent blocks when
the macroblock is decoded, it is not necessary to encode the motion
information of the encoding target block. Thus, it is necessary
only to encode the difference information indicating the difference
between the prediction image of the encoding target block and the
original image of the encoding target block. Accordingly, code
amounts can be reduced by performing encoding in the skip mode or
the like.
[0005] One conventionally known technique processes macroblocks in
parallel using a pipeline structure in order to speed up encoding
processing of an image.
[0006] As described above, in order to encode the encoding target
block in the skip mode or the like, the motion information of the
adjacent blocks are required.
[0007] For example, assume a case where the pipeline is divided
into a first stage and a second stage, the first stage sequentially
calculating the motion information of each macroblock encoded in
the skip mode or the like, and the second stage, with respect to a
macroblock which has been processed by the first stage, judging
whether or not to perform the encoding in the skip mode or the like
based on the code amount in the skip mode. In this case, when the
processing by the first stage starts, if the processing by the
second stage has not been completed on all the adjacent blocks of
the encoding target block, the encoding mode has not been
determined for all of the adjacent blocks.
[0008] In other words, the motion information has not been
determined for all of the adjacent blocks, and thus, in the first
stage, the motion information of the encoding target block cannot
be calculated in the skip mode or the like.
[0009] An example of a technique for solving this problem is
adopted by an image encoding apparatus disclosed by Patent Document
1. In the following, the image encoding apparatus according to
Patent Document 1 is described.
[0010] The image encoding apparatus according to Patent Document 1
encodes the encoding target image in units of macroblocks which are
processed in parallel using a pipeline structure. Starting from the
upper left macroblock of the encoding target image, the image
encoding apparatus processes one row of macroblocks in the
horizontal direction, moves on to process the next row, and
sequentially processes the macroblocks up to the bottom right
macroblock.
[0011] In a case where the motion information of the macroblock
(hereinafter, referred to as "previous block") on the left of the
encoding target block has not been calculated when beginning to
process the encoding target block, the image encoding apparatus of
Patent Document 1 uses a block (hereinafter, referred to as
"vicinity block") on the left of the previous block, for which the
motion vector has been calculated, instead of using the previous
block, to generate the motion information of the encoding target
block in the skip mode.
[0012] After that, upon completion of calculating the motion
information of the previous block, the image encoding apparatus
confirms whether the calculated motion information of the previous
block and the motion information of the vicinity block match each
other, and, if these two match each other, is able to encode the
encoding target block in the skip mode or the like.
[0013] Thus, even when the calculation of the motion information of
the previous block has not been completed, the image encoding
apparatus of Patent Document 1 calculates the motion vector of the
encoding target block using the motion information of the vicinity
block, thereby realizing speeding-up of the encoding processing
without hindering the pipelined parallel processing.
[0014] Patent Document 1: Japanese Patent Publication No.
DISCLOSURE OF THE INVENTION
Problems the Invention is Going to Solve
[0015] However, because the image encoding apparatus according to
Patent Document 1 calculates the motion vector of the encoding
target block using the motion information of the vicinity block, it
is often the case that the calculated motion information of the
previous block and the motion information of the vicinity block do
not match each other. In this case, the encoding target block
cannot be encoded in the skip mode or the like.
[0016] The present invention is conceived in view of the above
problem and aims to provide an image encoding apparatus that is
able to realize speeding-up of the encoding processing with use of
a method different from the conventional method.
Means of Solving the Problems
[0017] In order to solve the stated problem, the image encoding
apparatus of the present invention is an image encoding apparatus
which compression encodes an image in units of blocks of a
predetermined size and comprises: a first processing unit operable
to sequentially generate candidate designation information
corresponding to each of the blocks, based on candidate designation
information corresponding to a prior block which has been processed
prior to the block, each candidate designation information
designating a plurality of prediction images as candidates for a
prediction image of the corresponding block; and a second
processing unit operable to, in parallel with generation of the
candidate designation information corresponding to the block by the
first processing unit, (i) select, as a prediction image of one of
the blocks which has been processed by the first processing unit, a
prediction image from among the plurality of prediction images
indicated by the candidate designation information corresponding to
the one of the blocks, based on prediction image designation
information designating a prediction image of a block processed
prior to the one of the blocks, (ii) outputs a difference signal
showing a difference which is between the selected prediction image
and an original image of the one of the blocks, and is to be
reflected in a result of compression encoding of the one of the
blocks, and (iii) generates prediction image designation
information designating the selected prediction image.
[0018] Also, in order to solve the stated problem, the image
encoding method used by the image encoding apparatus of the present
invention is an image encoding method which compression encodes an
image in units of blocks of a predetermined size and comprises: a
first processing step of sequentially generating candidate
designation information corresponding to each of the blocks, based
on candidate designation information corresponding to a prior block
which has been processed prior to the block, each candidate
designation information designating a plurality of prediction
images as candidates for a prediction image of the corresponding
block; and a second processing method of, in parallel with
generation of the candidate designation information corresponding
to the block in the first processing step, (i) selecting, as a
prediction image of one of the blocks which has been processed by
the first processing unit, a prediction image from among the
plurality of prediction images indicated by the candidate
designation information corresponding to the one of the blocks,
based on prediction image designation information designating a
prediction image of a block processed prior to the one of the
blocks, (ii) outputting a difference signal showing a difference
which is between the selected prediction image and an original
image of the one of the blocks, and is to be reflected in a result
of compression encoding of the one of the blocks, and (iii)
generating prediction image designation information designating the
selected prediction image.
[0019] Also, in order to solve the stated problem, the computer
program used in the image encoding apparatus of the present
invention is a computer program for causing a computer to perform
compression encoding processing which compression encodes an image
in units of blocks of a predetermined size and comprises: a first
processing step of sequentially generating candidate designation
information corresponding to each of the blocks, based on candidate
designation information corresponding to a prior block which has
been processed prior to the block, each candidate designation
information designating a plurality of prediction images as
candidates for a prediction image of the corresponding block; and a
second processing method of, in parallel with generation of the
candidate designation information corresponding to the block in the
first processing step, (i) selecting, as a prediction image of one
of the blocks which has been processed by the first processing
unit, a prediction image from among the plurality of prediction
images indicated by the candidate designation information
corresponding to the one of the blocks, based on prediction image
designation information designating a prediction image of a block
processed prior to the one of the blocks, (ii) outputting a
difference signal showing a difference which is between the
selected prediction image and an original image of the one of the
blocks, and is to be reflected in a result of compression encoding
of the one of the blocks, and (iii) generating prediction image
designation information designating the selected prediction
image.
[0020] Further more, in order to solve the stated problem, the
integrated circuit used by the image encoding apparatus of the
present invention is an integrated circuit used for image encoding,
the integrated circuit compression encoding an image in units of
blocks of a predetermined size and comprising: a first processing
unit operable to sequentially generate candidate designation
information corresponding to each of the blocks, based on candidate
designation information corresponding to a prior block which has
been processed prior to the block, each candidate designation
information designating a plurality of prediction images as
candidates for a prediction image of the corresponding block; and a
second processing unit operable to, in parallel with generation of
the candidate designation information corresponding to the block by
the first processing unit, (i) select, as a prediction image of one
of the blocks which has been processed by the first processing
unit, a prediction image from among the plurality of prediction
images indicated by the candidate designation information
corresponding to the one of the blocks, based on prediction image
designation information designating a prediction image of a block
processed prior to the one of the blocks, (ii) outputs a difference
signal showing a difference which is between the selected
prediction image and an original image of the one of the blocks,
and is to be reflected in a result of compression encoding of the
one of the blocks, and (iii) generates prediction image designation
information designating the selected prediction image.
[0021] Here, the blocks of the predetermined size are macroblocks
of n*n pixels, for example, of 16*16 pixels, and the prediction
images are of the same size as the blocks.
EFFECTS OF THE INVENTION
[0022] According to the image encoding apparatus of the present
invention having the stated structure, (i) the first processing
unit generates candidate designation information for each of the
blocks based on candidate designation information of a prior block
and, (ii) the second processing unit, in parallel with the
processing by the first processing unit, for one of the blocks
which has been processed by the first processing unit, selects a
prediction image from among the plurality of prediction images
indicated by the candidate designation information of the one of
the blocks, based on the prediction image designation information
designating a prediction image for a prior block processed prior to
the one of the blocks. Consequently, for each of the blocks,
compression encoding processing can be started even when a
prediction image has not been determined for the prior block
processed prior to the block at the start of the processing by the
first processing unit.
[0023] In other words, because encoding processing of each block
can be started without waiting for the prediction image of the
prior block of the block to be determined, the encoding processing
can be speeded up.
[0024] In addition, by using, for example, motion information (a
motion vector and a reference image) candidates as the candidate
designation information, candidate designation information of each
block is composed of motion information candidates generated based
on motion information candidates of a prior block. This enables
encoding in the skip mode or spatial direct mode according to
MPEG4AVC, thereby suppressing code amounts.
[0025] The above-described image encoding apparatus may compression
encode a plurality of images which are input sequentially, and each
candidate designation information may include designation
information designating, as one of the candidates for the
prediction image of the corresponding block, one of the plurality
of prediction images which is generated based on a motion vector
detected by searching one of the plurality of images that is other
than the image being compression encoded.
[0026] According to the stated structure, when the image encoding
apparatus uses, for example, a motion vector defined by the inter
prediction (inter mode) according to MPEG4AVC as the motion vector
detected by searching another image, the candidate designation
information includes designation information designating a
prediction image according to the inter prediction (inter
mode).
[0027] Accordingly, the image encoding apparatus realizes
compression encoding with suppressed code amounts by selecting for
each block, for example, a prediction image which minimizes the
code amount, from among the multiple prediction image candidates
indicated by the prediction image designation information.
[0028] Each candidate designation information of the
above-described image encoding apparatus may include designation
information designating, as a plurality of candidates among the
candidates for the prediction image of the corresponding block, out
of the plurality of prediction images, two or more prediction
images generated based on motion vectors of a plurality of adjacent
blocks including the prior block of the corresponding block, the
plurality of adjacent blocks being located in predetermined
positions with respect to the corresponding block.
[0029] According to the stated structure, when the image encoding
apparatus uses, for example, predetermined macroblocks used in the
skip mode according to MPEG4AVC standard as the adjacent blocks,
the prediction image candidates indicated by candidate designation
information include prediction images encoded in the skip mode or
the spatial direct mode. Accordingly, the code amount can be
suppressed when these prediction images are selected.
[0030] The above-described image encoding apparatus may further
comprise a third processing unit operable to cause each candidate
designation information to include designation information
designating, as one of the candidates for the prediction image of
the corresponding block, one of the plurality of prediction images
which is generated based on one or more pixel values of one or more
pixels located in predetermined positions with respect to the
corresponding block, and the second processing unit may select the
prediction image for the prior block of the corresponding block,
from among the plurality of prediction images indicated by the
candidate designation information caused by the third processing
unit to have the designation information.
[0031] According to the stated structure, when the image encoding
apparatus uses, for example, pixel values of predetermined pixels
defined by the intra prediction (intra mode) according to MPEG4AVC,
the candidate designation information includes designation
information designating a prediction image candidate in the intra
prediction (intra mode).
[0032] Accordingly, the image encoding apparatus realizes
compression encoding with suppressed code amounts by selecting for
each block, for example, a prediction image which minimizes the
code amount, from among the multiple prediction image candidates
indicated by the candidate designation information.
[0033] Each candidate designation information of the
above-described image encoding apparatus may be generated also
based on prediction image designation information of one of the
blocks which has been processed by the second processing unit.
[0034] According to the stated structure, for example, given that
the processing target block of the first processing unit is MBn,
the processing target block of the second processing unit is MBn-1,
and the block which has been processed by the second processing
unit is MBn-2, the image encoding apparatus generates the candidate
designation information of MBn based on the candidate designation
information of MBn-1 which is generated based on the candidate
designation information of MBn-2. In this way, that is to say, by
generating the candidate designation information of MBn based on
the prediction image designation information of MBn-2 which has
been processed by the second processing unit, the image encoding
apparatus is able to suppress the number of the prediction image
candidates indicated by the candidate designation information of
MBn.
[0035] Each candidate designation information of the
above-described image encoding apparatus may be generated based on
information designating only some of the plurality of prediction
images which are candidates for the prediction image of the prior
block of the corresponding block.
[0036] According to the stated structure, the image encoding
apparatus generates the candidate designation information of the
encoding target block based on only some of the prediction image
candidates of the previous block. This suppresses the number of the
prediction image candidates indicated by the candidate designation
information, thereby enabling a reduction in the processing load of
the image encoding apparatus.
[0037] The above-described image encoding apparatus may perform
processing in a pipeline structure composed of two stages that are
a first stage and a second stage. Here, the first processing unit
processes the first stage and includes: a motion detection unit
operable to generate, for each of the blocks, the candidate
designation information including designation information
designating, as one of the candidates for the prediction image of
the block, one of the plurality of prediction images which is
generated based on a motion vector detected by searching an image
other than the image being compression encoded; and a motion vector
speculative calculation unit operable to cause the candidate
designation information of the block to include designation
information designating, as a plurality of candidates among the
candidates for the prediction image of the block, out of the
plurality of prediction images, two or more prediction images
generated based on motion vectors of a plurality of adjacent blocks
including the prior block, the plurality of adjacent blocks being
located in predetermined positions with respect to the block, and
the second processing unit processes the second stage and includes:
a motion vector determination unit operable to, based on the
prediction image designation information of the block prior to the
one of the blocks, select a prediction image out of the one or more
prediction images designated by the designation information caused
by the motion vector speculative calculation unit to be included in
the candidate designation information of the one of the blocks; and
a mode judgement unit operable to (i) select the prediction image
for the one of the blocks, from among the prediction image selected
by the motion vector determination unit and the prediction image
indicated by the designation information generated by the motion
detection unit, (ii) output the difference signal, and (iii)
generate the prediction image designation information designating
the selected prediction image.
[0038] According to the stated structure, (i) the candidate
designation information generated for each block by the first
processing unit includes designation information designating
multiple prediction image candidates of the block based on the
candidate designation information, generated by the motion vector
speculative calculation unit, of the prior block, and (ii) the
motion vector determination unit in the second processing unit
selects one of the multiple prediction image candidates indicated
by the designation information generated by the motion vector
speculative calculation unit based on the prediction image
designation information of the block prior to the one of the
blocks. Thus, even when the prediction image of the prior block has
not been determined when the first processing unit starts
processing, the compression encoding of the block can be
started.
[0039] Additionally, the candidate designation information
corresponding to each of the blocks includes designation
information generated by the motion detection unit, and the mode
judgement unit in the second processing unit selects the prediction
image from among (a) the prediction image candidate selected by the
motion vector determination unit and (b) the prediction image
candidate, among the prediction image candidates indicated by the
candidate designation information, indicated by the designation
information generated by the motion detection unit. Consequently,
by selecting for each block, for example, a prediction image which
minimizes the code amount, the image encoding apparatus realizes
the compression encoding which suppresses the code amount.
BRIEF DESCRIPTION OF THE DRAWINGS
[0040] FIG. 1 is a diagram for explaining a process flow of
pipeline processing performed by an image encoding apparatus 100 of
a first embodiment of the present invention;
[0041] FIG. 2 is a diagram showing a functional structure of the
image encoding apparatus 100 of the first embodiment;
[0042] FIG. 3 is a flowchart showing operations of the image
encoding apparatus 100 of the first embodiment;
[0043] FIG. 4 is a diagram showing a functional structure of an
image encoding apparatus 200 of a second embodiment;
[0044] FIG. 5 is a flowchart showing operations of the image
encoding apparatus 200 of the second embodiment;
[0045] FIG. 6 is a diagram for explaining relationships between an
encoding target block and adjacent pixels; and
[0046] FIG. 7 is a diagram for explaining relationship between an
encoding target block and adjacent blocks.
DESCRIPTION OF REFERENCE NUMERALS
[0047] 100, 200 Image encoding apparatus [0048] 101 Motion
detection unit [0049] 102, 201 Motion vector speculative
calculation unit [0050] 103 First motion compensation unit [0051]
104, 203 Motion vector determination unit [0052] 105 Second motion
compensation unit [0053] 106 Intra prediction unit [0054] 107, 204
Mode judgement unit [0055] 108 Orthogonal transform unit [0056] 109
Quantization unit [0057] 110 Inverse quantization unit [0058] 111
Inverse orthogonal transform unit [0059] 112 Addition unit [0060]
113 Deblocking filter unit [0061] 114 Variable length encoding unit
[0062] 115 DMA controller [0063] 116 External memory [0064] 202
Confirmation unit
BEST MODE FOR CARRYING OUT THE INVENTION
[0065] Embodiments of the present invention are described below
with reference to the drawings.
First Embodiment
[0066] <Outline>
[0067] An image encoding apparatus of a first embodiment is a
modification of an image encoding apparatus that conforms to the
MPEG4AVC standard and encodes an encoding target image in units of
macroblocks of a predetermined size (for example, 16*16 pixels)
using pipeline processing.
[0068] The image encoding apparatus of the first embodiment
selects, for each macroblock, an encoding mode out of three
encoding modes, which are the intra mode, inter mode, and skip
mode, so as to minimize the code amount of the macroblock, and
encodes the macroblock using the selected encoding mode.
[0069] In the following, a brief description is given on each
encoding mode.
[0070] The intra mode is an encoding mode that (i) generates a
prediction image of an encoding target block based on pixel values
of pixels (hereinafter, referred to as "adjacent pixels") adjacent
to the encoding target block which is the macroblock targeted for
encoding, and (ii) encodes the difference information indicating
the difference between the generated prediction image and the
original image.
[0071] In the following, the adjacent pixels are described with
reference to FIG. 6.
[0072] FIG. 6 is a diagram for explaining relationships between the
encoding target block and the adjacent pixels.
[0073] The adjacent pixels refer to, with respect to the encoding
target block shown in FIG. 6, 16 pixels on the left (pixels A in
the figure) and 16 pixels on the top (pixels B in the figure).
[0074] Generation of the prediction image is performed, upon
selection of a prediction mode out of multiple prediction modes
(vertical prediction, horizontal prediction, and the like), using
the pixel values of the adjacent pixels which correspond with the
selected prediction mode. In other words, pixel values of different
adjacent pixels are used in accordance with each prediction
mode.
[0075] For example, when the vertical prediction is selected as the
prediction mode, the prediction image is generated on the
assumption that the pixel value of each column of pixels in the
encoding target block is equivalent to the pixel value of the pixel
in the same column in B in FIG. 6. It should be noted that while
selection methods of the prediction mode are prior arts and thus
not described in detail here, it is possible to apply a method
which makes a selection based on prediction modes of macroblocks in
the vicinity.
[0076] The inter mode is an encoding mode which (i) detects a
motion vector indicating a relative position with respect to a
block resembling to the encoding target block, by searching an
image (hereinafter, referred to as "search area image") other than
the encoding target image containing the encoding target block,
(ii) generates a prediction image of the encoding target block
based on motion information composed of the detected motion vector
and a reference image number indicating the search area image, and
(iii) encodes (a) the difference information indicating the
difference between the generated prediction image and the original
image of the encoding target block and (b) the motion
information.
[0077] The skip mode is an encoding mode which (i) calculates the
motion information of the encoding target block using the motion
information of the adjacent blocks which are multiple macroblocks
adjacent to the encoding target block, (ii) generates a prediction
image of the encoding target block based on the calculated motion
information, and (iii) encodes the difference information
indicating the difference between the generated prediction image
and the original image of the encoding target block. Thus,
according to the skip mode, only the difference information is
encoded, and it is not necessary to encode motion information.
Accordingly, the code amount can be reduced.
[0078] Here, the adjacent blocks are described with reference to
FIG. 7.
[0079] FIG. 7 is a diagram for explaining relationship between an
encoding target block and adjacent blocks. Among macroblocks A to F
shown in the figure, if the block E is the encoding target block in
the skip mode, then the adjacent blocks refer to the blocks B to
D.
[0080] However, if the motion information of the block C cannot be
used because, for example, the block C does not exist, the block A
replaces the block C as one of the adjacent blocks of the block E.
In addition, if all the motion information of the blocks A to C
cannot be used because, for example, none of the blocks A to C
exists in the encoding target image, the block D becomes the only
adjacent block of the block E. Note that, hereinafter, the block D
on the left of the encoding target block E is called "the previous
block".
[0081] When the blocks B to D are the adjacent blocks of the block
E, the motion vector of the block E is determined by finding out
the medians of the horizontal components and vertical components of
the motion vectors of the blocks B to D. It should be noted that
when the adjacent blocks include a macroblock encoded in the intra
mode, the motion vector of the block E is calculated assuming that
the motion vector of the macroblock is "0".
[0082] Additionally, the reference image of the block E in the skip
mode is an image displayed immediately preceding the encoding
target image which contains the block E, and the reference image
number is "0".
[0083] It should be noted that the reference image number is a
number indicating the position (relative position) of the reference
image, in terms of display order, with reference to the encoding
target image containing the encoding target block. The further the
reference image is from the encoding target image in terms of the
display order, the larger the reference image number assigned to
the reference image is.
[0084] In the following, the image encoding apparatus of the first
embodiment is specifically described with reference to FIG. 1.
[0085] FIG. 1 is a diagram for explaining a process flow of
pipeline processing performed by the image encoding apparatus of
the first embodiment.
[0086] The image encoding apparatus of the first embodiment (i)
selects an encoding mode for each macroblock in the encoding target
image such that the code amount of the macroblock is minimized and
(ii) encodes the macroblock in the selected encoding mode. Starting
from the upper left macroblock, the image encoding apparatus
processes a row of macroblocks in the horizontal direction, moves
on to the next row to continue processing, and in a similar manner,
sequentially processes the macroblocks up to the bottom right
macroblock.
[0087] As shown in FIG. 1, processing by the image encoding
apparatus of the first embodiment is a 4-stage pipeline structure
composed of the first to fourth stages, and in each time slot (TS1,
TS2, TS3, TS4, . . . ) processing of the pipeline stages are
performed in parallel.
[0088] In the following, the outline of the processing in each
pipeline stage is described with a focus on a macroblock (MBn)
which is processed n-th.
[0089] First, in the first stage, (a) detection of the motion
information in the case of encoding MBn in the inter mode and (b)
calculation of the motion information in the case of encoding MBn
in the skip mode are performed in parallel.
[0090] Next, in the second stage, (a) generation of prediction
images assuming that MBn is encoded in the respective encoding
modes and (b) calculation of difference information each indicating
difference between each prediction image and the original image are
performed in parallel so as to determine, for MBn, the encoding
mode which minimizes the code amount thereof.
[0091] Next, in the third and fourth stages, MBn is encoded in the
encoding mode determined in the second stage as the encoding mode
for MBn, and an encoded stream is generated.
[0092] As is clear from above, to calculate the motion information
of MBn in the skip mode in the first stage, the motion information
of MBn-1, which is the previous block of MBn, is required. However,
the encoding mode of MBn-1 is determined upon completion of the
processing in the second stage.
[0093] Accordingly, when the first-stage processing of MBn starts
(when TS1 starts), the second-stage processing for MBn-1 is about
to start, and thus, the encoding mode of MBn-1 has not been
determined. In other words, the motion information of MBn-1 has not
been determined.
[0094] Consequently, in the first stage, the image encoding
apparatus of the first embodiment calculates pieces of motion
information (hereinafter, referred to as "motion information
candidates") for MBn in the skip mode, the pieces of motion
information corresponding to the cases where the encoding mode of
MBn-1 is assumed to be one of the above-described encoding modes
(intra mode, inter mode, and skip mode), respectively. In other
words, three pieces of motion information are calculated as
candidates for the motion information of MBn in the skip mode.
[0095] Also, since the second-stage processing of MBn-1 has been
completed when the second-stage processing of MBn starts (when TS2
starts), the encoding mode of MBn-1 has been determined.
[0096] Accordingly, in the second stage, from among the three
motion information candidates for the motion information of MBn in
the skip mode, which have been calculated in the first stage, the
motion information candidate corresponding to the determined
encoding mode for MBn-1 is selected as the motion information of
MBn in the skip mode. Following that, the prediction image of MBn
in the skip mode is generated based on the selected motion
information, and the difference information indicating the
difference between the generated prediction image and the original
image of MBn is calculated.
[0097] Additionally, in the second stage, in parallel with the
above processing, the prediction images of MBn in the intra mode
and inter mode are generated, and the difference information each
indicating difference between each prediction image and the
original image of MBn is calculated. It should be noted that the
prediction image of MBn in the inter mode is generated based on the
motion information detected in the first stage.
[0098] Furthermore, in the second stage, upon completion of
calculating the difference information for each of the encoding
modes (intra mode, inter mode, and skip mode) of MBn, the code
amount in each encoding mode is calculated using the difference
information of the encoding mode, and the encoding mode which
yields the least code amount is determined as the encoding mode for
MBn.
[0099] Thus, being able to start processing the encoding target
block without waiting for the determination of the motion
information of the previous block, the image encoding apparatus of
the first embodiment allows the encoding of the encoding target
block in the skip mode, thereby suppressing the coding amount while
maintaining the high-speed encoding processing using the pipeline
structure.
[0100] <Structure>
[0101] First, the structure of the image encoding apparatus 100 of
the first embodiment is described with reference to FIG. 2.
[0102] FIG. 2 is a diagram showing the functional structure of the
image encoding apparatus 100.
[0103] As shown in FIG. 2, the image encoding apparatus 100
includes a motion detection unit 101, a motion vector speculative
calculation unit 102, a first motion compensation unit 103, a
motion vector determination unit 104, a second motion compensation
unit 105, an intra prediction unit 106, a mode judgement unit 107,
an orthogonal transform unit 108, a quantization unit 109, an
inverse quantization unit 110, an inverse orthogonal transform unit
111, an addition unit 112, a deblocking filter unit 113, a variable
length encoding unit 114, a DMA controller 115, and an external
memory 116.
[0104] Also, although not depicted, the image encoding apparatus
100 is equipped with a CPU (Central Processing Unit) and an
internal memory, and each function of the above-mentioned motion
detection unit 101, motion vector speculative calculation unit 102,
first motion compensation unit 103, motion vector determination
unit 104, second motion compensation unit 105, intra prediction
unit 106, mode judgement unit 107, orthogonal transform unit 108,
quantization unit 109, inverse quantization unit 110, inverse
orthogonal transform unit 111, addition unit 112, deblocking filter
unit 113, and variable length encoding unit 114 is realized by
causing the CPU to execute programs stored in the internal
memory.
[0105] Additionally, in the present embodiment, pipeline processing
is configured such that (i) processing by the motion detection unit
101 and processing by the motion vector speculative calculation
unit 102 correspond to the first stage, (ii) processing by the
first motion compensation unit 103, processing by the motion vector
determination unit 104, processing by the second motion
compensation unit 105, processing by the intra prediction unit 106,
and processing by the mode judgement unit 107 correspond to the
second stage, (iii) processing by the orthogonal transform unit
108, processing by the quantization unit 109, processing by the
inverse quantization unit 110, processing by the inverse orthogonal
transform unit 111, and processing by the addition unit 112
correspond to the third stage, and (iv) processing by the
deblocking filter unit 113 and processing by the variable length
encoding unit 114 correspond to the fourth stage. Note that the
first-stage processing corresponds to the first processing unit of
the present invention, and the second-stage processing corresponds
to the second processing unit and the third processing unit of the
present invention.
[0106] The motion detection unit 101 (i) detects the motion vector
of the encoding target block in the inter mode, and (ii) transmits,
to the motion vector speculative calculation unit 102 and the first
motion compensation unit 103, the motion information composed of
the detected motion vector and the reference image number
indicating the search area image used to detect the motion
vector.
[0107] Specifically, the motion detection unit 101 (i) reads, from
the external memory 116 to the internal memory (not depicted) via
the DMA controller 115, the original image of the encoding target
block (for example, a macroblock of 16* 16 pixels) and the search
area image targeted for searching for the motion vector, (ii)
performs block-matching between the read original image of the
encoding target block and the read search area image, and (c)
detects, by finding a macroblock which is most similar to the
original image of the encoding target block, the motion vector
indicating the relative position to the found macroblock.
[0108] It should be noted that regarding the detection of a motion
vector, it is possible to detect a motion vector of fractional
pixel accuracy (1/2 pixel accuracy and 1/4 pixel accuracy).
However, for the sake of description, description is given on the
assumption that a motion vector of integer pixel accuracy is
detected.
[0109] Additionally, the search area image is a decrypted image
that has been deblocked and stored in the external memory 116 by
the deblocking filter unit 113, which is described later. In the
following, the search area image is described as an image
immediately preceding, such as in the display order, the encoding
target image containing the encoding target block.
[0110] Upon detection of the motion vector, the motion detection
unit 101 transmits the detected motion vector and the reference
image number indicating the search area image, that is to say, the
motion information, to the motion vector speculative calculation
unit 102 and the first motion compensation unit 103.
[0111] The motion vector speculative calculation unit 102
calculates motion information candidates in the skip mode for the
encoding target block, and transmits the calculated motion
information candidates of the encoding target block to the vector
determination unit 104.
[0112] That is to say, the motion vector speculative calculation
unit 102 calculates the motion information candidates of the
encoding target block based on (a) the motion information of the
adjacent blocks of the encoding target block except for the
previous block, the motion information being stored in the internal
memory by the mode judgement unit 107 which is described later, and
(b) the motion information each of which corresponds to one of the
encoding modes selectable by the previous block whose encoding mode
has not been determined.
[0113] Specific description is given below on the calculation of
the motion information candidates. It should be noted that
irrespective of the encoding mode of the previous block, the
reference image number is "0".
[0114] When it is assumed that the encoding mode of the previous
block is the intra mode, the motion vector of the encoding target
block is calculated based on the motion vectors of the adjacent
blocks including the previous block on the assumption that the
motion vector of the previous block is "0". Hereinafter, the motion
information calculated under the assumption that the previous block
is the intra mode is referred to as "m1".
[0115] When it is assumed that the encoding mode of the previous
block is the inter mode, the motion vector of the encoding target
block is calculated based on (a) the motion vector of the previous
block in the inter mode, which has been received from the motion
detection unit 101 and (b) the motion vectors of the other adjacent
blocks. Hereinafter, the motion information calculated under the
assumption that the encoding mode of the previous block is the
inter mode is referred to as "m2".
[0116] When it is assumed that the encoding mode of the previous
block is the skip mode, the previous block also has three
possibilities for its motion information in the skip mode.
[0117] Thus, the motion vector speculative calculation unit 102,
with use of mode information which indicates the encoding mode
determined for the macroblock on the left of the previous block and
has been received from the mode judgement unit 107, selects, out of
the three motion information candidates for the motion information
of the previous block in the skip mode, a motion vector
corresponding to the encoding mode determined for the macroblock on
the left of the previous block as the motion vector of the previous
block in the skip mode. The motion vector speculative calculation
unit 102 then calculates the motion vector of the encoding target
block based on (a) the motion vector in the motion information of
the previous block in the skip mode and (b) the motion vectors of
the other adjacent blocks. Hereinafter, the motion information
calculated under the assumption that the encoding mode of the
previous block is the skip mode is referred to as "m3".
[0118] It should be noted that the motion vector speculative
calculation unit 102 stores, in the internal memory, the motion
information candidates (m1 to m3) calculated for the encoding
target block. The stored motion information candidates (m1 to m3)
are used when calculating the motion information "m3" for the next
macroblock to be processed.
[0119] In addition, the motion vector speculative calculation unit
102 reads the reference images indicated by the calculated motion
information candidates (m1 to m3) from the external memory 116 to
the internal memory via the DMA controller 115. These reference
images are used when the second motion compensation unit 105
generates the prediction image of the encoding target block in the
skip mode, as described later.
[0120] The first motion compensation unit 103 (i) generates the
prediction image based on the motion information, received from the
motion detection unit 101, of the encoding target block in the
inter mode, (ii) calculates the difference information indicating
the difference between the prediction image and the original image
of the encoding target block stored in the internal memory, and
(iii) transmits the prediction image, difference information, and
motion information to the mode judgement unit 107.
[0121] As to the generation of the prediction image, more
specifically, the prediction image is a block indicated by the
motion vector, the block being within the search area image
indicated by the reference mode number received from the motion
detection unit 101.
[0122] The motion vector determination unit 104 (i) selects, from
among the three motion information candidates (m1 to m3), received
from the motion vector speculative calculation unit 102, of the
encoding target block in the skip mode, a piece of motion
information corresponding to the encoding mode of the previous
block indicated by the mode information received from the mode
judgement unit 107, and (ii) transmits the selected motion
information candidate as the motion information of the encoding
target block in the skip mode to the second motion compensation
unit 105. The mode judgment unit 107 is described below.
[0123] For example, when the encoding mode of the previous block is
determined to be the inter mode, "m2" is transmitted to the second
motion compensation unit 105 as the motion information of the
encoding target block in the skip mode.
[0124] The second motion compensation unit 105 (i) generates the
prediction image based on the motion information of the encoding
target block in the skip mode received from the motion vector
determination unit 104, (ii) calculates the difference information
indicating the difference between the prediction image and the
original image of the encoding target block stored in the internal
memory, and (iii) transmits, to the mode judgement unit 107, the
prediction image, difference information, and motion information in
the skip mode which has been received from the motion vector
determination unit 104.
[0125] As to the generation of the prediction image, specifically,
the prediction image is the reference image indicated by the motion
information received from the motion vector determination unit
104.
[0126] The intra prediction unit 106 (i) reads the images of the
adjacent pixels of the encoding target block from the external
memory 116 to the internal memory via the DMA controller 115, (ii)
generates the prediction image of the encoding target block in the
intra mode based on the pixel values of the adjacent pixels, (iii)
calculates the difference information indicating the difference
between the prediction image and the original image of the encoding
target block stored in the internal memory, and (iv) transmits the
prediction image, difference information, and the prediction mode
information which indicates the prediction mode used to generate
the prediction image, to the mode judgement unit 107.
[0127] It should be noted that the images of the adjacent pixels
read from the external memory 116 by the intra prediction unit 106
is a decrypted image stored into the external memory 116 by the
addition unit 112, which is described later.
[0128] The mode judgement unit 107 selects, as the encoding mode of
the encoding target block, an encoding mode which minimizes the
code amount, based on the difference information in the inter mode
received from the first motion compensation unit 103, the
difference information in the skip mode received from the second
motion compensation unit 105, and the difference information in the
intra mode received from the intra prediction unit 106. It should
be noted that when the encoding is performed in the inter mode, the
code amount includes the code amount of the motion information, in
addition to the code amount of the above-mentioned difference
information.
[0129] Furthermore, the mode judgement unit 107 (i) transmits the
mode information indicating the determined encoding mode to the
vector speculative calculation unit 102 and the motion vector
determination unit 104, (ii) transmits the difference information
corresponding to the determined encoding mode to the orthogonal
transform unit 108, (iii) transmits the prediction image
corresponding to the determined encoding mode to the addition unit
112, and (iv) transmits the mode information and information (the
motion information when the inter mode is selected, the prediction
mode information when the intra mode is selected, and the like) in
accordance with the determined encoding mode, to the variable
length encoding unit 114.
[0130] It should be noted that the mode judgement unit 107 stores
the motion information (the motion information in the inter mode or
the motion information in the skip mode) in accordance with the
determined encoding mode into the internal memory.
[0131] The orthogonal transform unit 108 performs orthogonal
transform processing such as discrete cosine transform with respect
to the difference information received from the mode judgement unit
107, and transmits coefficient information, which is the result of
the processing, to the quantization unit 109.
[0132] The quantization unit 109 performs quantization processing
with respect to the coefficient information received from the
orthogonal transform unit 108, and transmits the quantized
coefficient information to the inverse quantization unit 110 and
the variable length encoding unit 114.
[0133] The inverse quantization unit 110 performs inverse
quantization processing with respect to the quantized coefficient
information received from the quantization unit 109 and transmits
coefficient information, which is the result of the processing, to
the inverse orthogonal transform unit 111.
[0134] The inverse orthogonal transform unit 111 performs inverse
orthogonal transform processing with respect to the coefficient
information received from the inverse quantization unit 110, and
transmits difference information, which is the result of the
processing, to the addition unit 112.
[0135] The addition unit 112 (i) generates a decrypted image of the
encoding target block by adding the prediction image received from
the mode judgement unit 107 and the difference information received
from the inverse orthogonal transform unit 111, and (ii) transmits
the generated decrypted image to the deblocking filter unit 113, as
well as storing the generated decrypted image into the external
memory 116 via the DMA controller 115.
[0136] The deblocking filter unit 113, with respect to the
decrypted image received from the addition unit 112, performs block
noise removal processing (hereinafter, referred to as "deblocking
processing"), and stores the deblocking-processed decrypted image
into the external memory 116 via the DMA controller 115.
[0137] The variable length encoding unit 114 (i) performs variable
length encoding processing, arithmetic encoding processing, or the
like with respect to the quantitized coefficient information
received from the quantization unit 109 and (ii) stores the
processed encoded stream into the external memory 116 via the DMA
controller 115. It should be noted that the mode information
received from the mode judgement unit 107 and the information in
accordance with the determined encoding mode are used for
generating the header of the encoded stream.
[0138] The DMA controller 115 is a general DMA controller which
arbitrates access requests from the respective units to the
external memory 116 and transmits data between the external memory
116 and the internal memory.
[0139] The external memory 116 is a memory composed of a DRAM and
the like storing each encoding target block, the decrypted image
stored by the addition unit 112, the deblocking-processed decrypted
image stored by the deblocking filter unit 113, and the encoded
stream stored by the variable length encoding unit 114.
[0140] <Operations>
[0141] Next, operations of the image encoding apparatus 100 having
the stated structure are described with reference to FIG. 3.
[0142] FIG. 3 is a flowchart showing the operations of the image
encoding apparatus 100.
[0143] In the following, processing by the image encoding apparatus
100 is described in pipeline stage units. It should be noted that
the pipeline stages from the first stage to the fourth stage are
processed in parallel with one another, each of the pipeline stage
targeting a different macroblock for processing.
[0144] <Operations in First Stage>
[0145] First, operations by the image encoding apparatus 100 in the
first stage are described. In the first stage, the processing by
the motion detection unit 101 (steps S01 to S03) and the processing
by the motion vector speculative calculation unit 102 (steps S04
and S05), which are described below, are executed in parallel.
[0146] The motion detection unit 101 reads the original image of
the encoding target block and the search area image which is the
search target of the motion vector, from the external memory 116 to
the internal memory via the DMA controller 115 (steps S01 and
S02).
[0147] The motion detection unit 101 (i) detects the motion vector
by searching the search area image for a macroblock which is most
similar to the original image of the encoding target block (step
S03) and (ii) transmits the detected motion vector and the
reference image number indicating the search area image, that is to
say, the motion information, to the first motion compensation unit
103.
[0148] On the other hand, the motion vector speculative calculation
unit 102 calculates the motion information candidates (m1 to m3)
assuming the encoding mode of the previous block of the encoding
target block to be the intra mode, inter mode, and skip mode,
respectively (step S04), and transmits the calculated motion
information candidates (m1 to m3) to the motion vector
determination unit 104.
[0149] Furthermore, the motion vector speculative calculation unit
102 reads, from the external memory 116 to the internal memory via
the DMA controller 115 (step S05), the reference images indicated
by the motion information candidates (m1 to m3).
[0150] <Operations in Second Stage>
[0151] Next, operations by the image encoding apparatus 100 in the
second stage are described.
[0152] In the second stage, (a) the processing by the first motion
compensation unit 103 (steps S06 and S07), (b) the processing by
the motion vector determination unit 104 and the second motion
compensation unit 105 (steps S08 to S10), and (c) the processing by
the intra prediction unit 106 (steps S11 and S12) are executed in
parallel with one another. The processing by each unit is described
below.
[0153] The first motion compensation unit 103 generates a
prediction image of the encoding target block in the inter mode
based on the motion information received from the motion detection
unit 101 (step S06).
[0154] The first motion compensation unit 103 (i) calculates the
difference information indicating the difference between the
generated prediction image and the original image of the encoding
target block stored in the internal memory (step S07) and (ii)
transmits the calculated difference information, generated
prediction image, and motion information received from the motion
detection unit 101, to the mode judgement unit 107.
[0155] The motion vector determination unit 104 (i) selects, from
among the three motion information candidates (m1 to m3) of the
encoding target block in the skip mode received from the motion
vector speculative calculation unit 102, as the motion information
of the encoding target block in the skip mode, the motion
information candidate corresponding to the encoding mode of the
previous block indicated by the mode information received from the
mode judgement unit 107 (step S08), and (ii) transmits the selected
motion information to the second motion compensation unit 105.
[0156] The second motion compensation unit 105 generates a
prediction image of the encoding target block in the skip mode
based on the motion information received from the motion vector
determination unit 104 (step S09).
[0157] The second motion compensation unit 105 (i) calculates the
difference information indicating the difference between the
generated prediction image and the original image of the encoding
target block stored in the internal memory (step S10), and (ii)
transmits the calculated difference information, generated
prediction image, and the motion information in the skip mode
received from the motion vector determination unit 104, to the mode
judgement unit 107.
[0158] The intra prediction unit 106 reads the images of the
adjacent pixels of the encoding target block from the external
memory 116 into the internal memory via the DMA controller 115, and
generates a prediction image of the encoding target block in the
intra mode based on the pixel values of the adjacent pixels (step
S11).
[0159] The intra prediction unit 106 calculates the difference
information indicating the difference between the generated
prediction image and the original image of the encoding target
block stored in the internal memory (step S12), and transmits the
calculated difference information, generated prediction image, and
the prediction mode information, to the mode judgement unit
107.
[0160] The mode judgement unit 107 selects an encoding mode which
minimizes the code amount, from among the encoding modes (the inter
mode, skip mode, and intra mode) as the encoding mode of the
encoding target block, based on the difference information received
from the first motion compensation unit 103, second motion
compensation unit 105, and intra prediction unit 106, respectively
(step S13).
[0161] The mode judgement unit 107 (i) transmits the mode
information indicating the determined encoding mode to the vector
speculative calculation unit 102 and the motion vector
determination unit 104, (ii) transmits the difference information
corresponding to the determined encoding mode to the orthogonal
transform unit 108, (iii) transmits the prediction image
corresponding to the determined encoding mode to the addition unit
112, and (iv) transmits the mode information and information (the
motion information when the inter mode is selected, the prediction
mode information when the intra mode is selected, and the like) in
accordance with the determined encoding mode, to the variable
length encoding unit 114.
[0162] <Operations in Third Stage>
[0163] Next, operations of the image encoding apparatus 100 in the
third stage are described.
[0164] The orthogonal transform unit 108 performs orthogonal
transform processing such as discrete cosine transform with respect
to the difference information received from the mode judgement unit
107, and transmits coefficient information, which is the result of
the processing, to the quantization unit 109 (step S14). The
quantization unit 109 performs quantization processing with respect
to the coefficient information received from the orthogonal
transform unit 108, and transmits the quantized coefficient
information to the inverse quantization unit 110 and the variable
length encoding unit 114 (step S15).
[0165] The inverse quantization unit 110 performs inverse
quantization processing with respect to the quantized coefficient
information received from the quantization unit 109, and transmits
coefficient information, which is the result of the processing, to
the inverse orthogonal transform unit 111 (step S16). The inverse
orthogonal transform unit 111 performs inverse orthogonal transform
processing with respect to the coefficient information received
from the inverse quantization unit 110, and transmits difference
information, which is the result of the processing, to the addition
unit 112 (step S17).
[0166] <Operations in Fourth Stage>
[0167] Next, operations of the image encoding apparatus 100 in the
fourth stage are described.
[0168] In the fourth stage, the processing by the addition unit 112
and deblocking filter unit 113 (steps S18 to S20) and the
processing by the variable length encoding unit 114 (steps S21 and
S22) are executed in parallel. The processing by these units are
described below.
[0169] The addition unit 112 (i) generates the decrypted image of
the encoding target block by adding the prediction image received
from the mode judgement unit 107 and the difference information
received from the inverse orthogonal transform unit 111 (step S18),
and (ii) transmits the generated decrypted image to the deblocking
filter unit 113, as well as storing the generated decrypted image
into the external memory 116 via the DMA controller 115.
[0170] The deblocking filter unit 113 performs the deblocking
processing with respect to the decrypted image received from the
addition unit 112 (step S19), and stores the deblocking-processed
decrypted image into the external memory 116 via the DMA controller
115 (step S20).
[0171] The variable length encoding unit 114 (i) performs variable
length encoding processing, arithmetic encoding processing, or the
like with respect to the quantitized coefficient information
received from the quantization unit 109 (step S21) and (ii) stores
the processed encoded stream into the external memory 116 via the
DMA controller 115 (step S22).
[0172] <Observations>
[0173] In the following, effects achieved by the image encoding
apparatus 100 of the present embodiment are described in comparison
with the image encoding apparatus according to Patent Document
1.
[0174] The image encoding apparatus according to Patent Document 1
generates the motion information of the encoding target block in
the skip mode using motion information of a vicinity block instead
of using the motion information of the previous block. Accordingly,
although the high-speed encoding processing using the pipeline
structure can be maintained, it is not possible to perform encoding
in the skip mode in a case where the motion information of the
previous block, which is determined subsequently, differs from the
motion information of the vicinity block.
[0175] On the other hand, the image encoding apparatus 100
calculates the motion information candidates of the encoding target
block in the skip mode based on all the motion information
selectable by the previous block of the encoding target block.
Following that, upon determination of the motion information of the
previous block, the image encoding apparatus 100 selects, out of
the calculated motion information candidates, a motion information
candidate which corresponds with the determined motion information
of the previous block as the motion information of the encoding
target block in the skip mode.
[0176] Consequently, each macroblock can be encoded in the skip
mode reliably without exception, thereby enabling a reduction in
the code amount, while the high-speed encoding processing with the
pipeline structure is maintained.
Second Embodiment
[0177] <Outline>
[0178] The motion vector speculative calculation unit 102 of the
first embodiment (i) calculates all the motion information
candidates of the encoding target block in the skip mode based on
all the encoding modes (intra mode, inter mode, and skip mode)
selectable by the previous block, and (ii) acquires all the
reference images indicated by the above-mentioned respective motion
information candidates.
[0179] However, if it is necessary to calculate all the motion
information candidates of the encoding target block in the skip
mode and acquire all the reference images indicated by those motion
information candidates, processing time of the motion vector
speculative calculation unit 102, especially processing time for
transferring the reference images, requires a large amount of time.
Moreover, at the same time, a high-capacity internal memory is
required to store all the reference images.
[0180] Thus, the motion vector speculative calculation unit of the
second embodiment (i) calculates motion information candidates of
the encoding target block in the skip mode based on motion
information candidates corresponding to, among all the encoding
modes selectable by the previous block, the encoding modes which
are effective in improving image quality and encoding efficiency,
namely, the inter mode and skip mode, and (ii) acquires the
reference images indicated by these motion information
candidates.
[0181] As a result, processing time of the motion vector
speculative calculation unit, especially the processing time
required for transferring the reference images, can be suppressed,
and the capacity of the internal memory can be reduced.
[0182] <Structure>
[0183] First, the image encoding apparatus 200 of the second
embodiment is described with reference to FIG. 4.
[0184] FIG. 4 is a diagram showing a functional structure of the
image encoding apparatus 200.
[0185] As shown in FIG. 4, the image encoding apparatus 200
includes the motion detection unit 101, first motion compensation
unit 103, second motion compensation unit 105, intra prediction
unit 106, orthogonal transform unit 108, quantization unit 109,
inverse quantization unit 110, inverse orthogonal transform unit
111, addition unit 112, deblocking filter unit 113, variable length
encoding unit 114, DMA controller 115, external memory 116, a
motion vector speculative calculation unit 201, a confirmation unit
202, a motion vector determination unit 203, and a mode judgement
unit 204.
[0186] The above-mentioned components are the same as those of the
image encoding apparatus 100 of the first embodiment except for the
motion vector speculative calculation unit 201, confirmation unit
202, motion vector determination unit 203, and mode judgement unit
204, and therefore, their description is omitted here.
[0187] Although not depicted, the image encoding apparatus 200 is
equipped with the CPU and the internal memory, and each function of
the motion vector speculative calculation unit 201, confirmation
unit 202, motion vector determination unit 203, and mode judgement
unit 204 is realized by causing the CPU to execute programs stored
in the internal memory.
[0188] Additionally, in the present embodiment, the pipeline
processing is configured such that (i) processing by the motion
vector speculative calculation unit 201 corresponds to the first
stage, and (ii) processing by the confirmation unit 202, processing
by the motion vector determination unit 203, and processing by the
mode judgement unit 204 correspond to the second stage.
[0189] The motion vector speculative calculation unit 201 basically
has the same functions as the motion vector speculative calculation
unit 102 of the image encoding apparatus 100. However, the motion
vector speculative calculation unit 201 is different from the
motion vector speculative calculation unit 102 in the following
respects: the motion vector speculative calculation unit 201 (i)
calculates the motion information candidates (m2 and m3) based on
the motion information in the inter mode and in the skip mode,
respectively, among the encoding modes selectable by the previous
block whose motion information has not been determined, and (ii)
transmits the motion information candidates (m2 and m3) to the
motion vector determination unit 203 while reading the reference
images indicated by the calculated motion information candidates
(m2 and m3) from the external memory 116 to the internal memory via
the DMA controller 115 as well.
[0190] The confirmation unit 202 (i) confirms whether the encoding
mode indicated by the mode information of the previous block
received from the mode judgement unit 204 is either the inter mode
or the skip mode, and (ii) transmits confirmation result
information indicating a confirmation result to the motion vector
determination unit 203 and mode judgement unit 204.
[0191] The motion vector determination unit 203 basically has the
same functions as the motion vector determination unit 104 of the
image encoding unit 100. However, the motion vector determination
unit 203 is different from the motion vector determination unit 104
in the following respects: when the confirmation result indicated
by the confirmation result information received from the
confirmation unit 202 indicates that the encoding mode of the
previous block is neither the inter mode nor the skip mode, that
is, the encoding mode of the previous block is indicated to be the
intra mode, the motion vector determination unit 203 does not
select the motion information of the encoding target block in the
skip mode.
[0192] The mode judgement unit 204 basically has the same functions
as the mode judgement unit 107 of the image encoding apparatus 100.
However, the mode judgement unit 204 is different from the mode
judgement unit 107 in the following respects: when the confirmation
result indicated by the confirmation result information received
from the confirmation unit 202 indicates that the encoding mode of
the previous block is neither the inter mode nor the skip mode,
that is, when the encoding mode of the previous block is indicated
to be the intra mode, the mode judgement unit 204 determines an
encoding mode other than the skip mode as the encoding mode of the
encoding target block.
[0193] In other words, when the encoding mode of the previous block
is the intra mode, the mode judgement unit 204 determines, as the
encoding mode of the encoding target block, one of the inter mode
and intra mode so as to minimize the code amount of the encoding
target block, based on the difference information in the inter mode
received from the first motion compensation unit 103 and the
difference information in the intra mode received from the intra
prediction unit 106.
[0194] <Operations>
[0195] Next, operations of the image encoding apparatus 200 having
the stated structure are described with reference to FIG. 5.
[0196] FIG. 5 is a flowchart showing the operations of the image
encoding apparatus 200. Since the processing in the steps S01 to
S22 are the same as the processing in the corresponding steps of
the image encoding apparatus 100, their description is omitted. In
the following, only differences between the image encoding
apparatus 200 and the image encoding apparatus 100 are
described.
[0197] In the first stage, in parallel with the processing by the
motion detection unit 101 (steps S01 to S03), the motion vector
speculative calculation unit 201 (i) assumes the encoding mode of
the previous block of the encoding target block to be either the
inter mode or the skip mode, (ii) calculates the motion information
candidates (m2 and m3) for these modes, respectively (step S31),
and (iii) transmits the motion information candidates (m2 and m3)
to the motion vector determination unit 203.
[0198] Also, the motion vector speculative calculation unit 201
reads the reference images respectively indicated by the motion
information candidates (m2 and m3), from the external memory 116
into the internal memory via the DMA controller 115 (step S32).
[0199] In the second stage, in parallel with the processing by the
first motion compensation unit 103 (steps S06 and S07) and the
processing by the intra prediction unit 106 (steps S11 and S12),
the confirmation unit 202 (i) confirms whether the encoding mode
indicated by the mode information of the previous block received
from the mode judgement unit 204 is either the inter mode or the
skip mode (step S33) and (ii) transmits the confirmation result
information to the motion vector determination unit 203 and the
mode judgement unit 204.
[0200] If the confirmation result information indicates that the
encoding mode of the previous block is either the inter mode or the
skip mode (step S33:Y), the motion vector determination unit 203
(i) selects, from among the motion information candidates (m2 and
m3) received from the motion vector speculative calculation unit
201, one motion information candidate corresponding to the encoding
mode indicated by the mode information of the previous block
received from the mode judgement unit 204, as the motion
information of the encoding target block in the skip mode (step
S34) and (ii) transmits the selected motion information to the
second motion compensation unit 105.
[0201] As described in the first embodiment, the second motion
compensation unit 105 (i) generates the prediction image of the
encoding target block in the skip mode (step S09), (ii) calculates
the difference information indicating the difference between the
prediction image of the encoding target block and the original
image (step S10), and (iii) transmits the calculated difference
information, generated prediction image, and the motion information
in the skip mode received from the motion vector determination unit
203, to the mode judgement unit 204.
[0]
[0202] If the confirmation result information indicates that the
encoding mode of the previous block is neither the inter mode nor
the skip mode, that is to say, the encoding mode of the previous
block is indicated to be the intra mode (step S33:N), the
processing proceeds to the step S35, which is described later,
without performing the processing in the step S34, S09, and
S10.
[0203] The mode judgement unit 204 determines the encoding mode
which minimizes the code amount of the encoding target block, as
the encoding mode of the encoding target block (step S35)
[0204] In other words, in the case where the mode judgement unit
204 receives, from the confirmation unit 202, the confirmation
result information indicating that the encoding mode of the
previous block is either the inter mode or the skip mode, the mode
judgement unit 204 has received a piece of difference information
from each of the first motion compensation unit 103, the second
motion compensation unit 105, and the intra prediction unit 106.
Thus, in this case, the mode judgement unit 204 determines the
encoding mode of the encoding target block based on the three
pieces of difference information.
[0205] On the other hand, in the case where the encoding mode of
the previous block is indicated to be neither the inter mode nor
the skip mode, that is to say, the encoding mode of the previous
block is indicated to be the intra mode by the confirmation result
information received from the confirmation unit 202, the mode
judgement unit 204 has not received the difference information from
the second motion compensation unit 105. Thus, in this case, the
mode judgement unit 204 determines the encoding mode of the
encoding target block based on the two pieces of difference
information received from the first motion compensation unit 103
and the intra prediction unit 106.
[0206] <Additional Particulars>
[0207] Although the image encoding apparatus of the present
invention has been described by way of the embodiments above, it is
to be noted that the following modifications are possible, and
naturally, the present invention is not limited to the image
encoding apparatuses described in the embodiments above.
[0208] (1) According to the embodiments, the research area image
used by the motion detection unit 101 to detect the motion vector
is the decrypted image of the previous image of the encoding target
image which includes the encoding target block. However, it is not
limited to the decrypted image of the previous image, and can be
the decrypted image of an image which precedes the encoding target
image by more than one image.
[0209] The search area image is also described to be the entire
decrypted image of the previous image of the encoding target image.
However, it can be part of the encoding target image. For example,
the search area image can be a block and 15 pixels in its vicinity
in the decrypted image of the previous image of the encoding target
image, the position of the block in the previous image being the
same as the position of the encoding target block in the encoding
target image.
[0210] (2) According to the embodiments, processing is performed in
the four-stage pipeline structure. However, in accordance with the
processing capacity of the CPU, for example, the third stage and
the fourth stage in the first embodiment can be merged into one
stage so that the processing is performed in a three-stage pipeline
structure.
[0211] (3) According to the first embodiment, the motion vector
speculative calculation unit 102 calculates the motion information
candidates of the encoding target block in the skip mode. However,
it may calculate motion information candidates in the spatial
direct mode.
[0212] It should be noted that, according to the embodiments, in
the skip mode, the reference image number is "0" irrespective of
the encoding mode of the previous block. However, when using the
spatial direct mode, it is different from the skip mode in that,
with respect to each direction L0 and L1, the smallest image number
among the reference image numbers of the adjacent blocks is
selected to be the reference image number of the encoding target
block.
[0213] However, when the reference image number of the adjacent
blocks is "-1", that is to say, the encoding mode of the adjacent
block is the intra mode, the reference image number of the encoding
target block in the spatial direct mode is "0". It should be noted
that the motion vector speculative calculation unit 201 of the
second embodiment can be similarly modified.
[0214] (4) According to the second embodiment, the motion vector
speculative calculation unit 201 calculates the motion information
candidates (m2 and m3) based on the motion information in the inter
mode and in the skip mode, respectively, among the encoding modes
selectable by the previous block whose motion information has not
been determined. However, it is not limited to this, and the motion
vector speculative calculation unit 201 may calculate motion
information candidates (m1 and m3) of the encoding target block
based on the motion information in the intra mode and the skip
mode, or may calculate a motion information candidate (m3) of the
encoding target block based only on the motion information in the
skip mode.
[0215] In addition, the number of motion information candidates
calculated by the motion vector speculative calculation unit 201 is
not limited, and it can be determined in accordance with the
processing ability and the like of the CPU.
[0216] (5) According to the embodiments, the processing by the
intra prediction unit 106 is included in the processing in the
second stage. However, the processing by the intra prediction unit
106 can be included in the processing in the first stage. That is
to say, there is no particular restriction as to in which stage the
processing by the intra prediction unit 106 should be included, as
long as it is completed before the mode judgement unit 107 (or 204)
determines the encoding mode of the encoding target block.
[0217] (6) According to the first embodiment, each function of the
motion detection unit 101, motion vector speculative calculation
unit 102, first motion compensation unit 103, motion vector
determination unit 104, second motion compensation unit 105, intra
prediction unit 106, mode judgement unit 107, orthogonal transform
unit 108, quantization unit 109, inverse quantization unit 110,
inverse orthogonal transform unit 111, addition unit 112,
deblocking filter unit 113, and variable length encoding unit 114
is realized by causing the CPU to execute programs stored in the
internal memory. However, part or an entirety of the processing may
be realized by hardware.
[0218] In this case, each function is typically realized by an LSI,
an integrated circuit. These functions may be integrated into a
single chip individually, or part or all of these functions may be
integrated into a single chip.
[0219] It should be noted that it is the same with each function of
the motion vector speculative calculation unit 201, confirmation
unit 202, motion vector determination unit 203, and mode judgement
unit 204 in the second embodiment.
[0220] (7) It is also possible to circulate/distribute a program
for CPU to realize each function of the image encoding apparatus by
recording the program on a recording medium or using various
communication channels. The recording media can be an IC card, an
optical disc, a hard disk, a flexible disc, a ROM or the like. The
circulated/distributed program is made available for use by being
stored in a memory or the like that can be read by the CPU of an
apparatus, and each function of the image encoding apparatuses
described in the embodiments is realized by execution of the
program by the CPU.
[0221] (8) According to the embodiments, the image encoding
apparatuses conform to MPEG4AVC standards. However, they may
conform to another compression encoding method such as MPEG4 or the
like.
INDUSTRIAL APPLICABILITY
[0222] The image encoding apparatus of the present invention is
utilized to speed up encoding processing.
* * * * *