U.S. patent application number 10/618979 was filed with the patent office on 2004-01-22 for apparatus and method of building an electronic database for resolution synthesis.
Invention is credited to Allebach, Jan P., Atkins, Brian, Bouman, Charles A..
Application Number | 20040013320 10/618979 |
Document ID | / |
Family ID | 25274975 |
Filed Date | 2004-01-22 |
United States Patent
Application |
20040013320 |
Kind Code |
A1 |
Atkins, Brian ; et
al. |
January 22, 2004 |
Apparatus and method of building an electronic database for
resolution synthesis
Abstract
An electronic database for image interpolation is generated by a
computer. The computer generates a low-resolution image from a
training image, a plurality of representative vectors from the
low-resolution image, and a plurality of interpolation filters
corresponding to each of the representative vectors. The
interpolation filters and the representative vectors are generated
off-line and can be used to perform image interpolation on an image
other than the training image. The database can be stored in a
device such as computer or a printer.
Inventors: |
Atkins, Brian; (Mountain
View, CA) ; Bouman, Charles A.; (West Lafayette,
IN) ; Allebach, Jan P.; (West Lafayette, IN) |
Correspondence
Address: |
HEWLETT-PACKARD COMPANY
Intellectual Property Administration
P. O. Box 272400
Fort Collins
CO
80527-2400
US
|
Family ID: |
25274975 |
Appl. No.: |
10/618979 |
Filed: |
July 14, 2003 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
10618979 |
Jul 14, 2003 |
|
|
|
10193931 |
Jul 11, 2002 |
|
|
|
10193931 |
Jul 11, 2002 |
|
|
|
09064638 |
Apr 21, 1998 |
|
|
|
6466702 |
|
|
|
|
10618979 |
Jul 14, 2003 |
|
|
|
08837619 |
Apr 21, 1997 |
|
|
|
6075926 |
|
|
|
|
Current U.S.
Class: |
382/300 |
Current CPC
Class: |
G06T 3/4007
20130101 |
Class at
Publication: |
382/300 |
International
Class: |
G06K 009/32 |
Claims
What is claimed is:
1. A method of building an electronic database for data resolution
synthesis from at least one training file, the method comprising
the steps of: generating a low-resolution file from each training
file; generating a plurality of representative vectors from each
low-resolution file; and generating a set of interpolation filters
for each of the representative vectors; whereby the interpolation
filters and the representative vectors can be used to perform data
resolution synthesis on a file other than the training file.
2. The method of claim 1 wherein the representative vectors are
generated by computing a number NCV of cluster vectors from each
low-resolution file and using the cluster vectors to compute the
representative vectors; and wherein low-resolution observation
vectors, the cluster vectors, the representative vectors and a
high-resolution file corresponding to each low-resolution file are
used to compute the interpolation filters, whereby a high
resolution file may be a training file.
3. The method of claim 2, further comprising the step of generating
a sharpened high-resolution file, the sharpened high-resolution
file being used to compute the interpolation filters.
4. The method of claim 2, wherein the representative vectors ar
generated by using a maximum likelihood estimate.
5. The method of claim 4, wherein the vectors are generated by
using an expectation maximization technique.
6. The m thod of claim 4, wherein a classifier including the
representative v ctors is computed by initializing th classifier
and updating the classifier until optimal values for the classifier
have been obtained.
7. The method of claim 6, wherein the classifier further includ s a
variance and a number M of class weights, and wherein the
representative vectors, the class weights and the variance are
computed simultaneously.
8. The method of claim 2, wherein each cluster vector is generat d
by forming an observation window about sampled data in a low
resolution file, extracting a vector including neighboring data of
the sampled data, and scaling the vector.
9. The method of claim 2, wherein coefficients for the
interpolation filters are computed by: computing a number NFDV of
filter design triplets from data in the low-resolution file, where
NFDV is a positive integer, each filter design triplet
corresponding to sampled data in the low-resolution file, each
filter design triplet including an observation vector for the
sampled data, a cluster vector for the sampled data, and a vector
of high resolution data from a high-resolution file; computing
training statistics from the filter design triplets; and computing
the coefficients from the training statistics.
10. The method of claim 2, wherein the steps are run off-line in a
computer.
11. The method of claim 1, wherein the interpolation filters are
linear filters.
12. The m thod of claim 1, wherein the representative vectors are
generated by using a parameter optimization technique.
13. A method of using a computer to compute a plurality of
resolution synthesis parameters from a training image, the method
comprising the steps of: computing a low-resolution image from the
training image; computing a plurality of cluster vectors for a
number NCV of pixels in the low-resolution image, where NCV is a
positive integer; using the cluster vectors to compute a number M
of representative vectors for the low resolution image, where M is
a positive integer that is less than NCV; and using low-resolution
observation vectors, the cluster vectors, the representative
vectors and vectors from a high-resolution image to compute sets of
interpolation filter coefficients corresponding to each of the
representativ vectors; whereby the high-resolution image may be the
training image; and whereby the interpolation filter coefficients
and the number M of representative vectors are stored in the
database for later interpolation of an image other than the
training image.
14. The method of claim 13, wherein the number NCV is between
25,000 and 100,000, whereby between 25,000 and 100,000 cluster
vectors are computed.
15. The method of claim 13, wherein each cluster vector for a
non-border pixel is computed by extracting a first vector from a
square observation window centered about a sampled pixel in the
low-resolution image, and scaling the first vector.
16. Th method of claim 13, where the number M of r pres ntative
vectors is between 50 and 100.
17. The method of claim 13, wherein the representative vectors are
computed using a maximum likelihood estimate.
18. The method of claim 17, wherein a classifier including th
representative vectors is computed by initializing the classifier
and updating the classifier until optimal values for the classifier
have been obtained.
19. The method of claim 13, wherein the representative vectors are
computed using an expectation-maximization algorithm.
20. The method of claim 19, wherein the representative vectors are
computed by: setting initial values for a classifier including a
number M of class weights, a variance and the number M of
representative vectors; computing a quality measure of how well the
cluster vectors are represented by the initial values for the
classifier; updating the classifier; recomputing the quality
measure for the updated classifier; and determining whether the
cluster vectors are suitably represented by th updated classifier,
the classifier being updated until the cluster vectors are suitably
represented.
21. The method of claim 13, further comprising the st p of
computing a sharpened high-resolution image from the training
image, wherein the sharpened image is used along with
low-resolution observation vectors, th cluster vectors and the
representative vectors to compute the interpolation filter
coefficients.
22. The method of claim 13, wherein the interpolation filter
coefficients are computed by: computing a number NFDV of filter
design triplets from the low-resolution image, where NFDV is a
positive integer, each filter design triplet corresponding to a
sampled pixel in the low-resolution image, each filter design
triplet including an observation vector for the sampled pixel, a
cluster vector for the sampled pixel, and a vector of
high-resolution pixels corresponding to the sampled pixel, the
high-resolution pixels being taken from the high-resolution image;
computing training statistics from the filter design triplets; and
computing the coefficients from the training statistics.
23. The method of claim 22, wherein the number NFVD of filter
design triplets is between 500,000 and 1,000,000, whereby between
500,000 and 1,000,000 filter design triplets are computed.
24. The method of claim 22, wherein the interpolation filter
coefficients are computed for linear interpolation filters.
25. The method of claim 13, wherein the steps are run off-line in
the computer.
26. The method of claim 25, wherein the database is stored for
transfer to a second computer, whereby the second computer can
access the database to perform image interpolation on images other
than the training images.
27. The method of claim 25, wherein the database is stored in
memory of a printer, whereby the printer can access the database to
p rform image interpolation on images other than the training
images.
28. The method of claim 13, wherein the repres ntativ vectors are
generated by using a parameter optimization technique.
29. Apparatus comprising: a processor; and memory means for storing
an electronic database and a plurality of executable instructions,
the instructions, when executed, instructing the processor to
access a training file; generate a low-resolution file from the
training fil; generate a plurality of representative vectors from
the low-resolution file; generate a set of interpolation filters
for each of the representative vectors; and store the interpolation
filters and the representative vectors in the memory means as part
of the database.
30. The apparatus of claim 29, wherein the instructions instruct
the processor to generate the representative vectors by computing a
number NCV of cluster vectors from the low-resolution file, and
using the cluster vectors to generate the representative vectors;
and wherein the instructions instruct the processor to generate the
interpolation filters from low-resolution observation vectors, the
cluster vectors, the representative vectors and a plurality of
vectors from a high-resolution file corresponding to the
low-resolution file.
31. The apparatus of claim 30, wherein the instructions further
instruct the processor to generate a sharpened high-resolution file
from the training file, the sharpened high-resolution file being
used to comput the interpolation filters.
32. The apparatus of claim 30, wherein the instructions instruct
the processor to generate a classifier including the representative
vectors by initializing the classifier and updating the classifier
until optimal values for the classifier have been obtained.
33. The apparatus of claim 30, wherein the instructions instruct
the processor to generate each cluster vector by forming an
observation window about sampled data in the low-resolution file,
extracting a vector including neighboring data of the sampled data,
subtracting a value of the sampled data from values of the data in
the vector; and scaling the vector.
34. The apparatus of claim 30, wherein the instructions instruct
the processor to compute coefficients for the interpolation filters
by: computing a number NFDV of filter design triplets from data in
the low-resolution file, where NFDV is a positive integer, each
filter design triplet corresponding to sampled data in the
low-resolution file, each filter design triplet including an
observation vector for the sampled data, a cluster vector for th
sampled data, and a vector of high resolution data from a
high-resolution fil, the high resolution data corresponding to the
sampled data; computing training statistics from the filter design
triplets; and computing the coefficients from the training
statistics.
35. The apparatus of claim 30, wherein the interpolation filters
are linear filters.
36. An article of manufacture for instructing a processor to
compute a resolution synthesis database from a training image, the
article comprising: computer memory; and a plurality of executable
instructions stored in the computer memory, the instructions, when
executed, instructing the processor to compute a low-resolution
image from the training image; compute a plurality of
representative vectors from th low-resolution image; and comput a s
t of int rpolation filt rs for ach of the representative vectors;
whereby the interpolation filters and the repres ntative vectors
form a part of the database.
37. The article of claim 36, wherein the instructions instruct the
processor to compute the representative vectors by computing a
number NCV of cluster vectors from the low-resolution image, and
using the cluster vectors to compute the representative vectors;
and wherein the instructions instruct the processor to compute the
interpolation filters from low-resolution observation vectors, the
cluster vectors, the representative vectors and vectors from a
high-resolution image corresponding to the low-resolution
image.
38. The article of claim 37, wherein the instructions further
instruct the processor to compute a sharpened high-resolution image
from the training image, the sharpened high-resolution file being
used to compute the interpolation filters.
39. The article of claim 37, wherein the instructions instruct the
processor to compute a classifier including the representative
vectors by initializing the classifier and updating the classifier
until optimal values for the classifier have been obtained.
40. The article of claim 37, wherein the instructions instruct the
processor to compute each cluster vector by forming an observation
window about a sampled pixel in the low-resolution image,
extracting a vector including neighboring pixels of the sampled
pixel, subtracting a value of the sampled pixel from values of the
pixels in the vector; and scaling the vector.
41. The article of claim 37, wherein the instructions instruct the
processor to compute coefficients for the interpolation filters by:
computing a numb r NFDV of filter design triplets from pix Is in th
low-resolution image, where NFDV is a positive integ r, ach fift r
design triplet corresponding to a sampled pixel in the
low-resolution image, each filter design triplet including an
observation vector for the sampled pixel, a cluster vector for the
sampled pixel, and a vector of high resolution pixels from a
high-resolution image, the high resolution pixels corresponding to
the sampled pixel; computing training statistics from the filter
design triplets; and computing the coefficients from the training
statistics.
42. The article of claim 36, wherein the representative vectors are
generated by using a parameter optimization technique.
43. An article of manufacture comprising: computer memory; and a
database encoded in the computer memory, the database including a
plurality of sets of resolution synthesis parameters, each set
corresponding to an interpolation factor, each set including a
classifier and a number M of resolution synthesis filters, each
classifier including a number M of representative vectors, where M
is a positive integer.
44. The article of claim 43, wherein each classifier further
includes a variance and a number M of class weights.
45. The article of claim 43, wherein the number M is between 50 and
100.
Description
[0001] This is a continuation-in-part of Ser. No. 08/837,619 filed
on Apr. 21, 1997.
BACKGROUND OF THE INVENTION
[0002] The invention relates to digital imaging. More specifically,
the invention relates to interpolation of higher resolution images
from lower-resolution imag s.
[0003] Image interpolation is performed routinely by printers and
computers. In one instance, a printer might receive source image
data having a fixed resolution of 75 dots per inch (dpi), yet the
printer is commanded to print at a higher resolution such as 300
dpi. In such an instance, the printer performs interpolation on the
source image data.
[0004] In another instance, a computer might perform interpolation
in ord r to display an image compressed according to a lossy
algorithm such as JPEG. Once the image is displayed, the computer
might receive a user command to zoom in on a portion of the image
to magnify a particular detail. In response to the us r command,
the computer would perform interpolation on the pixels in that
portion.
[0005] Simple linear interpolation lacks the precision necessary to
reproduce an image with quality at a higher resolution. Edges in
the interpolated image have poor quality. Shading problems and
jagged transitions can be viewed wh n an interpolated image is
displayed or printed. The shading problems and jagg d transitions
become even worse when a region of the interpolated image is
magnified.
[0006] Low order B-spline methods such as pixel replication and
bilin ar interpolation are satisfactory in terms of interpolating
smooth textures, and th y are easy to implement in terms of
processing power and memory requirem nts. However, pixel
replication tends to produce sharp edges that are not straight, and
bilinear interpolation tend to produce images that include
artifacts and relatively blurry lines.
[0007] Higher-ord r B-spline interpolation methods such as cubic
B-spline interpolation tend to provide smooth, continuous images.
However, smoothn ss and continuity come at a cost: increased
computational power. Higher-order B-spline interpolation methods
are computationally intensive. They also give ringing effects.
Additionally, B-spline methods in general are linear methods and,
therefore, are limited in terms of quality of the
interpolation.
[0008] Other interpolation methods include edge-directed methods,
fractal interpolation and methods that employ stochastic models.
For an example of a method that employs a stochastic model, see
Schultz et al., "A Bay sian approach to image expansion for
improved definition," IEEE Transactions on Image Processing, vol.
3, no. 3, pp. 233-242 (May, 1994). The method disclosed therein is
computationally intensive.
[0009] There is a need for an image interpolation method that
produces high quality images, yet is easy to implement in terms of
processing power and memory requirements.
SUMMARY OF THE INVENTION
[0010] The invention can be regarded as a method of generating an
electronic database of interpolation parameters that can be used to
produce high quality images. The parameters can be computed from
one or more training files. A low-resolution file is computed from
each training file. A plurality of representative vectors from each
low-resolution file are then computed. Next, a set of interpolation
filters are computed for each of the representative vectors. The
interpolation filters and the representative vectors can be used to
perform interpolation on a file other than the training file.
[0011] The method of generating the electronic database can be
performed off-line. Thus, by the time the parameters in the
database are used for interpolating an image, th bulk of th
computational activity has already been performed. Consequently,
interpolation tim and m mory requirements are reduced.
[0012] The training file could be an image file. The electronic
database could be stored in a device such as computer or a
printer.
[0013] Other aspects and advantages of the present invention will
become apparent from the following detailed description, taken in
conjunction with th accompanying drawings, illustrating by way of
example the principles of the invention.
BRIEF DESCRIPTION OF THE DRAWINGS
[0014] FIG. 1 is a block diagram of a system according to the
present invention, the system including a printer and first and
second computers;
[0015] FIGS. 2a, 2b and 2c are flowcharts of different ways in
which the first computer sends an image to the printer and the
printer prints an interpolated image;
[0016] FIG. 3 is an illustration of an image interpolation method
according to the present invention;
[0017] FIG. 4 illustrates the generation of a cluster vector, which
forms a part of the image interpolation method according to the
present invention;
[0018] FIG. 5 is an illustration of a non-linear filtering
operation, which forms a part of the image interpolation method
according to the present invention;
[0019] FIG. 6 is a method of generating a database of parameters
for the non-linear filtering operation; and
[0020] FIG. 7 is a more generalized method of generating the
database.
DETAILED DESCRIPTION OF THE INVENTION
[0021] As shown in the drawings for purposes of illustration, the
present invention is embodied in a m thod of g n rating an
electronic database of param ters us d for int rpolating higher
resolution images from lower resolution images. An interpolation
method uses the parameters to produce images of high quality, yet
the interpolation method is easy to implement in terms of
processing power and memory requirements. The electronic database
can be used advantageously by a system including a computer and a
printer. However, as will be discussed below, the invention is not
limited to generating a database for image interpolation by the
computer or printer. More generally, the invention can be applied
to th restoration or enhancement of speech, still images, video and
other multidimensional data.
[0022] FIG. 1 shows a system 10 including a first computer 12 and a
printer 14 that communicate over a cable 16. A resolution synthesis
database 18 is stored in either computer memory 20 (e.g., a hard
drive, an EEPROM) or in printer memory 22 (e.g., an EEPROM). The
database 18 provides resolution synth sis parameters that are used
for interpolating higher resolution images from lower resolution
images. Resolution synthesis (i.e., interpolation using the
resolution synthesis database 18) can be performed by a host
processor 24 in th first computer 12 or by an embedded processor 26
in the printer 14.
[0023] The database 18 includes a first group of parameters for
resolution synthesis at a first interpolation factor, a second
group of parameters for resolution synthesis at a second
interpolation factor, a third group for resolution synthesis at a
third interpolation factor, and so on. Any positive integer could
provide a suitable interpolation factor. If, for example, the image
has a resolution of 75 dpi, but it will be printed at a resolution
of 300 dpi, a group of resolution synthesis parameters
corresponding to an interpolation factor of four will be accessed
from the database 18.
[0024] The database 18 can be generated off-line by the first
computer 12 or another (second) computer 2. The second computer 2
includes a processor 4 and computer memory 6. A program 8 is stored
in the memory 6 of the s cond computer 2. The program 8 includes a
plurality of executable instructions that, wh n executed, instruct
the processor 4 of the second computer 2 to gen rat th database 18.
A method of generating the database 18 will b d scribed below in
connection with FIGS. 6 and 7. Once computed, the database 18 is
transferred from the second computer 2 to the computer memory 20 of
the first computer 12 (via a CD ROM, for example) or to the printer
memory 22 of the printer 14. The method of generating the database
18 is performed off-line. Since the bulk of the computational
activity is done prior to the actual interpolation, interpolation
tim and memory requirements of the first computer 12 and the
printer 14 are reduced.
[0025] Additional reference is now made to FIGS. 2a, 2b and 2c,
which illustrate three different examples in which the first
computer 12 and the printer 14 us the database 18 to perform
resolution synthesis. In the first example, as shown in FIG. 2a,
the database 18 is stored in computer memory 20. To print an image
in a source image file, a printer driver 19 instructs the first
computer 12 to access the database 18 from its memory 20 (block
200), perform resolution synthesis on the source image file to
compute an interpolated image (block 202), and send a file (block
204) including the interpolated image to the printer 14 for
printing.
[0026] In the second example, as shown in FIG. 2b, the database 18
is also stored in computer memory 20, but the first computer 12
sends the source image file and a file including the database 18 to
the printer 14 (block 210). Und r instruction of a program 21
stored in the printer memory 22, the embedd d processor 26 of the
printer 14 performs resolution synthesis on the source image file
(bock 212) and prints out the interpolated image.
[0027] In the third example, as shown in FIG. 2c, the database 18
is stored in the printer memory 22. The computer 12 sends only the
source image file to the printer 14 (block 220), which performs
resolution synthesis on the source image file (block 222) and
prints out the interpolated image.
[0028] FIG. 3 illustrates a method of performing the resolution
synthesis on a pixel L4 of an input image 30. An interpolation
factor of 2 will be used by way of example. A small observation
window 32 is located about the pixel L4 to be interpolated (the
"sampled pixel") and encompasses neighboring pixels of the sampled
pixel L4. The observation window 32 provides suffici nt information
for th sampl d pixel L4 to be classified and int rpolated Th
observation window 32 could be a 3.times.3 window, a 3.times.5
window, a 5.times.5 window, or larg r. By way of example, however,
the observation window 32 will hereinafter be described as a
3.times.3 window centered about the sampled pixel L4.
[0029] An observation vector L and a cluster vector y are extracted
from the observation window 32. The observation vector L is
extracted by stacking pixel values (e.g., intensity values of the
luminance component) of the pixels in the observation window 32.
Thus, nine pixel values would be stacked into the observation
vector L extracted from a 3.times.3 window. Such an observation
vector L would be represented as
L={L0, L1, L2, L3, L4, L5, L6, L7, L8}.sup.t (1)
[0030] where t is a vector transpose.
[0031] The cluster vector y is extracted (by block 34) using a
nonlinear process. The cluster vector y contains information about
a general type of image behavior (e.g., an edge, a smooth surface)
in the observation window 32. To begin cluster vector extraction
for the 3.times.3 window shown in FIG. 3, the eight nearest
neighbors of the sampled pixel L4 are stacked into a first vector
y1, where
y1={L0, L1, L2, L3, L5, L6, L7, L8}.sup.t. (2)
[0032] A nominal value such as the value of the sampled pixel L4 is
subtracted from each value in the first vector y1 to yield a second
vector y2:
y2={L0-L4, L1-L4, L2-L4, L3-L4, L5-L4, L6-L4, L7-L4, L8-L4}.sup.t
(3)
[0033] Thus, the second vector y2 indicates differences between the
sampled pixel L4 and its neighboring pixels. The cluster vector y
is computed by scaling the second vector y2. Scaling can be
performed by performing a projection operation f on the second
vector y2. A non-linear scaling operation might be perform d as
follows: 1 ; y2 r; = i = 0 i 4 i = 8 ( Li - L4 ) 2 ( 4 ) 2 y = f (
y2 ) = { y2 ; y2 r; p - 1 if y2 0 0 else ( 5 )
[0034] where p is any scalar between zero and one. For example, the
scalar p can equal 0.25. The projection function ultimately affects
how well the edges and details are rendered in the interpolated
image. Such generation of the cluster vector y is illustrated in
FIG. 4.
[0035] A non-linear filtering operation is performed (by block 36)
on th 9-dimensional observation vector L and the 8-dimensional
cluster vector y. Parameters used to perform the non-linear
filtering operation are accessed from the database 18. An output of
each non-linear filtering operation produces a 2.times.2 window 38
of interpolated pixels H0, H1, H2, H3.
[0036] Thus, interpolation of a single pixel L4 in the input image
30 has been described. To interpolate the other pixels in the input
image 30, the observation window 32 is moved around from pixel to
pixel in the input image 30. For each non-border pixel L4 in the
input image 30, four interpolated pixels H0, H1, H2, H3 are
produced by the non-linear filtering. Border pixels can be
interpolated by substituting the closest known pixels for all
unknown pixels. A border pixel might be a pixel that cannot be
centered within the observation window 32. After all of the pixels
in the input image 30 have been interpolated, there is an interp
lated image 40 having a resolution twice that of the input image
30.
[0037] The interpolation method can be applied to monochrome images
and color images. When a color image is interpolated, cluster
vectors from a luminance component of the color image are
extracted, and the cluster vectors are us d to perform non-linear
filtering in each of the constituent Red-Green-Blue plan s of the
color image. Thus, the same database parameters are used to perform
non-linear filtering on the Red plane as are used to perform
non-linear filtering on the Green and Blue planes.
[0038] FIG. 5 shows the non-linear filtering operation in detail.
Inputs include the resolution synthesis parameters stored in the
database 18. The parameters includ a number M of repr sentative
vectors RV, a number M of class weightings CW, a variance VAR and a
s t of int rpolation filter coeffici nts a and b. Th representative
v ctors RV, th relative class weights CW and th variance VAR will
collectively be referred to hereinafter as a "classifier." Each
representativ vector RV represents a different image behavior.
Examples of image behavior include horizontal edges, vertical
edges, angled edges, smooth textures, etc. There might be between
50 and 100 representative vectors RV stored in the database 18.
Thus, there might be between 50 and 100 representative vectors RV
that can be used to synthesize the interpolated image 40 from the
input image 30. The class weights CW correspond to how often the
different representativ vectors RV or behaviors occur. They convey
the relative frequencies of th representative vectors RV. Thus, the
class weight for flat pixels would typically be quite large
relative to the class weight for vertical edges. The variance VAR
is a positive number representing the level of variation within
classes associated with the representative vectors.
[0039] The pixels in the input image 30 are sampled one at a time.
When a pixel is sampled, its observation vector L and its cluster
vector y are also provided as inputs to the non-linear filtering
operation.
[0040] The cluster vector y is classified to determine the type of
image behavior within the observation window 32 (block 42). That
is, the cluster vector y is classified to determine whether the
sampled pixel is part of a horizontal edge, a smooth texture, etc.
A classification parameter p(j.vertline.y) is computed for j=0 to
M-1 to indicate the representative vector RV or vectors RV that
best represent the cluster vector of the sampled pixel. The
classification parameter p(j.vertline.y) for a cluster vector y can
be computed as follows: 3 p ( j | y ) = CW j exp ( - ; y - RV j r;
2 2 VAR ) d = 0 M - 1 ( CW d ( - ; y - RV d r; 2 2 VAR ) ) ( 6
)
[0041] wh re 4 ; y - RV j r; 2 = m = 0 7 ( y ( m ) - RV j ( m ) ) 2
( 7 )
[0042] In equation (7), y(m) is the m.sup.th element in the cluster
vector y, and RV.sub.j (m) is the m.sup.th element in the j.sup.th
representative vector RV.
[0043] Equations (6) and (7) are derived under the assumption that
a particular behavior might encompass a population of cluster
vectors, with certain cluster vectors being more likely to fall
under that behavior than others. Tak an example in which pixels in
the upper half of the observation window 32 are at full intensity
while pixels in the lower half of the observation window 32 are at
zero intensity. A cluster vector extracted from such a window 32
would indicate with high probability that the sampled pixel in the
observation window 32 is part of an edge. If, however, the pixels
in the lower half of the observation window 32 are half-scale, the
probability is not as high.
[0044] Moreover, the equations (6) and (7) are derived under the
assumption that the cluster vectors y are distributed as a
multivariate Gaussian mixture. Tak n as a group, the distributions
of the various behaviors form a Gaussian mixture model. The
distributions overlap. Consequently, a cluster vector (such as the
cluster vector for an observation window having the upper half of
its pixels at full scale and the lower half of its pixels at
half-scale) might indicate a behavior that falls under multiple
distributions. Thus, classification of a cluster vector y according
to equations (6) and (7) is equivalent to computing probabilities
of class membership in the Gaussian mixture model. The
classification parameter p(j.vertline.y) for a cluster vector y
indicates a series of weightings or probabilities corresponding to
the different behaviors and, therefore, the different
representative vectors RV.
[0045] Classifying a cluster vector y reveals the set or sets of
interpolation filters 44 that should be used to interpolate the
sampled pixel. Each set of interpolation filters 44 corresponds to
a representative vector RV. After a cluster vector y has been
classified, the corresponding observation vector L is filtered by
th coeffici nts a(j,k), b(j,k) of the selected interpolation filter
or filters 44, and the output of each filter 44 is multiplied (by
blocks 46) by the probability that the observation vector L is
represented by the representative vectors RV corresponding to that
interpolation filter. Products of the multiplication are summed
together (block 48). Thus, an interpolated pixel H(k) is computed
as follows: 5 H ( k ) = j = 0 M - 1 ( a ( j , k ) L + b ( j , k ) )
p ( j | y ) ( 8 )
[0046] for k=0, 1, 2, and 3, where H(k) is the k.sup.th
interpolated pixel in a vector h, and a(j,k) and b(j,k) are scalars
representing the filter coefficients for the j.sup.th class and the
k.sup.th interpolated pixel.
[0047] A computation could be performed for each interpolation
filter 44, and outputs of all of the interpolation filters 44 could
be multiplied by the classification parameter p(j.vertline.y).
However, if a computation is to be performed for ach interpolation
filter 44, computational time might be excessive. Moreover,
interpolating with representative vectors RV that are different
from the image data might result in a waste of computational
resources. Therefore, a different non-linear filtering operation
might use the outputs of only one or two of the interpolation
filters 44. This different operation would be more computationally
efficient if many interpolation filters 44 were involved.
[0048] The interpolation method described in connection with FIGS.
3, 4 and 5 is also described in U.S. Ser. No. 08/837,619 filed Apr.
21, 1997 and entitled "Computerized Method for Improving Data
Resolution." U.S. Ser. No. 08/837,619, which is assigned to the
assignee of the present invention, is incorporat d herein by
reference.
[0049] FIG. 6 shows a training method of generating the resolution
synthesis parameters for the electronic database 18. The parameters
are computed from at least one, but preferably more than one
training image. Training images can include photo-quality color
images, 600 dpi black and whit images, etc. A collection of digital
images are commonly available on photo CD and might provide a
possible source for the training images. To simplify th d scription
of the training method, the training method will hereinafter be
described in connection with only a single training image.
[0050] The general premise of the training method is to generate a
corrupted (e.g., low resolution) image from the training image and
then compute a set of resolution synthesis parameters that can be
used to predict or estimate th uncorrupted training image from the
corrupted training image.
[0051] The training image is converted to a monochrome image, if
necessary (block 100). For example, if the training image is a
color image, it can be converted to a representation having a
single plane. In the alternative, the luminance component of the
color image can be extracted, whereby values of the pixels in the
observation vector L would represent intensities of pixels in the
luminance component.
[0052] A scaled-down or low resolution rendering of the monochrome
imag is computed (step 102). The interpolation factor dictates the
level of image scaling. For an interpolation factor of two, each
pixel in the low-resolution image is the average of the
corresponding 2.times.2 block of pixels in the monochrome image.
Specifically, the pixel LR(i,j) in the i.sup.th row and j.sup.th
column of the low-resolution image is computed according to
equation (9): 6 LR ( i , j ) = T ( 2 i , 2 j ) + T ( 2 i + 1 , 2 j
) + T ( 2 i , 2 j + 1 ) + T ( 2 i + 1 , 2 j + 1 ) 4 ( 9 )
[0053] where T(i,j) is the i.sup.th row and j.sup.th column of the
training image. If the monochrom training image has a height of H
pixels and a width of W pixels, the low-resolution image has a
height of H/2 pixels and a width of L/2 pixels, the number of pix
is being rounded down to an integer.
[0054] A sharpened high-resolution rendering of the monochrome
image may also b computed (st p 104). The high-resolution image can
be sharpened using an unsharp mask. A pixel SHR(i,j) in a sharp
ned, high-resolution image can be computed as follows: 7 A ( i , j
) = 1 9 m = - 1 1 n = - 1 1 T ( i + m , j + n ) ( 10 )
SHR(i,j)=T(i,j)+.lambda.(T(i,j)-A(i,j)) (11)
[0055] where .lambda. is 1.0. Generally, .lambda. is a non-negative
constant which controls the level of sharpening and A(i,j) is an
average of the pixels in a neighborhood around pixel (i,j). The
dimensions of the sharpened, high-resolution image are the same as
the dimensions of the monochrome image. This step 104 is optional.
The uncorrupted training image can be used in place of the
sharpened image. Howev r, sharpening the training image improves
edge quality in the interpolated image without the need for
post-processing. For exemplary purposes, the training method will
be described in connection with the sharpened high-resolution
image.
[0056] A number NCV of cluster vectors are then extracted from the
low-resolution image (block 106). The cluster vectors can be
extracted from an observation window of any size and shape.
Although the cluster vector extraction will b described hereinafter
in connection with a 3.times.3 pixel observation window, a larger
window such as a 5.times.5 pixel window could be used.
[0057] As for the number NCV of cluster vectors that may be
extracted from th low resolution image, there are as many cluster
vectors available as there are low-resolution pixels that are not
on an image border. A reasonable rang of numbers of cluster vectors
is between 25,000 and 100,000. Using 25,000 cluster vectors will
yield a fair-quality classifier at moderate computational cost,
whil using 100,000 cluster vectors will yield a high-quality
classifier at significantly greater computational cost.
[0058] The cluster vectors are extracted from spatially different
regions of the low-resolution image. To ensure this, the pixels are
sampled at a period S1=N/NCV, where N is the total number of
cluster vectors available. The sampling period S1 is truncated to
an integer. Thus, a cluster vector is extracted at every S1.sup.th
pixel of th low-r solution image. Th pix is ar sampl d in a raster
ordering, with the primary dir ction being from left to right and
with the secondary dir ction b ing from top to bottom (th
"rastering order").
[0059] For a 3.times.3 observation window, the cluster vector can
be computed according to equations (2) to (5). However, it is not
required to subtract the nominal value from the neighboring pixels
in the first vector y1, nor is it required to scale the second
vector y2. However, subtracting the nominal value and scaling will
make it easier to discern the different types of cluster vectors.
The purpos and effect of scaling is to warp the space of all of the
cluster vectors so that edges of different magnitude but similar
orientation or shape are grouped together.
[0060] After the cluster vectors have been extracted, a number M is
selected (block 108). The number M represents the number of
representative vectors that will be computed for the low-resolution
image. The number M might be between 50 and 100. Larger numbers M
of representative vectors will allow for more freedom because a
wider variety of behaviors can be represented and used for
resolution synthesis. However, the larger numbers M of
representative vectors will also require more memory and
computational resources.
[0061] After the number M is selected, the classifier (i.e., the
representative vectors RV, the class weights CW and the variance
VAR) are computed (blocks 110 to 118). A method of maximum
likelihood estimation can be used to comput the classifier. One
instantiation of the method of maximum likelihood estimation is the
well-known expectation-maximization (EM) algorithm applied to
Gaussian mixture models. Initial values for the classifier are
selected and then iteratively updated until a locally optimal set
of parameters has been obtained.
[0062] Thus, initial values are selected for the classifier (block
110). The initial values for the class weights CW can be selected
as follows: 8 CW 0 ( 0 ) = 1 M , , CW M - 1 ( 0 ) = 1 M ( 12 )
[0063] The superscripts "(0)" indicate initial (zeroth) values for
the class weights CW. Th class weights CW are positive numbers that
add up 1. That is, CW.sub.0>0, . . . , CW.sub.M-1>0 and
.SIGMA.CW.sub.i=1 from i=0 to i=M-1.
[0064] Th initial valu s for th repres ntative vectors RV.sub.0, .
. . , RV.sub.M-1 can b s t equal to the cluster vectors sampled at
every S2.sup.th pixel of th low resolution image, where the
sampling period S2=NCV/M. Thus, 9 RV i ( 0 ) = y S2xi ( 13 )
[0065] for i=0, . . . , M-1. The sampling period S2 ensures that
the representative vectors RV.sub.0, . . . , RV.sub.M-1 are
initialized by cluster vectors from spatially different regions of
the low-resolution image. The cluster vectors are selected from
spatially separate regions of the low resolution image (or from
different images if multiple images are available for training) to
encourage the formation of distinct representative vectors.
[0066] The variance VAR can be initialized as follows: 10 VAR ( 0 )
= 1 NCV .times. 8 m = 0 7 i = 0 NCV - 1 ( y i ( m ) - y _ ( m ) ) 2
( 14 )
[0067] where y.sub.i(m) is the m.sup.th element in the i.sup.th
cluster vector, and {overscore (y)}(m) is the sample mean of the
m.sup.th element in the cluster vectors. The sample mean {overscore
(y)}(m) can be calculated as follows. 11 y _ ( m ) = 1 NCV i = 0
NCV - 1 y i ( m ) ( 15 )
[0068] Next a log likelihood LL(kl) is computed for the initial
values (block 112). A log likelihood LL(yi;kl) for each cluster
vector is calculated according to equations (16) and (17), and the
log likelihood LL(kl) is computed by adding th log likelihoods of
each cluster vectors according to equation (18): 12 ; y i - RV j (
kl ) r; 2 = m = 0 7 ( y i ( m ) - RV j ( kl ) ( m ) ) 2 ( 16 ) LL (
y i ; kl ) = log [ j = 0 M - 1 CW j kl ( 2 VAR ( kl ) ) 4 exp ( - 1
2 VAR ( kl ) ; y i - RV j ( kl ) r; 2 ) ] ( 17 ) LL ( kl ) = i = 0
NCV - 1 LL ( y i ; kl ) ( 18 )
[0069] where the index kl indicates the kl.sup.th iteration (for
example, LL(0) is the log likelihood for the initial values);
y.sub.i(m) is the m.sup.th element in the i.sup.th cluster v ctor;
RV.sub.j.sup.(kl)(m) is the m.sup.th element of the j.sup.th
representative vector in the kl.sup.th iteration; LL(y.sub.i; kl)
is the log likelihood of an individual cluster vector y.sub.i
during the kl.sup.th iteration; log is base e, and "exp" denotes
the exponential function. The log likelihood is a number that is
analogous to a measure of the quality of the current values for the
classifier. A higher log likelihood indicates a better "fit" for
the classifier.
[0070] After the log likelihood LL(0) is computed for the initial
values of the classifier, the classifier is updated. The index kl
is incremented (kl.fwdarw.kl+1) and the updates are made as
follows. 13 p ( j | y i ; kl - 1 ) = exp ( - ; y i - RV j ( kl - 1
) r; 2 2 VAR ( kl - 1 ) ) CW j ( kl - 1 ) d = 0 M - 1 exp ( - ; y i
- RV d ( kl - 1 ) r; 2 2 VAR ( kl - 1 ) ) CW d ( kl - 1 ) ( 19 )
NCV j ( kl ) = i = 0 NCV p ( j | y i ; kl - 1 ) ( 20 ) CW j ( kl )
= NCV j ( kl ) NCV ( 21 ) RV j ( kl ) = 1 NCV j ( kl ) i = 0 NCV -
1 y i p ( j | y i ; kl - 1 ) ( 22 ) VAR ( kl ) = 1 8 j = 0 M - 1 CW
j ( kl ) NCV j ( kl ) i = 0 NCV - 1 ; y i - RV j ( kl ) r; 2 p ( j
| y i ; kl - 1 ) ( 23 )
[0071] The parameter p(j.vertline.y.sub.i;kl-1) can be considered a
measure of the extent to which a cluster vector y.sub.i belongs to
the jth representative vector of the (kl-1).sup.th it ration.
NCV(kl) can be considered as an estimate of the number of cluster
vectors in the jth class of the previous iteration. Th updates in
equations (21), (22) and (23) can be considered as sample
statistics that are weighted and normalized in accordance with the
classifications under the previous iteration.
[0072] Next, a log likelihood LL(1) is computed for the updated
classifier (block 116). The log likelihood LL(1) for the updated
classifier is also computed according to the equations (16), (17)
and (18) above.
[0073] A decision is then made as to whether the current values for
the representative vectors RV, the class weights CW and the
variance VAR are optimal (block 118). The decision can be made by
taking the difference between the log likelihoods of the current
and previous iterations LL(1) and LL(0) and comparing the
difference to a threshold THRES. The threshold THRES indicat s
whether the likelihood is approaching a local maximum. The
threshold THRES can be computed as follows:
THRES=0.09.times.log(8.times.NCV) (24)
[0074] where the log is base e. If the difference is greater than
the threshold THRES, another update is performed (block 114). If
the difference is less than the threshold THRES, the values of the
kl.sup.th iteration for the classifier are stored in the database
(block 120).
[0075] After the classifier has been computed, the coefficients a,
b for the interpolation filters are computed (blocks 122 to 126). A
number NFDV of filter design triplets are computed from pixels in
the low resolution image (block 122). Each filter design vector
triplet includes a cluster vector y, an observation vector L which
contains low-resolution pixels, and a vector h which contains
high-resolution pixels. All three of these vectors y, L and h
correspond to exactly one pixel in the low-resolution image. The
set of filter design vector triplets will be referred to as 14 { (
y i , L i , h i ) } NFDV - 1 i = 0 ,
[0076] where (y.sub.i,L.sub.i,h.sub.i) is the filter design vector
tripl t for the i.sup.th pixel in the low-resolution image.
[0077] As for the number NFDV of design triplets, there may be as
many filter d sign vector triplets available as th re are low-r
solution image pix Is from which cluster v ctors may b extracted.
Higher numbers NFDV of filt r d sign vector triplets will yield
better results at th cost of incr ased computation. Satisfactory
results can be obtained by using 500,000 filter design vector tripl
ts, while better results can be obtained by using over 1,000,000
filter design vector triplets.
[0078] The high-resolution pixel vector h is computed by stacking a
2.times.2 block of high-resolution pixels which correspond to a
pixel in the low-resolution image. Specifically for a
low-resolution pixel LR(i,j) at location i,j in the low resolution
image, the corresponding pixels in the 2.times.2 block of the high
resolution image are at locations (2*i,2*j), (2*i+1, 2*j), (2*i,
2*j+1), and (2*i+1, 2*j+1).
[0079] The filter design triplets can be sampled from spatially
different regions of the low-resolution image. A sampling period S3
might be N/NFDV, where N is the total number of filter design
vector triplets available. The sample period S3 is truncated to an
integer. One by one, in the rastering order, a filter design vector
triplet is extracted at every S3.sup.th pixel in the low-resolution
image.
[0080] Next, training statistics are computed for each
representative vector (block 124). Training statistics for the
j.sup.th representative vector can be computed as follows: 15 NFDV
j = i = 0 NFDV - 1 p ( j | y i ) ( 25 ) m L j = 1 NFDV j i = 0 NFDV
- 1 L j p ( j | y i ) ( 26 ) mh j = 1 NFDV j i = 0 NFDV - 1 h i p (
j | y i ) ( 27 ) GLL j = 1 NFDV j i = 0 NFDV - 1 ( L i - m L j ) (
L i - m L j ) t p ( j | y i ) ( 28 ) GhL j = 1 NFDV j i = 0 NFDV -
1 ( h i - mh j ) ( L i - m L j ) t p ( j | y i ) ( 29 )
[0081] where mL.sub.j is a 9-dimensional vector and mh.sub.j is a
4-dimensional vector, GLL.sub.j is a 9.times.9 matrix, and
GhL.sub.j is a 4.times.9 matrix. Sup rscript "t" denotes the v ctor
transpos.
[0082] Finally, the coefficients a, b for th interpolation filters
are computed from the training statistics (block 126). Each
interpolation filter produces a pixel in th high resolution image.
The set of interpolation filters can be represented as 16 { { a ( j
, k ) , b ( j , k ) } 3 k = 0 } M - 1 j = 0 ,
[0083] where k represents the k.sup.th output pixel in the high
resolution image (for k=0, . . . 3). The vector a(j,k) is the kth
row of a 4.times.9 matrix Aj, and the scalar b(j,k) is the k.sup.th
element in a 4-dimensional vector bj. The matrix Aj and the vector
bj can be computed according to equations (30) and (31). Th
superscript "-1" denotes taking the matrix inverse (or
pseudo-inverse, if necessary).
A.sub.j=GhL.sub.j(GLL.sub.j).sup.-1 (30)
b.sub.j=mh.sub.j-GhL.sub.j(GLL.sub.j).sup.-1mL.sub.j (31)
[0084] The coefficients for the interpolation filters are stored in
the databas 18 (block 128). The database 18 can be stored on a
medium such as CD-ROM for transfer to another computer, it can be
transferred to another computer or print r via a network, or it can
be programmed into an EEPROM of a printer (block 130). The database
18 can be used as described above in connection with FIGS. 2a to
2c.
[0085] Thus disclosed are an interpolation method for producing
high quality interpolated images and a training method for
generating a database of parameters for the interpolation method.
The training method is run off-line in a computer. The database of
parameters need only be generated once, but it may be used many
times by the interpolation method. Once the parameters have been
generated, the interpolation method is easy to implement in terms
of processing power and memory requirements.
[0086] Although the system shown in FIG. 1 was described in
connection with a printer, the system is not so limited. The
invention could be applied to the Internet. For example, many web
pages display thumbnails of images. Clicking on a thumbnail causes
a higher resolution image to be downloaded. If a web brows r is
capable of performing resolution synthesis according to th present
invention, the higher resolution image need not be downloaded when
the thumbnail is clicked. Instead, the web browser would access the
database (from either its hard drive or the server) and generate a
high-resolution image from the thumbnail. The high-resolution image
would be displayed much faster than downloading a high resolution
image file over a modem connection. Additionally, less server
memory would be needed to store the web pages.
[0087] The invention can be used by any computer to increase
resolution of a displayed image. For example, the interpolation
method can be used by a computer to zoom in on an area of an image.
The zoomed-in area would hav smooth, continuous edges.
[0088] Moreover, the invention is not limited to image processing.
The invention could be applied to the enhancement or restoration of
speech, still image, video, and other multidimensional signals.
[0089] FIG. 7 shows a more generalized method of building a
database that could be applied to resolution synthesis of speech or
the restoration of images. Resolution synthesis parameters are
generated from at least one training file. All training files are
inputted (block 300), low-resolution files are generated from the
training files (block 302), and a plurality of representative
vectors are computed from the low-resolution files (block 304). A
set of interpolation filters is generated for each of the
representative vectors (block 306). The interpolation filters and
the representative vectors are stored in the database (block 308)
and can be used later to perform data resolution synthesis on a
file other than the training file. If generated from training files
including speech pattern data, the database can b used by a
computer to restore or synthesize low-quality audio files
containing speech. Similarly, if generated from training files
including data representing a damaged image, the database can be
used by a computer to restore the image.
[0090] Specific embodiments of the invention have been described
and illustrat d above. However, the invention is not limited to
these specific embodiments. Any number and type of training images
can be used to create the database. The low-resolution imag s can
be computed in any number of ways other than block av raging. The
optional sharpened images can b computed in any numb r of ways
other than unsharp masking.
[0091] The actual number NCV of cluster vectors will depend upon
the computational power of the computer that is generating the
database and the desired quality of the interpolated images.
Increasing the number NCV of the cluster vectors will also increase
the computational time. However, increasing the number NCV of
cluster vectors will increase the amount of information used for
interpolation and, therefore, will increase the quality of the
interpolated image. Therefore, it might be feasible to use far more
than 100,000 cluster vectors.
[0092] The same holds true of the number M of classes. Increasing
the number M of classes will increase computation time. However,
increasing the number of classes will increase the variety of
behaviors that can be recognized and used to perform image
interpolation. A single class could even be used, in which case the
best class would be selected. If only a single class is selected,
class weights would not have to be computed.
[0093] It is assumed that the observation window gives all the
information needed to interpolate a sampled pixel. Of course, the
assumption will hold truer for larger observation windows.
Therefore, it might be desirable to use an observation window that
is larger than 3.times.3 pixels.
[0094] Generating the representative vectors is not limited to the
expectation maximization technique. Any of a large number of
suitable parameter optimization techniques could be used for
generating representative vectors from a group of cluster vectors.
Examples include conjugate gradient and gradi nt search techniques
and simulating annealing.
[0095] Generating the cluster vectors is not limited to the methods
described above. The cluster vectors could be generated by a wide
variety of clustering methods such as iso-data clustering, K-means
clustering, general vector quantization, and tree-structure
quantization.
[0096] Therefore, th inv ntion is not limited to the specific
embodiments described and illustrated abov. Instead, th invention
is construed according to the claims that follow.
* * * * *