U.S. patent application number 09/222393 was filed with the patent office on 2001-11-29 for photocollage generation and modification.
Invention is credited to MCBRIDE, JOHN K., SHAFFER, STEPHEN L., SQUILLA, JOHN R..
Application Number | 20010046330 09/222393 |
Document ID | / |
Family ID | 22832001 |
Filed Date | 2001-11-29 |
United States Patent
Application |
20010046330 |
Kind Code |
A1 |
SHAFFER, STEPHEN L. ; et
al. |
November 29, 2001 |
PHOTOCOLLAGE GENERATION AND MODIFICATION
Abstract
A method and system for employing image recognition techniques
to produce a photocollage from a plurality of images wherein the
system obtains a digital record for each of the plurality of
images, assigns each of the digital records a unique identifier and
stories the digital records in a database; automatically sorts the
digital records using at least one date type to categorize each of
the digital records according at least one predetermined criteria;
and generates a first draft of a photocollage from the digital
records. The method and system provides for alterations of the
first draft using either a sticker selected from a plurality of
stickers which each have a unique identifier within a data base or
a digital file format that allows drag and drop features of both
the photocollage and alternative images to be placed within the
photocollage. The system employs data types selected from pixel
data; metadata; product order information; processing goal
information; or customer profile to automatically sort data
typically by culling or grouping to categorize according to either
an event, a person, or chronologically.
Inventors: |
SHAFFER, STEPHEN L.;
(PENFIELD, NY) ; SQUILLA, JOHN R.; (ROCHESTER,
NY) ; MCBRIDE, JOHN K.; (ROCHESTER, NY) |
Correspondence
Address: |
PATENT LEGAL STAFF
EASTMAN KODAK COMPANY
343 STATE STREET
ROCHESTER
NY
14650-2201
US
|
Family ID: |
22832001 |
Appl. No.: |
09/222393 |
Filed: |
December 29, 1998 |
Current U.S.
Class: |
382/284 ;
707/E17.023; 707/E17.026 |
Current CPC
Class: |
Y10S 707/99931 20130101;
G06F 16/5838 20190101; G06F 16/58 20190101; G06T 11/60 20130101;
Y10S 707/99943 20130101 |
Class at
Publication: |
382/284 |
International
Class: |
G06K 009/36 |
Claims
What is claimed is:
1. A method of constructing a photocollage comprising the steps of:
obtaining a version of a plurality of images; storing the digital
images in a database; determining a set of parameters that pertain
to the digital images such that each of the digital images contains
at least one of the parameters; sorting the images according to the
parameters; employing the parameters to automatically construct the
photocollage from the stored images by generating a plurality of
pages of stored images; and providing means for altering the
pages.
2. The method of claim 1 wherein the step of providing means for
altering the pages further comprises providing a set of stickers
wherein each of the stickers can be placed over one of the images
and each of the stickers has a thumbnail representation of one of
the images containing a unique identifier that cross references
that image to the data base.
3. The method of claim 1 wherein the photocollage which is
constructed in the step of employing is a scaled representation of
the photocollage.
4. The method of claim 1 wherein the step of providing means for
altering the pages further comprises providing means for using a
writing instrument to indicate modifications to the images and the
reconstruction of the photocollage.
5. The method of claim 1 wherein the step of employing the
parameters further comprises constructing the photocollage using a
set of data that is stored in the data base relating to one of the
following: pixel information; metadata; order information or
customer profile information.
6. The method of claim 1 wherein the step of employing the
parameters further comprises constructing the photocollage
employing one of the following techniques: culling; grouping;
enhancing; annotating; composing; or associating.
7. The method of claim 5 wherein the set of data was at least
partially contained in the data base prior to the step of obtaining
a digital version of a plurality of images.
8. The method of claim 7 wherein the step of employing the
parameters further comprises constructing the photocollage in
accordance with the set of data that is stored in the data base
relating to one of the following: pixel information; metadata;
order information or customer profile information.
9. The method of claim 2 further comprising the step of generating
a final version of the photocollage is accordance with alteration
made by the stickers.
10. The method of claim 2 further comprising the step of generating
a final version of the photocollage is accordance with the use of a
markup method using a writing instrument.
11. The method of claim 1 further comprising means for of adding
material to the photocollage.
12. The method of claim 11 wherein the means for adding additional
material further comprises means for adding additional images to
the photocollage that are stored in the data base.
13. The method of claim 11 wherein the means for adding additional
material further comprises means for adding material selected from
a predetermined set of graphics.
14. The method of claim 1 wherein the step of employing the
parameters to automatically construct the photocollage further
comprises selecting a theme to construct the photocollage.
15. The method of claim 1 wherein the step of employing the
parameter by selecting of a theme further comprises employing image
understanding principles to identify a key characteristic in at
least one of the image and apply the key characteristic throughout
the database to generate the photocollage in accordance with the
key characteristic.
16. The method of claim 1 wherein the photocollage is a computer
based photocollage contained in a digital file and the means for
altering the pages further comprises software means for modifying
the images on pages.
17. The method of claim 16 wherein the software means is operative
to display the photocollage and to replace images on the pages with
images selected from any image within the database.
18. The method of claim 16 wherein the software means is operative
to alter the page layout with a different page template.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is related to formerly filed applications
filed by the same inventors and assigned commonly with the present
application for invention entitled "Photocollage Generation And
Modification Using Image Recognition" filed on Nov. 25, 1998 as
Docket No.: 78616; and "System And Method Of Constructing A
Photoalbum" filed on Aug. 19, 1998 as Docket No.: 78616.
FIELD OF THE INVENTION
[0002] The invention relates generally to the field of photography,
and in particular to photo collections. More specifically, the
invention relates to generating photocollages automatically and
providing for alteration of the automatically created
photocollage.
BACKGROUND OF THE INVENTION
[0003] Photographs, videos, and memorabilia collections are very
commonly used to maintain memories and events that form part of an
individual's life. These collections serve to augment the human
memory and enrich the process of sharing stories related to the
memories. When organized, viewed and shared on a regular basis a
collection of memory artifacts generates a large reward, enriching
the lives of all involved. The nature of these collections is such
that they grow steadily, event by event, year by year, and soon
become large and difficult to manage. Collections of photos and
memorabilia are considered one of the most important and valued
possessions by most people. They are the first things that people
think of when forced to flee their homes due to fire, flood or
other natural disaster. These collections possess intrinsic,
emotional value, even if they are never viewed, because the need to
preserve a memory of life is strong and universal. Because of the
relative importance of these memories to the persons involved, the
prior art is replete with teachings that disclose organizational
methods.
[0004] The most common manner of organizing these collections
within the prior art is to place the photos, videos or memorabilia
into either an album or a box. Common vinyl album pages provide the
means to store and view between one and five standard sized photos
per page. Creative people often spend hours carefully selecting and
arranging photos, writing captions, clipping newspaper articles,
and other memorabilia to create visual stories or scrapbooks. Once
organized into groups or pages these photocollages greatly enhance
a person's ability to remember and share the story surrounding the
depicted events. These simple organization tools allow the
collections to be easily viewed and also serves to protect the
artifacts themselves. There are numerous types of albums and boxes
available in the market today, ranging from simple vinyl sleeves to
boxes manufactured from specialized materials designed to preserve
the artifacts. Album vendors include Pioneer Photo Albums, Design
Vinyl and Cason-Talens. Box vendors include Exposures. None of
these prior art disclosures provide a means by which a photocollage
of these memorable events can be easily constructed by persons to
who these event means so much.
[0005] As used herein photocollage refers to a single page having a
plurality of images, such as a page in a photo album, or a
composite image having a number of images relating to a single
theme such as a vacation, wedding, birthday party or the like. The
concept of photocollage as used herein also includes the concept of
a bound photo album having a plurality of pages, one or more of
which is a photocollage.
[0006] Despite the fact that many people are engaged in collecting
these memorable artifacts, few people have the free time available
to invest on a regular basis to organize and maintain them. Before
long, the amount of unorganized material becomes a significant
psychological barrier to getting organized. Other barriers exist
which prevent people from actively maintaining these memorabilia
collections such as confidence in their process, access to the
materials, or remembering the details about the event. Often, once
people get started on this organizational task they find it
rewarding and fun, but still a significant amount of work.
[0007] Many attempts have been made to provide tools for working
with or organizing photo and memorabilia collections. Computer
software programs such as Picture-It.TM., by Microsoft, or Creative
Photo Albums.TM., by Dog Byte Development, allow people to work
with digital versions of their photos and create digital versions
of an album or print them on a home printer. Software products such
as these require each photo or artifact exist in digital form
before they can be used. Although these products increase the
ability to change and enhance photos and scanned memorabilia they
do not reduce the amount of work needed to organize collections or
create visual stories. Other services such as Photo-Net.TM. by
PictureVision.TM. will scan photographs in a high-quality format at
the time of photo processing and provide a thumbnail image of the
scanned images via the Internet. A customer, using these scanned
images can create collections of photos which can be viewed on the
Internet or have prints generated. Currently some of these services
do not allow for the arrangement of several photos on a page and
are limited to consumers who have a collection of digital images
and a computer connected to the Internet and who are both computer
and web literate.
[0008] It should be apparent from the foregoing discussion that
there remains a need within the art for a method by which consumers
can create photocollages and photo albums (or have them made for
them) in a manner that is as simple as ordering prints.
SUMMARY OF THE INVENTION
[0009] The present invention addresses the shortcomings within the
prior art and provides an improved method of generating photo
albums from consumer photographs that requires a minimum amount of
effort but yields a high-quality product and is reasonably
priced.
[0010] The present invention is directed to overcoming one or more
of the problems set forth above. Briefly summarized, according to
one aspect of the present invention, a system and method for
producing a photocollage from a plurality of images, comprising the
steps of: a) obtaining a digital record for each of the plurality
of images, each of the digital records having a unique identifier
and storing the digital records in a database; b) automatically
sorting the digital records using at least one date type to
categorize each of the digital records according at least one
predetermined criteria; c) employing means responsive to the
sorting step to compose a photocollage from the digital records.
The system then associates each of the images with at least one of
the categories followed by a sorting step that arranges the images
according to the categories. The system then employs the categories
to automatically construct the photocollage from the stored images
by generating a plurality of pages of the stored images.
[0011] These and other aspects, objects, features and advantages of
the present invention will be more clearly understood and
appreciated from a review of the following detailed description of
the preferred embodiments and appended claims, and by reference to
the accompanying drawings.
ADVANTAGEOUS EFFECT OF THE INVENTION
[0012] The present invention has the following advantages: Allows
the user to have (1) an easy method for creating professional
looking photocollages, (2) duplication of photocollages, and (3)
keeping photocollage files for later use.
BRIEF DESCRIPTION OF THE DRAWINGS
[0013] FIG. 1 is a block diagram illustrating the basic system
elements used in practicing the present invention; and
[0014] FIG. 2 is a system diagram showing the collection steps that
take place once a customer has delivered images to the system;
[0015] FIG. 3 is a flow diagram showing the active processing goal
steps used by the system of the present invention;
[0016] FIG. 4 is a flow chart showing the steps performed by the
present invention towards a story preparation based
photocollage;
[0017] FIG. 5 is a flow chart showing the steps performed by the
present invention towards preparation of a photocollage that allows
alterations after automatic creation;
[0018] FIG. 6 is a flow chart showing the steps performed by the
present invention; towards preparation of a photocollage that is in
a digital file format that can be sent across the internet;
[0019] FIG. 7 is a flow chart showing the steps performed in the
processing facility illustrated in FIG. 2;
[0020] FIG. 8 is a schematic diagram showing a sticker sheet of
reduced resolution images to be used with the present
invention;
[0021] FIG. 9 is a drawing showing some of the layout styles
available for a page in a photocollage.
[0022] To facilitate understanding, identical reference numerals
have been used, where possible, to designate identical elements
that are common to the figures.
DETAILED DESCRIPTION OF THE INVENTION
[0023] An acceptable photo album or photocollage can be created
automatically from customers' exposed film, negatives, or digital
images.
[0024] Referring to FIG. 1, which is the method as envisioned by
the present invention designed to automatically produce
photocollages including albums 17, CDs 15 and other image-based
keepsakes 11. The process of transforming the supplied image
material into a photocollage is referred to as Story Preparation
services. Story Preparation services applies the necessary amount
of image understanding embodied in a collection of processing
modules executed in a non-linear sequence to create a story
representation of the images. In a preferred embodiment these
processing modules include: collecting, sorting, culling,
annotating, grouping, enhancing, associating and, composing. These
steps can be performed in the order listed or they may be
rearranged in different sequences depending upon the desired output
product. The production of a particular photocollage product may
include one, two or, more of the available steps. Which steps are
used and in what sequence is determined by the system based upon
the desired product. It is important to remember that the present
invention envisions providing sufficient image understanding of the
subject images with the ultimate goal of teaching the system to
understand that there are pervasive themes that exist within
various sets of images and recognizing these themes thereby
creating an image story product, or photocollage.
[0025] As shown in FIG. 2, the collection step begins when a
customer, having completed picture taking for one or more events
delivers one or more exposed film strips or cartridges 10, digital
still camera memory cards 12, photographic prints 6 or video camera
media 8 to a processing facility 14. At the time that the customer
delivers the exposed film cartridge(s) to the processing facility
the customer's identity is recorded and associated with the suite
of film cartridges and other image sources. Alternatively the
customer identity may be encoded on the film by exposing a
machine-readable sequence of marks along the edge of the film or by
using the magnetic coating on the Advanced Photo System film. The
conventional film processing takes place with conventional
correction of exposed photographs to balance color and brightness
and contrast. The exposed film images are chemically processed 16
to generate an optical image. These optical images are then scanned
18 to produce a high-resolution digital file that is archived in a
data store 20. In general, to produce a high-resolution printed
image of 8.times.10 inch size, a resolution of 1538.times.1024
pixels is required. Digital Still Camera images from Digital Still
Camera memory cards that are delivered to the processing facility
are digitally processed 22 to produce an equivalent digital image
file of similar size and stored on the data store 20. Analog video
camera images on media 8 delivered to the processing facility are
digitized and processed before storage on the data store 20. At the
conclusion of this step in the process there exists a collection of
digital image files associated with a customer by means of a
customer identification. This collection of digital image data, or
pixel data 32, is now available as a source of data for the Story
Services processing modules. Data attached or associated with
individual images or groups of images provided by the customer such
as date, time, location, sound, camera identification, customer
identification, exposure parameters, Advanced Photo System IX data,
are all examples of meta-data. This meta-data 34 is stored on image
store 20 and associated with the available pixel data 32. In
addition the customer selects the desired output product or product
type from a listing of available choices. This selection may be
indicated by the customer marking the photo processing bag 24,
verbally indicating the desired product to a clerk 26 or by input
at a computer terminal or kiosk 28. The order description 36
generated from any one of the indicated sources provided by the
customer is stored on the system and is associated, by means of a
customer identification, with an existing customer profile 30 in
the data store 20.
[0026] At the conclusion of the collection step the pixel data,
meta-data, customer profile and order description are available as
input data for the subsequent processing steps. These steps can be
performed in the order listed or they may be rearranged in
different sequences depending upon the desired output product. The
production of a particular photocollage product may include one,
two or, more of the available steps. Which steps are used and in
what sequence is determined by the system based upon the desired
product. Following the collection step but before subsequent
processing steps are performed, the specific processing to be
applied to the collected set of image material is determined in a
process goal generation step 37 and embodied in a set of
photocollage processing goals 38. For each of the output products
requested the system determines an optimum sequence of processing
steps to be applied. The processing of the output product or
products will follow this system determined processing sequence,
unless the sequence is modified by one of the processing modules.
The processing output for each module is stated as one or more
processing step goals. The processing in each step uses the stated
processing step goals to control the logic, rules, and parameters
applied during the processing of the collected image material to
satisfy the stated processing step goal or goals. In addition to
accomplishing processing goals each processing step can also create
goals for itself or future processing steps. New goals, which
result from executing a particular processing step, reflect
necessary adjustments of desired parameters because the original
goal is impossible to achieve or the processing step may generate
new goals for subsequent processing steps. The initial set of
processing goals 38 is determined by retrieving a default
processing description from a database of available products 40
maintained by the system. Once the initial set of processing goals
is determined the photocollage processing is initiated by the
system.
[0027] As described, the automatic processing of photocollages is
carried out in processing modules employed to perform these image
understanding and interpretation steps. Each processing module
relies on data from a number of sources in order to satisfy the
processing goal. Each processing module has access to five sources
of data: pixel information contained in the individual images,
meta-data attached to images or groups of images, the original
product or service order description, the list of active processing
goals maintained by the system and, a customer profile containing
information about the customer. The pixel data 32 for each image
provides each module with the opportunity to apply image processing
and image understanding algorithms. The meta-data 34 as used herein
refers to the information attached to the individual images and to
groups of images will contain information about the image or group
of images which originated at the time of capture or was generated
in prior processing of the image or group of images. Date, time,
location, sound, camera identification, customer identification,
exposure parameters, Advanced Photo System IX data, are all
examples of meta-data that are, in a preferred embodiment, be
attached to the original input images. The original product or
service order description 36 contains the specific product request
from the customer and any notes that were captured in the ordering
process. At any time in the processing of the image or group of
images the system will have a list of active processing goals 38.
Active goals are the processing goals for each processing module,
which have yet to be accomplished by the system. At the start of
processing this list will include the specific product or products
requested by the customer, along with the translation of these
product goals into system and module goals. The customer profile
includes both factual and uncertain information related to the
customer. The factual data would include general items such as
name, address, names and ages of individuals in the household,
important dates, anniversaries, product preferences and purchase
history. In addition factual data such as face recognition feature
vectors of the immediate family and extended family, voice training
sets, handwriting samples, would also be included in the customer
profile database. Uncertain information would include processing
goals recommended by processing modules from previous order
processing, image understanding assertions about the contents of
the image or groups of images which have not been verified or other
unverified information or assertions. Information and data
contained in the customer profile is updated with every order
processed to reflect changes in order preferences, order history
and update uncertain information. In order to supply the system
with the necessary amount of image understanding required to
arrange the images into set in accordance with themes relating to
predetermined criteria that must be provided for the system to have
the capability to identify attributes within the images. For
example, in a preferred embodiment, if a customer requests a
birthday photocollage the system will retrieve the default
processing goals which indicate that the steps of collecting,
sorting, culling, annotating, and composing will be involved in the
processing of the requested product. In addition the processing
goals for each of the modules will reflect the default attributes
necessary to process a birthday product. In this example the
sorting module processing goals will include a sort by date goal
and a sort by content goal. The sort by date processing goal is
further refined to sort the images that occur on or near a target
list of dates which are determined from a list of known birthdays
retrieved from the customer profile into the photocollage.
[0028] In a preferred embodiment each processing module performs
processing on two distinct levels: objective and subjective. The
objective processing deals with factual information about the
images or groups of images or with data calculated deterministic
algorithms. Examples of factual data include the size of the image,
capture parameters, histograms, image transforms, etc. Commercially
available software programs such as Adobe Photoshop.RTM. and Corel
Draw.RTM. process images using objective data. Subjective
processing deals with information that is uncertain or data is the
result of non-deterministic algorithms. Often subjective results
carry with them a confidence factor which allows subsequent
processing steps to interpret the results. Usually subjective
results occur when attempting to determine abstract information
about an image or group of images. Examples of this type of the
processing would be face detection, face recognition, facial
expression determination, location determination, assertions,
interpretations, etc. Commercially available software programs such
as FaceIT.COPYRGT. by Visionics Corp. process images to associate
faces in images with names and other information. Some processing
modules process only subjective information, others process only
objective information and still others process both.
[0029] As shown in FIG. 3 the processing of a photocollage is
directed by the system using the active processing goals 38, 48,
and 49. Because the processing goals are non-deterministic, vary by
requested product and may be modified during the processing there
exist a large number of possible processing sequences. The
progression of processing applied to the collected image material
is applied in sequential steps. Each step implements a single
processing module that is intended to satisfy or partially satisfy
an active processing goal. At each step in the process any one of
the available processing modules (culling 62, grouping 64,
enhancing 66, annotating 68, associating 70, or composing 72) may
be executed. At the conclusion of the processing of a particular
step the active processing goals 38 that existed before the step
processing are updated to reflect the changes in goals that
resulted from the execution of the processing step 52. These
updated active processing goals 48 serve as the input processing
goals for the subsequent processing step 53. Each of the available
processing modules will be described. In addition to the processing
goals 38, 48, and 49, the processing modules at each processing
step have access to four sources of information: pixel data 32
contained in the individual images, metadata 34 attached to images
or groups of images, the original product or service order
description 36 and, a customer profile 30 containing information
about the customer. This process can iterate as many times as
necessary to complete the desired product objectives.
[0030] The processing of the collection of digital image material
is reviewed to remove unwanted images. This step is called culling.
Culling is performed on images that are deemed unwanted due to poor
quality, or if there are several images that are similar in
composition and only one is desired. To accomplish this culling
process involves the computation of several image based metrics
using the pixel data 32. These metrics fall into two basic
categories: technical quality measures and abstract feature
metrics. The technical quality measures would involve calculations
to assess the overall sharpness of the image, exposure quality,
grain quality of the image. Algorithms used to calculate technical
quality measures are common in the art, examples would include:
"Estimation of Noise in Images: An Evaluation" by S. I. Olson in,
55 (4), 1993, pp. 319-323 and "Refined filtering of image noise
using local statistics" by J. S. Lee in Computer Vision Graphics
and Image Processing, 15, 1981, pp. 380-389.
[0031] Images are then given an overall rating of technical quality
based on the technical quality metrics. The overall image quality
metric is then compared to a threshold associated with the
processing goal. Images whose quality rating falls below the
threshold are flagged as unwanted. The specific threshold applied
to a given image or set of images is parametrically determined
using the processing goal 38 and the customer profile 30 as inputs.
In addition to the determination of specific objective image
features, abstract subjective image features are also calculated.
These subjective measures of the image allow the module to cull
unwanted or unnecessary images. A typical example of images which
require culling occurs when several images in the group are
similarly composed and contain nearly identical content such as a
group photo which is repeated several times. In order to choose the
best of this group of images the digital image is analyzed to
determine a quality metric composed of both objective and
subjective quality features. The basis for this subjective
assessment would be the presence or absence of faces, the
identification of the faces in the image, the number of faces
present, if the eyes are open on each face, if the subject is
smiling, or orientation of the face to the camera. Examples of
algorithms used to determine subjective features of an image are
described in Proceedings of the IEEE Computer Society conference on
Computer Vision and Pattern Recognition, June 1997. These
subjective features provide clues about the subjective quality of
the image. The objective and subjective features calculated for
each image are combined and the images are ranked according to the
relative importance of each feature analyzed to the processing goal
38 and which matches the stored customer preferences 30. The
ranking, because it is partially based upon subjective information
also carries a probability weighting factor. The combination of the
ranking and the probability weighting factor is used to assert a
decision about the quality of the image. This quality assertion is
retained with the image as metadata 34 and is passed to the next
processing step. In the grouping step images are grouped according
to criteria derived from the analysis of the customer profile 30,
the active processing goals 38 and the requested product or service
36. The goal of grouping images is to associate images that are a
part of a common theme or story in the mind of the customer.
Typical groupings derived from customer profiles 30 would be to
associate images according to the preferred organization scheme of
the customer. The attributes of typical organization schemes would
include organizing by events such as birthday party, vacation,
holiday, graduation, organizing by time, and organizing by people.
Customer requested products 36 could also dictate grouping
parameters in accordance with the product design. Examples of these
grouping schemes include grouping by location in a theme park, by
common life-cycle event such as graduation, by image content type
such as outdoor, indoor, nature, person, group, etc. A variety of
software algorithms exist which are applicable to grouping. One
class of software algorithms operates on metadata 34 associated
with the image where the grouping process is similar to searching a
media database. Several commercial examples of media databases that
allow searching are Picture Network Incorporated, Publishers Depot
(www.publishersdepot.com). A second class of algorithms employed
for this grouping process would include image processing algorithms
which identify objects and feature sets within an image by way of
the pixel information 32. These objects could represent typical
cultural icons such as birthday cakes, Christmas trees, graduation
caps, wedding dresses, etc. An example of such algorithms is
reported by J. Edwards and H. Murase, "Appearance Matching of
Occluded Objects Using Course-to-fine Adaptive Masks" in
Proceedings of the IEEE Computer Society Conference on Computer
Vision and Pattern Recognition, June 1997, pp. 533-546. The Product
Order Information 36 can also be used in this grouping by simply
stating the event or location or person on which to base the final
product. The Processing Goals 38 can also be used as a grouping
tool where the specific products offered mandate a specific
grouping.
[0032] The annotation step seeks to generate annotation that can be
composed on the story product or photocollage. Annotation is
designed to provide context information about the image or group of
images and assist the customer in communicating the story
surrounding the images. A very common form of annotation is text
associated with each image and with a group of images which
explains the "who, what, when, where, and why". Such context
information is generally derived from metadata 34 generated in
previous processing steps, user profiles 30 or, image understanding
algorithms applied in the annotation module. "Who" information may
be determined using face recognition algorithms applied to the
pixel data 32 tuned by the training data contained in the customer
profile 30. Using a commercial product such as "Facelt" by
Visionics, a database of known faces can be retrieved from the
customer profile 30 and used to guide the face recognition
software. "What" information can be asserted by correlating date
and time of the image capture available in the image metadata 34
with key dates contained in the customer profile 30. In addition
more "what" data can also be asserted by looking within the image
32 for specific cultural icons, derived from object recognition
algorithms, such as birthday cakes, Christmas trees or, graduation
robes. "When" information for annotation is easily determined using
metadata 34 such as date and time recorded at the time of capture.
"Where" information may be provided from the capture device
integrated with GPS (Global Positioning System) technology and then
added to the metadata 34 for the image or it can be guessed at
using image understanding algorithms which correlate known location
scenes with the image content from available images 32. "Why"
information is very difficult to determine without some input from
the image owner via the product order information 36. The
annotation, once determined, can be rendered as text, graphics,
images, video or sound and associated with the individual images or
with the group of images in either a traditional photocollage or
digital product.
[0033] The image enhancement module applies image processing to
improve the image for the intended story based purpose. Two
categories of image enhancement are considered for each image.
Basic image enhancements are applied to all images, as necessary,
would include: red-eye removal, brightness and contrast adjustment,
color adjustment, and crop and zoom to improve composition. Image
enhancements are applied directly to the pixel data 32 and recorded
in the image metadata 34 from the images. A second category of
image enhancement is applied to the images in order to enhance
their use within a story context to communicate the emotion or to
support the story theme as described in the processing goal 38. For
instance, selected images from a wedding story would be softened
using a blur algorithm or the colors of an image could be made to
match those found in a comic strip. The application of this type of
enhancement processing is either specified in the product
description 36 generated from the customer profile 30. All of these
operations are available in image editing software programs such as
Adobe PhotoShop, with the exception of red-eye removal and red-eye
removal can be accomplished via the Kodak Imaging Workstation and
other software such as Picture-It from Microsoft.
[0034] Some product specifications will include a processing goal
to associate external content that is relevant to the product and
the image content. This associated content provides additional
context and interest in the final rendered product. External
content takes several forms including weather reports, newspaper
headlines, stock photographs, advertisements, historical
references, travel brochure copy, popular music clips, video
segments, etc. In order to locate and find appropriate content for
the image story/photocollage the metadata 34 attached to each image
and image group and is interrogated to derive searchable topics or
specific information located in the customer profile 30. These
search topics would focus on the when, where, what aspects of the
image story. Once a series of potential search topics have been
assembled they are formatted into queries and searches are
performed on a variety of databases. The results of these content
queries is then refined by applying priority rules from the product
description database and customer preferences stored in the
customer profile 30. These types of searches are common in
information databases such as Yahoo.RTM. on the internet
(www.yahoo.com) or from Dialog corp. If there are items within the
image that can be identified with specific events (such as a
wedding or birthday) or if a person can be identified from imaging
algorithms, the Pixel Information 32 is used to determining the
types of additional content that are added. The Processing Goals 38
dictate the forms and types of associated content that is added.
The Product Order Information 36 is also a source of information
regarding the association of content by a special request on the
form (such as a theme park or wedding motif).
[0035] The layout processing module places the image data,
annotations, and associated content into an output ready format.
This layout step must be repeated for each product that was
requested by the customer and for each layout goal that was
generated during the processing steps. The main task involved in
the layout of a multi-page photocollage is the determination of
which images are to be placed in specific locations on specific
pages. In addition the external content which has been associated
with the individual images or groups of images must be composited
onto the page. The layout process may be accomplished in numerous
ways. One means of performing this processing step is to employ a
parametric photocollage description. This description specifies the
general structure of the photocollage but is not sufficient, of
itself, to specify a final photocollage product. The description
file includes photocollage features such as number of sections,
size and format of individual pages, maximum and minimum number of
pages allowable, maximum and minimum number of customer images
allowable per page, section descriptions, cover designs, location
of allowable stock content, etc. By employing a photocollage
description file a variety of tailored photocollage products may be
designed incorporating features specific to the design goals. The
layout step begins by reading the photocollage description file to
initialize a layout algorithm. The layout algorithm then works page
by page and section by section to apply specific layout rules
guided by the values in the photocollage description file. A
variety of page/section layout algorithms could be employed which
embody different design philosophies for photocollage products. The
final photocollage product can be rendered on a variety of output
media types which would include paper, fabric, as a digital file or
on any one of a number of digital media forms such as a CD. The
Processing Goals 38 are used to determine the capabilities of the
specific devices being used. Customer Profiles 30 are used to
determine color preferences, layout preferences, and design
considerations. Metadata 34 is used to determine the placement of
associated content with specific images. In a preferred embodiment,
the first cut version of the photocollage is rendered as a scaled
replica of the final version. This scaled replica is also
identified using a template ID in the form of a bar code.
[0036] FIG. 4 shows the system diagram for Story Preparation
Services. Input to the system comes from Personal Computers 90
(using albuming software such as Microsoft Picture It, Family Base
from Micro Dynamics or a myriad of others), an interactive kiosk,
via the phone or over-the-counter order forms 84, a retail outlet
82 with links to the system, or digitization services 80
specializing in converting analog material into digital
information. The information needed to perform the story services
is communicated via traditional means including mail, phone, modem,
Internet or other on-line services. The components required by such
as system include a digitization system for pictures, video and
audio 104, a storage facility 102, an operator interface 96,
algorithms and software 98 for the analysis of the data and to
provide the necessary steps to complete the product, and an output
delivery system 100 for printing or other media.
[0037] FIG. 5 is a flow chart showing the steps performed by the
present invention towards preparation of a photocollage that allows
alterations after automatic creation. The automatic creation of a
first cut version of a photocollage proceeds as described above.
After the first cut version is complete, the album is reviewed by
the customer to determine if any changes are desired. In a
preferred embodiment the user is also supplied with a set of scaled
image stickers and templates to be used to modify the first cut
version of the photocollage. The sticker images are generated at
the time of the original photocollage processing as described
below. These stickers and templates are illustrated and described
in FIGS. 7, 8 and 9 and the discussion related, thereto. A customer
wishing to construct additional pages for a photocollage or to
modify the page layout of the first cut version selects one or more
images by peeling the sticker image from the sticker sheet 118 and
placing it on the page layout form 160 as shown in FIG. 9. The page
layout form 160 includes one or more scaled graphic depictions of
the final page each comprising a unique page layout. In one
embodiment each scaled page representation includes one or more
graphic image boxes each of which is an allowable location to place
a sticker. Different scale page layout representations depict the
available album page layouts made available by the service.
Normally this variety of page layout choices would include variety
along the dimensions of: the number of images per page, the
resulting size of the images on the final printed page,
arrangements accommodating both portrait (vertical) and landscape
(horizontal) captured original images, different final printed page
sizes, provision for annotation text, different size original
images including standard 35 mm, Advanced Photo System classic,
HDTV and Panoramic images and, Digital Still Camera formats. The
scale of the sticker images and the scale of the graphic image
boxes on the page layout order form do not have to be the same. If
the page layout form image box is larger than the sticker image the
image is enlarged to fill the box when producing the final printed
album page. If the page layout form image box is smaller than the
sticker image the image is reduced to fill the box when producing
the final printed album page. In this way a variety of final output
sized images can be made available to the user without the need to
produce multiple sizes of sticker images. The page layout sheet and
cover sheet would have identifiers 164 on them so that the page
type can be identified during the scanning process. Bar code or
alphanumeric characters can be used for this identification.
[0038] Other means to alter the placement of images in the
photocollage can also be used including the use of a marking means
using an "X" over the image to indicate deletion and a circle
around the image and an arrow showing where the image is being
moved on the photocollage. Photocollage templates can also be
modified by other means such as the selection via ID numbers
representing layouts in a catalogue showing the layout choices.
After alteration, the modified photocollage is brought or sent to a
location, such as a retailer, to be scanned and printed in final
form.
[0039] FIG. 6 is a flow chart showing the steps performed by the
present invention; towards preparation of a photocollage that is in
a digital file format that can be sent across the internet. The
process is similar to that shown in FIG. 5 except that all steps
are performed on a digital computer with images being transmitted
as digital files. It is envisioned that any alterations, both for
changing the images within a page and changing the templates
describing the layout of the pages, are performed by employing
traditional software methodologies like drag-and-drop or choosing
from a list for replacement of images and templates. In this
digital case, the information needed to convey the changes can be
sent electronically to the retail site for printing of the final
photocollage.
[0040] Scaled image stickers used by the consumer to correct the
first cut version of the photocollage are generated by using the
original high-resolution images archived in processing facility 14
as seen in FIG. 2. This archive of digital images would, in most
cases, comprise a software database application and a large
collection of optical or magnetic storage media. In order to allow
for the future association of the high-resolution images with their
corresponding reduced resolution thumbnail images, each image is
assigned a unique identifier. There are a variety of means, known
in the art, for the generation of this unique identification. One
such means is disclosed in Townsend and Shaffer, U.S. Pat. No.
5,784,461, in which a combination of the customer identification
information and the photofinishing location, equipment, and date
and time are combined. Other means of unique identification could
be employed so long as they provide a one-to-one association of a
person to a single image. The assigned unique image identification
is associated with both the high and reduced resolution versions of
the image. The unique identifier embodiment can be any of either:
embedded data contained within the pixels themselves, invisible
ink, visible codes, histogram, color information, or the like. The
reduced resolution images are further processed to generate a
formatted array or sticker sheet 118 as seen in FIG. 8.
[0041] Referring to FIG. 8, the sticker sheet 118 also includes an
alpha numeric human readable image identifier 119 which allows the
user to locate the corresponding photographic print of the sticker
sheet image. In addition a unique machine-readable image
identification is also included in the printed sticker sheet. The
machine readable image identification could take the form of a bar
code, machine readable characters or any one of a number of
data-encoding patterns. A preferred implementation this unique
machine readable image identification is to embed it directly into
the pixels of the thumbnail image using signal processing
techniques. An example of such techniques can be seen in Daly et
al. (U.S. patent application Ser. No. 08/565,804).
[0042] The method of embedding digital data in an image can be best
considered in two stages; an encoding process and a decoding
process. First, a multi-level data image is generated from digital
data. The multi-level data image as described in detail below is an
image having a constant background value and an array of spots on
the background representing the digital data. The data image may be
produced using digital image processing techniques, or may be
produced optically for example by exposing spots on a photographic
film. Next, the data image is convolved with an encoding carrier
image to form a frequency dispersed data image. The encoding
carrier image is preferably a rotationally symmetric, low
amplitude, high frequency pattern employed to spatially disperse
the data image to mask its visibility when added to the source
image. Preferably the convolution is performed on a digital
computer using a well known Fourier Transform method and digital
versions of the data image and the encoding carrier image. The
convolution may also be performed using a direct convolution
algorithm on the digital computer. Alternatively, the convolution
may be performed optically using well known optical convolution
techniques and optical versions of the data image and the encoding
carrier image. The frequency dispersed data image is then added to
the source image to form a source image with embedded data. As
described in more detail below, the addition may be performed
either optically using traditional photographic processes, or
digitally using a digital computer and digital versions of the
source image and the frequency dispersed data image. If the
addition is performed digitally, a hard copy version of the digital
source image having embedded data may be produced using a digital
printer such as a thermal, ink-jet, electrophotographic, or silver
halide printer.
[0043] Once the generation of the final photocollage from the
modified first cut version of the photocollage has been accepted or
modified by the user the modified version is rendered into a final
photocollage. These modified photocollage orders are scanned to
generate a final version processing order. If the modified
photocollage contains image stickers the photocollage processing
identifies the original high resolution data by recovering the
embedded digital identification data. The digital identification
data is recovered from the source image having embedded data by
first cross correlating the source image having embedded data with
a decoding carrier image to produce a recovered data image.
Preferably, the decoding carrier image is identical to the encoding
carrier image and the cross correlation is performed by a digital
computer on digital versions of the source image having embedded
data and the decoding carrier image. If a hard copy of the source
image having embedded data is used, the hard copy is first scanned
as described in detail below to produce a digital version.
Alternatively, the cross correlation may be performed using well
known optical techniques and optical versions of the source image
having embedded data and the decoding carrier image. Finally, the
digital data is extracted from the recovered data image.
Preferably, the digital data is extracted from a digital version of
the recovered data image using a pattern recognition process in a
digital computer. Alternative methods for encoding data within an
image can be found in issued U.S. Pat. Nos.: 5,636,292; 5,128,525;
5,285,438; and 5,126,779 which can also be used with the present
invention.
[0044] Referring again to FIG. 2, after scanning the modified first
version photocollage and template forms the processing facility
uses the embedded image identification to generate the list of
images needed to generate the final photocollage. Using the list of
images the processing facility 14 finds and retrieves the
corresponding high resolution images that were previously stored in
the image storage 24 which are associated with the unique image
identification numbers that were supplied by the scanned layout
sheets 160. These high-resolution images are then assembled into a
photocollage using a page description language or template
following graphical rules. These graphical rules incorporate
customer preferences such as size of the pages, background styles
and colors, etc. In this page layout process it may be necessary to
zoom and crop the images to optimize the layout or to match
customer preferences.
[0045] FIG. 7 illustrates the process steps that are performed from
developing photographs to generation of photocollages by the
processing facility of FIG. 2. The customer brings in exposed film
142 or a memory card from a digital camera 143 containing images
that are to be made into a photocollage. The photofinisher
generates traditional prints 146 for the customer. In the case of
film, the film is scanned 145 to create a digital image. Once
acquired, the digital images are assigned a unique ID 149 and
placed into digital storage 150. A sticker sheet 118 and a first
cut version of a photocollage is generated 148 and sent to customer
for review and modification. The sticker sheet 118, first cut
version of the photocollage and the layout pages 160 can be sent to
the customer along with their prints. If modifications to the first
cut photocollage are desired the customer selects images from the
sticker sheet 118 places them on the first cut photocollage or on
new page layout sheets or cover sheets 160 as they wish 151. They
would also select the type of product 153 from a standard order
form supplied with sticker sheet 118. The customer then returns the
modified first cut photocollage, new cover and layout sheets 160 to
the retailer 155 where the pages are scanned with the cover and
layout sheet scanner 126 to create a photocollage order. The
photocollage order is processed to determine which images are to be
used in the photocollage, the cover, the layout and the product
type 157. The unique identifiers from the reduced resolution images
are then associated with the images stored in the online storage
facility and these full scale images are retrieved 152. The
photocollage is then constructed and printed 154 and sent to the
customer 156.
[0046] Depending upon the method employed for rendering the image
identification number within the sticker sheet a variety of image
processing techniques would be employed. In the case of human
readable image identification numbers optical character recognition
software is employed that translates the scanned image of the
characters to a digital number representation. In the case where
the image identification was hidden within the pixels of the
reduced resolution image, a reverse of the data embedding signal
processing technique is employed. The images are processed to
remove any rotation and scale variations introduced in the printing
and scanning steps. The result of this processing is to generate a
list of the identification numbers of the images which the customer
desires to have placed into the photocollage.
[0047] The invention has been described with reference to a
preferred embodiment However, it will be appreciated that
variations and modifications can be effected by a person of
ordinary skill in the art without departing from the scope of the
invention.
PARTS LIST
[0048] 6 photographic prints
[0049] 8 camera media
[0050] 10 cartridges
[0051] 11 keepsakes
[0052] 12 memory cards
[0053] 14 processing facility
[0054] 15 CDs
[0055] 17 albums
[0056] 20 data store
[0057] 22 digital processing
[0058] 24 processing bag
[0059] 28 kiosk
[0060] 32 pixel data
[0061] 34 meta-data
[0062] 36 order description
[0063] 37 goal generation
[0064] 38 processing goals
[0065] 40 products
[0066] 48 processing goals
[0067] 49 processing goals
[0068] 62 culling
[0069] 64 grouping
[0070] 66 enhancing
[0071] 68 annotating
[0072] 70 associating
[0073] 72 composing
[0074] 118 sticker sheets
[0075] 119 numeric identifier on stickers
[0076] 142 exposed film
[0077] 144 film processed
[0078] 145 film digitization
[0079] 146 prints returned
[0080] 148 materials for photocollages returned to customer
[0081] 149 unique ID
[0082] 150 digital storage
[0083] 151 image selection and placement
[0084] 152 high resolution image retrieval
[0085] 153 product decision
[0086] 154 photocollage printing
[0087] 155 return of materials
[0088] 156 finished product sent out
[0089] 157 layouts scanned
[0090] 160 page layout sheet
[0091] 162 cover layout sheet
[0092] 164 identifier for page layout sheet or cover sheet
* * * * *