U.S. patent application number 12/558227 was filed with the patent office on 2010-09-16 for digital camera with built-in lens calibration table.
This patent application is currently assigned to FOTONATION IRELAND LIMITED. Invention is credited to Petronel Bigioi, Peter Corcoran, Yury Prilutsky, Eran Steinberg, Adrian Zamfir.
Application Number | 20100231727 12/558227 |
Document ID | / |
Family ID | 34966489 |
Filed Date | 2010-09-16 |
United States Patent
Application |
20100231727 |
Kind Code |
A1 |
Steinberg; Eran ; et
al. |
September 16, 2010 |
DIGITAL CAMERA WITH BUILT-IN LENS CALIBRATION TABLE
Abstract
A digital camera that automatically corrects dust artifact
regions within acquired images by compiling a dust map includes an
optical system for acquiring an image with a corresponding dust
calibration table for such optical system, including a lens
assembly and an aperture stop, in which the corresponding dust
calibration map can reside. A transformation between the dust map
and the specific lens calibration table, enables the use for a
single dust map in multiple instances of lenses and focal length,
without the need to recalibrate the digital camera for each
instance.
Inventors: |
Steinberg; Eran; (San
Francisco, CA) ; Prilutsky; Yury; (San Mateo, CA)
; Corcoran; Peter; (Galway, IE) ; Zamfir;
Adrian; (Bucuresti, RO) ; Bigioi; Petronel;
(Galway, IE) |
Correspondence
Address: |
Tessera/FotoNation;Patent Legal Dept.
3025 Orchard Parkway
San Jose
CA
95134
US
|
Assignee: |
FOTONATION IRELAND LIMITED
Galway
IE
|
Family ID: |
34966489 |
Appl. No.: |
12/558227 |
Filed: |
September 11, 2009 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
10842244 |
May 10, 2004 |
7590305 |
|
|
12558227 |
|
|
|
|
10676820 |
Sep 30, 2003 |
7676110 |
|
|
10842244 |
|
|
|
|
Current U.S.
Class: |
348/207.2 ;
348/241; 348/360; 348/E5.024; 348/E5.078 |
Current CPC
Class: |
H04N 5/2171 20130101;
H04N 5/23209 20130101; H04N 1/4097 20130101; H04N 5/217
20130101 |
Class at
Publication: |
348/207.2 ;
348/241; 348/360; 348/E05.024; 348/E05.078 |
International
Class: |
H04N 5/217 20060101
H04N005/217; H04N 5/225 20060101 H04N005/225 |
Claims
1. A digital camera that automatically corrects dust artifact
regions within acquired images, comprising: (a) a main camera body
and an optical system for acquiring an image including a removable
lens subsystem, wherein the optical system comprises: (i) a lens
assembly, (ii) a diaphram or other aperture adjustment unit, (iii)
an electronic control processing subsystem, and (iv) a
communications interface between said lens subsystem and an
electronic processing subsystem within the main camera body; (b) an
electronic sensor array for collecting image data with multiple
pixels that collectively correspond to the acquired image; (c) a
lens calibration table containing optical system data relating to
said removable lens subsystem; (d) a master dust map describing a
physical manifestation of one or more dust artifacts on said sensor
array; (e) an intermediate dust map determined to match a specific
lens, focal length, or aperture, or combinations thereof, and
calculated as a transformation of the master dust map based at
least in part on one or more parameters included in the lens
calibration table; (f) an electronic subsystem for digital
processing including a processor for converting processing the
digital data according to programming instructions; (g) a
communications interface between said electronic subsystem and said
removable lens subsystem, and (h) wherein the camera is configured
to correct for the dust artifacts on the sensor array in an
acquired image based on the intermediate dust map.
2. The camera of claim 1, wherein said lens calibration table
comprises processor-readable digital code embedded within a memory
component that is located inside the lens assembly.
3. The camera of claim 1, wherein said lens calibration table is
located in an external application.
4. The camera of claim 1, wherein said lens calibration table is
downloadable from a server.
5. The camera of claim 1, wherein said master dust map comprises a
statistical dust map.
6. The camera of claim 5, wherein additional images are
digitally-acquired with said digital camera, and said statistical
dust map is dynamically updated.
7. The camera of claim 6, wherein said statistical map comprises
probabilities based on comparisons with suspected equivalent dust
artifact regions within said additional images.
8. The camera of claim 7, wherein determining of said probabilities
is based on a pixel analysis of the suspected dust artifact regions
in view of predetermined characteristics indicative of the presence
of a dust artifact region.
9. The camera of claim 5, wherein whether said additional
digitally-acquired images have non-contradicting data that said
probability that certain pixels correspond to dust artifact regions
within said images is validated prior to correcting pixels
corresponding to correlated dust artifact regions within said
images based on the manifestation of said master statistical dust
map.
10. The camera of claim 1, wherein said dust artifacts are
corrected on a image from raw format.
11. The camera of claim 1, wherein said dust artifacts are
corrected on a processed image after being converted from raw
format to a known red, green, blue representation.
12. The camera of claim 1, wherein said dust artifacts are
corrected by replacing said pixels within said one or more
digitally-acquired images with new pixels.
13. The camera of claim 1, wherein said dust artifacts are
corrected by enhancing values of pixels within said one or more
digitally-acquired images.
14. The camera of claim 1, wherein the parameters included in the
lens calibration table comprise exit pupil dimension of the lens
assembly, or distance of dust from a surface of the electronic
sensor array that corresponds to a focal plane of the lens
assembly, or both.
15. A method of automatically correcting dust artifact regions
within acquired images, comprising: (a) acquiring a digital image
with a digital camera that includes an optical system including a
removable lens subsystem, wherein the optical system comprises: (i)
a lens assembly, (ii) a diaphram or other aperture adjustment unit,
(iii) an electronic control processing subsystem, and (iv) a
communications interface between said lens subsystem and an
electronic processing subsystem within the main camera body; and
wherein the acquiring comprises collecting image data with an
electronic image sensor that includes multiple pixels that
collectively correspond to the acquired image; (b) reading a lens
calibration table containing optical system data relating to said
removable lens subsystem; (c) calibrating a dust correction
component with a master dust map describing a physical
manifestation of one or more dust artifacts on said sensor array or
one or more components of said optical system, or both; and wherein
the calibrating comprises matching with an intermediate dust map a
specific lens, focal length, or aperture, or combinations thereof,
and including calculating the intermediate dust map as a
transformation of the master dust map based at least in part on one
or more parameters included in the lens calibration table; (d)
digitally processing the digital image including converting
processing the digital data according to programming instructions;
(e) providing communications interfacing between said electronic
subsystem and said removable lens subsystem, and (f) correcting for
the dust artifacts on the sensor array or the one or more
components of the optical system, or both, in the acquired digital
image based on the intermediate dust map.
16. The method of claim 15, further comprising downloading the lens
calibration table from a server.
17. The method of claim 15, further comprising acquiring one or
more further digital images with said digital camera and
dynamically updating the master dust map based in-part on an
analysis thereof.
18. The method of claim 15, further comprising converting one or
more pixels corresponding to one or more dust artifacts from raw
format to a known red, green, blue representation.
19. The method of claim 15, wherein the correcting comprises
replacing pixels within said one or more digitally-acquired images
with new pixels.
20. The method of claim 15, wherein the correcting comprises
enhancing values of pixels within said one or more
digitally-acquired images.
21. One or more processor-readable media having code embodied
therein for programming a processor to perform a method of
automatically correcting dust artifact regions within acquired
digital images, wherein the method comprises: (a) acquiring a
digital image with a digital camera that includes an optical system
including a removable lens subsystem, wherein the optical system
comprises: (i) a lens assembly, (ii) a diaphram or other aperture
adjustment unit, (iii) an electronic control processing subsystem,
and (iv) a communications interface between said lens subsystem and
an electronic processing subsystem within the main camera body; and
wherein the acquiring comprises collecting image data with an
electronic image sensor that includes multiple pixels that
collectively correspond to the acquired image; (b) reading a lens
calibration table containing optical system data relating to said
removable lens subsystem; (c) calibrating a dust correction
component with a master dust map describing a physical
manifestation of one or more dust artifacts on said sensor array or
one or more components of said optical system, or both; and wherein
the calibrating comprises matching with an intermediate dust map a
specific lens, focal length, or aperture, or combinations thereof,
and including calculating the intermediate dust map as a
transformation of the master dust map based at least in part on one
or more parameters included in the lens calibration table; (d)
digitally processing the digital image including converting
processing the digital data according to programming instructions;
(e) providing communications interfacing between said electronic
subsystem and said removable lens subsystem, and (f) correcting for
the dust artifacts on the sensor array or the one or more
components of the optical system, or both, in the acquired digital
image based on the intermediate dust map.
22. The one or more processor-readable media of claim 21, wherein
the method further comprises downloading the lens calibration table
from a server.
23. The one or more processor-readable media of claim 21, wherein
the method further comprises acquiring one or more further digital
images with said digital camera and dynamically updating the master
dust map based in-part on an analysis thereof.
24. The one or more processor-readable media of claim 21, wherein
the method further comprises converting one or more pixels
corresponding to one or more dust artifacts from raw format to a
known red, green, blue representation.
25. The one or more processor-readable media of claim 21, wherein
the correcting comprises replacing pixels within said one or more
digitally-acquired images with new pixels.
26. The one or more processor-readable media of claim 21, wherein
the correcting comprises enhancing values of pixels within said one
or more digitally-acquired images.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a divisional of application Ser. No.
10/842,244, filed on May 10, 2004, entitled, "Digital Camera With
Built-In Lens Calibration Table," which is a continuation in part
of application Ser. No. 10/676,820, filed on Sep. 30, 2003,
entitled, "Determination Of Need To Service A Camera Based On
Detection of Blemishes in Digital Images", which is related to a
family of patent applications filed on the same day, including U.S.
application Ser. No. 10/676,823, entitled, "Automated Statistical
Self-Calibrating Detection and Removal of Blemishes in Digital
Images Based On Multiple Occurrences Of Dust In Images"; U.S.
application Ser. No. 10/677,134, entitled, "Automated Statistical
Self-Calibrating Detection and Removal of Blemishes in Digital
Images Based on a Dust Map Developed From Actual Image Data"; U.S.
application Ser. No. 10/677,139, entitled, "Automated Statistical
Self-Calibrating Detection and Removal of Blemishes in Digital
Images Dependent Upon Changes in Extracted Parameter Values"; U.S.
application Ser. No. 10/677,140, entitled, "Automated Statistical
Self-Calibrating Detection and Removal of Blemishes in Digital
Images Based on Determining Probabilities Based On Image Analysis
Of Single Images"; U.S. application Ser. No. 10/676,845, entitled,
"Method Of Detecting and Correcting Dust in Digital Images Based On
Aura And Shadow Region Analysis"; U.S. application Ser. No.
10/676,716, entitled, "Digital Camera"; and U.S. application Ser.
No. 10/676,835, entitled, "Digital Image Acquisition And Processing
System", which are all hereby incorporated by reference.
BACKGROUND
[0002] 1. Field of the Invention
[0003] This invention related to digital photography and in
particular, automated means of in-camera removal of blemish
artifacts from images captured and digitized in a digital
process.
[0004] 2. Description of the Related Art
[0005] Many problems are caused by dust in particular and blemishes
in general on imaging devices in general and digital imaging
devices in particular. In the past, two distinct forms of image
processing included providing means to detect and locate dust,
scratches or similar defects and providing means to remedy the
distortions caused by the defects to an image. It is desired to
have an advantageous system that combines these functions, and can
automatically detect and correct for the effects of dust, scratches
and other optical blemishes.
[0006] Image correction has been studied in relation to display
devices, output apparatuses such as printers, and digital sensors.
Image correction of dust artifacts can be used to recreate missing
data, also referred to as in-painting or restoration, or undoing
degradation of data, which still remains in the image, also
referred to as image enhancement. It is desired to have a system
including a digital camera and an external device or apparatus that
can facilitate a defect detection and/or correction process
involving sophisticated and automated computerized programming
techniques.
[0007] The manifestation of the dust in a digital image is a
function of several optical parameters representing the dust. It is
desired to create a system that can automatically take into account
changes in such parameters without the need to manually recalibrate
the camera system.
SUMMARY OF THE INVENTION
[0008] A digital camera is provided that automatically corrects
dust artifact regions within acquired images by compiling a
statistical dust map from multiple images acquired under different
image acquisition conditions. An optical system that acquires an
image includes a lens assembly and an aperture stop. The optical
system may include the lens specific calibration information in the
lens. An electronic sensor array is disposed approximately at an
image focal plane of the optical system for collecting image data
according to spectral information associated with multiple pixels
that collectively correspond to the image. Digital processing
electronics include a processor for converting the image data to
digital data and processing the digital data according to
programming instructions. A memory has programming instructions
stored therein for performing a method of automatic image
correction of dust defect regions.
[0009] The specific dust map can be converted between different
lenses and between different focal lens setting of the same lens,
using a numerical formulae. By doing so, only a single dust map
needs to be maintained.
[0010] The further digitally-acquired images may include different
images than the originally acquired images. The method may further
include correcting pixels corresponding to correlated dust artifact
regions within each of the original images based on the associated
statistical dust map. The method may include correcting pixels
corresponding to correlated dust artifact regions within the
original images based on the associated statistical dust map. The
method may include digitally-acquiring additional images with the
digital camera, repeating the determining and associating, and
updating the statistical dust map including updating the mapped
dust regions based on the additional dust artifact determining and
associating.
[0011] The image correction method may be performed on a processed
image after being converted from raw format to a known red, green,
blue representation. The correcting may include replacing pixels
within the one or more digitally-acquired images with new pixels.
The correcting may include enhancing the values of pixels within
the one or more digitally-acquired images.
[0012] The dust artifact determining may include loading the
statistical dust map, loading extracted parameter information of a
present image, performing calculations within the statistical dust
map having extracted parameter variable-dependencies, and comparing
dust artifact detection data with the extracted parameter dependent
statistical dust map data. The dust artifact determining may also
include loading the statistical dust map, loading extracted
parameter information of a present image, loading extracted
parameters regarding the optical system, performing a calculation
for relating the statistical dust map with the present image
according to a selected value of an extracted parameter which is
otherwise uncorrelated between the present image and the dust map,
and comparing dust artifact detection data with the now correlated
statistical dust map data. The suspected dust artifact regions of
at least two images may include shadow regions and aura regions,
and wherein the method may include a first comparison of the shadow
regions and a second comparison of the aura regions.
[0013] The method may include digitally-acquiring further images
with the digital camera, repeating the determining and associating,
and updating the statistical dust map including updating the mapped
dust regions based on the further dust artifact determining and
associating. The determining may include determining probabilities
that certain pixels correspond to dust artifact regions within the
acquired images based at least in part on a comparison of suspected
dust artifact regions within two or more digitally-acquired images,
or on a pixel analysis of the suspected dust artifact regions in
view of predetermined characteristics indicative of the presence of
a dust artifact region, or both. The determining may be based at
least in part on a comparison of suspected dust artifact regions
within two or more digitally-acquired images.
[0014] The correcting may further include replacing the pixels
within the one or more digitally-acquired images with new pixels.
The correcting may include enhancing the values of pixels within
the one or more digitally-acquired images.
BRIEF DESCRIPTION OF THE DRAWINGS
[0015] FIG. 1 illustrates a main workflow of a dust removal process
in accordance with a preferred embodiment.
[0016] FIG. 2 illustrates the creation of a dust map.
[0017] FIG. 3 outlines a correlation of a dust map to image
shooting parameters.
[0018] FIG. 4 illustrates a procedure for detecting and removing
dust from images in accordance with a preferred embodiment.
[0019] FIG. 5 illustrates an adjustment of the dust map based on
aperture.
[0020] FIG. 6 illustrates an adjustment of a dust map based on
focal length.
[0021] FIG. 7 illustrates the plot describing a the dust movement
as a function of a lookup table for a hypothetical map.
[0022] FIG. 8 describes the workflow of correcting and detecting
the dust images based on a lens lookup table and a dust map.
[0023] FIG. 9 schematically illustrates coupling components of a
lens assembly with a housing of a digital image acquisition
device.
BRIEF DESCRIPTION OF TABLES
[0024] Table 1 lists potential Extracted Lens Parameters.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS SOME
DEFINITIONS
[0025] Dust specs: The preferred embodiment takes advantage of the
fact that many images may have repetitive manifestation of the same
defects such as dust, dead-pixels, burnt pixels, scratches etc.
Collectively, for this specifications all possible defects of this
nature are referred to in this application as dust-specs or dust
defects. The effects of those dust specs on digital images are
referred to herein as dust artifacts.
[0026] Acquisition device: the acquisition device may be a
multi-functional electronic appliance wherein one of the major
functional capabilities of the appliance is that of a digital
camera. Examples can be include digital camera, a hand held
computer with an imaging sensor, a scanner, a hand-set phone, or
another digital device with built in optics capable of acquiring
images. Acquisition devices can also include film scanners with a
area-capture CCDs, as contrasted, e.g., with a line scan mechanism.
D-SLR: Digital Single Lens Reflex Camera. A digital camera where
the viewfinder is receiving the image from the same optical system
as the sensor does. Many D-SLR, as for SLR cameras have the
capability to interchange its lenses, this exposing the inner
regions of the camera to dust.
[0027] A few parameters are defined as part of the process:
N Number of images in a collection. HIdp Number of images for
occurrence to be high dust probability HSdp Number of recurring
specs for label a region to be high dust probability p(hsdp)
Probability threshold for high confidence of a dust spec. N dp
Number of images to determine that a region is not dust p(ndp)
Probability threshold to determine that a region is not a dust
region. Most likely Hidp<=HSdp I--is a generic image I(x,y)
pixel in location x horizontal, y vertical of Image I pM is a
continuous tone, or a statistical representation of a dust map dM
is a binary dust map created form some thresholding of pM.
Mathematical Modeling of the Optical System
[0028] Prior to understanding the preferred embodiments described
herein, it is helpful to understand the mathematical modeling of
the camera optical system. This model is defined in application
Ser. No. 10/676,820 filed on Sep. 30, 2003, which is incorporated
by reference. With this modeling, the preferred embodiments may
advantageously utilize a single dust map for dust detection and/or
correction techniques rather than creating a dust map for each
instance of dust artifact having its own shape and opacity
depending on extracted parameters relating to the imaging
acquisition process. With an ability to model the optical system
and its variability, a single map may suffice for each lens or for
multiple lenses, and for multiple focal lengths, multiple
apertures, and/or other extracted parameters as described in more
detail below. This information, which is typical to a lens and a
camera combination may be stored in an external application, in the
camera processing memory and or inside the lens.
[0029] The main workflow of detecting and removing the dust from an
image is illustrated in FIG. 1. The preferred embodiment is
activated in four different cases. In general, this preferred
embodiment works for removing dust form a collection of images
having the same acquisition device. Specifically, a user may
acquire a picture on her digital camera (as illustrated in Block
101). Alternatively (102), a user may open a single image on a
external device such as a personal computer, open (103) a folder of
images on an external device or open a collection of images on a
digital printing device (104).
[0030] The preferred embodiment then extracts the shooting
parameters (120). Such parameters include, but not limited to data
about: the generic device parameters such as Camera parameters
(122) such as Camera name, Model, and conversion data specific for
the camera, Lens parameters (124) such as Lens brand, lens type,
lens focal length, and lens calibration tables; as well as
parameters specific to the image (126) such as focal length at
acquisition, aperture range, aperture at acquisition.
[0031] In addition, some parameter, in particular on the lens (126)
and the camera (124) may be also stored in the device, which is the
acquisition device such as the digital camera or the processing
device such as the personal computer or digital printer. Such
information, which may include parameters such as exit pupil, exit
pupil distance regarding the lens, or distance of dust to sensor
for the camera. Currently lenses store some data in the lens which
can communicate this information to the camera processor. In one
embodiment of this invention the specific lens calibration tables
can may be stored as part of the lens assembly. By doing so, the
introduction of new lenses to the market may automatically be
supported with the shipping lenses.
[0032] A table with such data may look like:
TABLE-US-00001 TABLE 1 Extracted Lens Parameters Field Example of
data Category Lens Nikon lens Manufacturer Lens Type AF 24 mm-45 mm
f2.8-f3.5 lens Focal Length 38 mm Acquisition data Aperture f-16
Acquisition data Dust distance 0.156 mm Camera data Exit pupil 19
mm Lens data Exit pupil 230 mm Lens data distance
Alternatively, the information may be related to the resulting dust
shift, as opposed to the theoretical parameters such as an exit
pupil, of which the shift is calculated. Such can be in a form of a
lookup table or a analytical formulae.
[0033] FIG. 7 describes the graph created by such an analytical
formulae or a lookup table describing the dust movement as a
function of the focal length for a specific hypothetical lens. This
plot describes a 70-210 mm zoom lens (1000). The interpolated
values based on measured 7 points 1010. The X-axis, 1020 defines
the focal length in mm of the lens. The plot describes a dust spot
in a distance of roughly 960-970 away from the optical center of
the lens, as illustrated in the Y-axis, 1030. As can be seen, the
dust shifts as a function of the focal length. This extrapolated
empirical result, 1050, corroborates the analytical explanation
provided in FIG. 6. It is important to note that the plot may and
is different based on the actual lens configuration and is neither
predictable by the mere focal length nor is monotonicity
guaranteed.
[0034] The dust map may also include meta-data that are different
the list of extracted parameters. Moreover, that which is described
as being extracted parameter dependent or encoded with extracted
parameter value data or based on a value of an extracted parameter
can be broadened to include other meta-data than just the extracted
parameters listed in Table 1. For example, certain meta-data are
dependent on parameters existing at the time of acquisition of the
image, and can be camera-specific or not. The amount of ambient
light available will depend on whether there is artificial lighting
nearby or whether it is a cloudy day. Discussion of meta-data as it
relates to image acquisition is found in more detail at U.S. patent
application Ser. No. 10/608,810, filed Jun. 26, 2003, and is hereby
incorporated by reference.
[0035] In the case that the system deals with multiple images (as
defined in 102, 103, and 104), the algorithm describes a loop
operation on all images (110). The first step is to open a dust map
(130). If non exists (132) the system will create a new dust map
(200) as further described in FIG. 2. In the case that the system
has a few dust maps (130) the software will try to correlate one of
the maps (300) to the image. This process is further illustrated in
FIG. 3. In particular this correlation refers to the adjustment of
the shooting conditions to some accepted standard. The acquisition
conditions in particular refer to the aperture and the focal
length. The correlation process is interchangeable and can be done
by adjusting the image to the map or adjusting the map to the
image. In some cases both acquired image and dust map should be
adjusted to some common ground. Such an example may happen when the
ma is calculated based on a aperture that the lens does not reach
or a different lens than the one used with different optical
configuration. Alternatively, such as in the case of a new lens,
this process (300) as called by 130, may be used to adjust the map
onto a new map and from that stage onwards continue with a single
map.
[0036] If no dust map corresponds with the image (140), a new dust
map is created (200). When a dust map does correspond (140) to the
image, the preferred embodiment checks if the dust specs as defined
in the dust map are of high enough confidence level to being dust
regions (150). The statistical decision as to the way such
confidence level is calculated for the dust map in general and for
the individual dust specs, is further discussed with reference to
FIG. 2 and FIG. 3. If the confidence level is low, the image is
added to the updating of the dust map (200). If after the image is
added, the confidence level is high enough (152) the software
continues to the dust removal process (160). Otherwise, the
software progresses to the next image (170).
[0037] For example, a dust map is considered valid after 10 images
are analyzed, and a dust spec is considered valid after a dust is
detected in 8 images. In this case, after analyzing 9 images, the
software may continue to the stage of updating the dust map (200)
but upon completion (10 images) there is a valid dust map, and the
software will continue to the correction (160) stage. If however
the loop (110) is only on its 1.sup.st to 8.sup.th image, no
correction will be done.
[0038] As an additional embodiment, images can be corrected
retroactively after the dust map reached high enough confidence.
This can be used for batch processing or off line processing or
processing where the information is gathered in parallel to the
needed correction. In this embodiment, when the confidence level is
not enough (152, NOT-YET) the images, a pointer to them, or a list
of them, or a representation of them, that were used to create the
dust map are stored in temporary location (154) and when the dust
map is ready (151-YES), the software will retroactively remove the
dust from all those images (156). In this fashion, all images,
including ones that originally did not hold sufficient statistical
information, may be corrected.
[0039] Referring to the dust detection and correction process
(160). This process may be executed as a loop on every dust spec in
the dust map, with a stage of detection and correction (400,
followed by 166 and 168). Alternatively, the process can be
implemented where all dust specs are detected first (162) and then
all dust specs are corrected (164). The decision as to the sequence
of operations varies based on implementation criteria such as what
regions of the image are in memory, and should not affect the
nature of the preferred embodiment. In the case where the lens
specific calibration table is in the lens, there is a need for a
intermediate step which converts the dust map via those tables. By
having this functionality, a single dust map may suffice to
accommodate changes in lenses and changes in the focal length of a
single zoom lens.
[0040] This workflow is illustrated in FIG. 8 which describes the
workflow of correcting and detecting the dust images based on a
lens lookup table and a dust map. If a dust map exists (1120) the
system will load the dust map (1130). Otherwise, a new dust map
needs to be created as described in FIG. 2 (200). The lens
calibration is then loaded (1140). Such map can exist in the
software, or as part of the lens memory, or in the camera. Together
with the dust map, a specific manifestation of the dust for the
specific camera, lens combination is created (1160). This is the
dust map that will be used to remove dust specs (400) from an image
(404) that was captured using the specific lens on the specific
camera.
In this scenario, if a user lads anew lens, 1180, there is no need
for a new calibration stage, but rather a computational step (1160)
is preferred to calculate the new Specific Dust Map.
[0041] Referring to FIG. 2 where the Dust map creation and updating
is defined:
this process can receive a collection of images as defined by FIG.
1 blocks 108 and 109) or one image at a time is refereed to this
process, as defined by FIG. 1 block 110. If the function is called
with a single image (220-SINGLE IMAGE) the image is directly
provided to the calculations (270). If multiple images are provided
(240 MULTIPLE IMAGES), then an initial step is to define if there
are more than enough images for defining the map. This step is
designed to optimize the creation process for dust in case of large
amount of images.
[0042] The sequence of the images that are to be referenced based
on the original collection of N images as defined in FIG. 1 blocks
102, 103 or 104. The sequence of images is based on a few criteria
such as: giving more weight to images shot last, and if images are
shot in a relatively small time frame, allocate the sequence is
large distances to try and assure minimal repetitiveness between
similar images that may have been taken of the same object with
little movement. The sequence will not be limited to the number of
images (HIDP) because it may well be that some regions will not
have enough data in them to evaluate the dust. This may happen in
cases where part of the image is very dark in some of the
images.
[0043] As an example: if N (number of images in a selection)=30;
and HSdp (number of images needed to determining map)=10; and all
images were shot in a space of an hour; then a potential sequence
may be:
[0044] 30, 27, 24, 21, 18, 15, 12, 9, 6, 3, 29, 26, 25 . . . , 2,
28, 25, . . . 1
[0045] Alternatively if the same 30 images were taken over a period
of a month it may be beneficial to select images sequentially (last
one shot is the first to be calculated):
[0046] 30, 29, 28, . . . 20, 19 . . . 2, 1
And in some cases this process of sampling the series (270) may
also decide not to use images that are taken too long from the last
image. In this case, for example if image 1-15 were taken in July
and 16-30 were taken in November, this process may limit the new
map to 16-30 or even create two different maps, one for images 1-15
and the other for 16-30.
[0047] In a different criteria, the parameters as extracted from
the images will determine the sequence and the number of dust maps
that are to be calculated. For example if a folder contains N=30
images, where 15 were taken with one camera and 15 with another,
the sampling step (270) may create two map sets.
[0048] Another criteria for creating a new set or checking for new
dust is the type of lens. If a lens is changed, it means that the
CCD-cavity was exposed to potential new dust. This may trigger a
new set of images to be looked at. It may also be an indication
that the camera was serviced, or that the photographer cleaned the
camera. Of course, if there is a parameter that defines when a
camera was serviced, this will trigger the creation of a new dust
map.
[0049] Those familiar in the art may be able to determine the right
combinations of creating this sequence based on the nature of the
dust, the camera and the lens. The next loop (270-271) defines the
marking of each region and the addition of the region to the dust
map if it is not already there. There are three type of regions in
an image, the first are images with sufficient information to
detect whether they are of dust nature. As an example, dark regions
surrounded by a light background. Other criteria may be regions
with a relatively small color saturation. The second group are
regions that are definitely non-dust. Such regions are for example
all clear, or of high color saturation. Other regions are
inconclusive such as a very dark segment of the image. In this
case, it will be hard to detect the dust even if it was part of the
image. Alternatively when looking for over exposed or "dead pixels"
the criteria may be reversed, if the pixels appear as a white spec
in the image.
[0050] The criteria may be also a function of the acquisition
parameter. For example an image with a open aperture may all be
marked as in-decisive, because the dust may not appear on the
image. Regions that are potentially dust are marked (292) and then
added to the dust mask (294). The addition may be the creation of a
new dust spec on the map or the modification of the probability
function or the confidence level counter for the region. Regions
that are most likely non-dust are marked (282) and then added to
the dust mask (284). The addition may be the creation of a new dust
spec on the map or the modification of the probability function or
the confidence level counter for the region. The additions of the
regions needs to be normalized to the shooting conditions as
defined by the dust map (300) if this step was not performed prior
to entering this function, as optionally defined in FIG. 1.
[0051] This loop continues over all regions of the image (271).
Alternatively (272), each region is compared (500) with the map
dust to se if there is no case where the monotonicity is broken,
i.e. a region that was of high probability to be dust is now non
dust.
[0052] FIG. 3 illustrates the process of correlating the image to a
default settings of the dust map. This process defines correlating
the image o the dust map, the dust map to a new dust map of the
dust map to the mage. In particular this correlation refers to the
adjustment of the shooting conditions to some accepted standard.
The acquisition conditions in particular refer to the aperture and
the focal length. The correlation process is interchangeable and
can be done by adjusting the image to the map or adjusting the map
to the image. In some cases both acquired image and dust map may be
adjusted to some common ground. Such an example may happen when the
ma is calculated based on a aperture that the lens does not reach
or a different lens than the one used with different optical
configuration. Alternatively, in case of a new lens, this process
(300) may be used to adjust the map onto a new map and from that
stage onwards continue with a single map.
[0053] To begin with, the dust map is being loaded (112) and the
default data on which the map was generated is extracted (310).
Such data may include the lens type, the aperture and the focal
length associated with the default state. In concurrence, the
information form the acquired image (304) is extracted (320) and
compared to the one of the dust map.
[0054] A explained in the mathematical model of the optical system,
the two main adjustments between the dust map and the image are
based on focal length, and on aperture, each creating a different
artifact that should be addressed. Knowledge of the phenomena may
assist in creating a better detection and correction of the dust
artifact. Alternatively, in a separate embodiment, analysis of the
image and the modification of the dust as changed by aperture and
focal length, may provide the necessary mathematical model that
describes transformation that defines the changes to the dust as a
function of change in the lens type, the focal length and the
aperture.
[0055] Referring to FIG. 3, after extracting the data, the
following step is modification of the map and or the image based no
focal length (900), and based on aperture (800). The following
steps are further defined with reference to FIG. 6 and FIG. 5,
respectively.
[0056] Following the two steps (800 and 900); the Image and the
Dust Map are considered to be correlated. The correlated map cM is
no longer binary because it defines both the shift and the fall off
which is continuous. FIG. 4 defines the preferred process of
detecting and removing the dust from the image. The input is the
image I is loaded, if it is not already in memory (404) and the
correlated dust map is cM is loaded (402) if already not in
memory.
[0057] The process of detecting and removing the dust is done per
dust spec. This process is highly parallelized and can be performed
as a single path over the image, or in strips. The flexibility of
performing the operation on the entire image, or in portions,
combined with the correlation or as a separate process, enables a
flexible implementation of the algorithm based on external
restrictions defined by the hardware, the run time environment,
memory restrictions and processing speed.
[0058] The acquisition information and the corresponding dust map
default setup are extracted in blocks 326 and 312 respectively.
Then, for each dust spec in the image 810, the size of the region
that is still obscured by the dust is calculated, as defined by
mathematical model. In some cases, when the aperture is very open,
this region may decline to 0. In others, where the aperture is
still very close, the size may be close to the size of the dust.
Alternatively, This step, 820, may be done as a preparation stage,
and kept in a database, which can be loaded.
[0059] The process then splits in two. The fully obscured region
will be marked in 834 pixel by pixel 832 in a loop 834, 835 and
will be treated by the in-painting process as defined in FIG. 4. A
semi opaque dust map, is created in the loop 840, 841 for each
pixel. Each of the pixels 842, is assigned an OPACITY value 844,
based on the mathematical model as described previously. The dust
spec that is only partially attenuated will go through a inverse
filtering of the information already there, as described in FIG. 4
block 430, with a specific embodiment in block 432. The process of
the inverse filtering may take into account the signal to noise
ratio to avoid enhancing data which is not part of the original
image. For example, the region around the dust may have a
over-shoot similar to a high pass filter, which may manifest itself
in the form of an aura around the dust. This aura should to be
taken into account before enhancing the regions.
[0060] FIG. 6 describes the adjustment of the Dust Map based on the
Focal length, and the specific lens. As described before, the shift
of the dust spec as a function of focal length for a specific lens
is a function of equation (13 the thickness of the window--tw,
which is constant for a given camera and exit pupil position--Pe,
which varies based on the lens system and the variable focal length
in case of a zoom lens. Given an image and a dust map, the
pertinent information is loaded, as described in FIG. 3, namely the
focal lens and lens type of the camera, 326, the focal length and
lens type in the dust map 312 and the camera distance of dust to
the sensor 318.
[0061] The process then goes through all knows dust specs in the
image 910 and calculates the shift of the dust. The coordinates of
the pixel are calculated from the center of the optical path, 922,
and the shift is calculated 924. Alternatively to going through
each pixel, in order to speed the process, only the periphery of
the dust spec can be calculated and the rest will be filled in.
Moreover, because the shift is a function of the location (x,y) of
the dust, in the case where dust is far enough from the origin, the
dust shape will not change. It is then sufficient to calculate only
a shift of a single point and displace the entire dust spec
accordingly. The regions, which are only partially attenuated due
to the change in aperture, may be calculated in this stage 940,
941, if calculated already as illustrated in FIG. 8, or
alternatively, the displacement can be calculated first as
explained in blocks 942,944.
[0062] In some cases, it is impossible to get the data on the exit
pupil, nor the distance the dust is from the sensor. Such cases may
be when the application has no a-priori knowledge of the camera or
the lens that was used.
[0063] FIG. 9 schematically illustrates coupling components of a
lens assembly with a housing of a digital image acquisition device,
such as a Digital Single Lens Reflex Camera, or D-SLR. The lens
assembly 1310 includes various optical, mechanical, electrical
and/or signal connector or coupling components for coupling with a
camera housing 1320 or other digital image acquisition device
housing or component. In FIG. 9, the components 1310 and 1320 are
illustrated as if the lens 1310 has been uncoupled from the camera
1320 and rotated 180.degree. about an axis running vertically
within the plane of the drawing between the components 1310 and
1320.
[0064] The physical or mechanical coupling between the lens
assembly 1310 and the camera body 1320 is preferably via a bayonet
type mount where the grooves on the lens bayonet mount 1332 fit
into the camera body mount 1330. When the lens is correctly held
within the camera bayonet, electrical connectors 1342 on the back
of the lens align and couple with the camera electrical connectors
1340 on the camera body. FIG. 9 also shows a further coupling
component pair including a meter coupling ridge 1350 and an Ai
coupling lever 1352. In addition, an aperture indexing post 1360 of
the lens 1310 is shown which couples with an aperture control
coupling lever 1362 of the camera 1320. Although not shown in the
illustration of FIG. 9, the interface of the camera 1320 and the
lens 1310 may preferably include several other coupling or related
optical system components including a focal length indexing ridge
and focal length indexing pin pair, a lens-type signal notch and a
lens-type signal pin pair, and a lens speed indexing post and a
lens speed indexing lever pair. One or more of these coupling
component pairs may be including within the electrical connectors
1340, 1342, or may be otherwise disposed at the interface.
Moreover, an AF coupling on the lens 1310 may be coupled with an AF
coupler on the camera body 1320, e.g., in a exemplary Auto focus
lens Nikkor.RTM. AF lens-type and Nikon.RTM. F4 body
configuration.
[0065] Through these connectors, information may be communicated to
and from the lens 1310 to the camera 1320. Such information may
include details about the lens type, magnification and/or focal
length. Other parameters may include aperture size or F-number.
Conversely, the camera 1320 may send information to the lens 1310
such as for setting up a focus ring or otherwise initializing or
calibrating with the camera 1220 or other digital image acquisition
device 1220. The connectors illustrated at FIG. 13 and/or those
just described above, may also serve to transfer lens calibration
data relevant to dust or dust artifact within images. The lens
parameter data may be entirely digitally-stored in a permanent
memory of the lens assembly, and may be made accessible by a camera
micro-processor upon electrical, optical or other signal coupling.
This data may be in the form of analytical parameters, such as exit
pupil dimension, exit pupil distance regarding the lens or distance
of dust to the sensor (which may often include a thickness of a
protective CCD sensor cover material, such as an anti-aliasing or
an optical spacer), or other parameters, and may be a table, e.g.,
a look up table, describing the dust movement as a function of the
focal length, etc., as discrete points, or a mathematical formula
describing this relationship, or otherwise as set forth above or
below herein. The lens calibration may include other data such as
the effect of the aperture on the dust.
[0066] Values of extracted parameters relating to the optical
system including the lens assembly may be embedded within the lens
system, wherein by "embedded" it is meant that the information is
stored or contained in whatever form within or on or in connection
with the lens assembly. This embedded information may preferably be
within a Flash or EEPROM memory chip. RAM is an alternative, but is
"volatile" and would utilize a back-up battery to retain data
storage when powered down. Preferably, the lens system would use
the power of the camera when coupled thereto, and only
alternatively would have its own separate power supply or back-up
battery. In that case, the battery may be charged when coupled to
the camera.
[0067] Therefore, relevant data is advantageously digitally-stored
in an optical calibration table, or otherwise, within a chip or
other digital storage component of the lens holder 1310, optical
mount 1310 or the lens 1310 itself. The table is made accessible to
a camera-resident micro-processor through the electrical or other
signal connections 1340, 1342. When the lens is mechanically
coupled to the camera, it is also electrically and/or optically
coupled specifically so that the camera can access the table. There
may be an automatic focusing that is facilitated by this
communication, or other advantage involving this feature of
camera-lens communication. The lens parameter data may be entirely
digitally-stored in the table that is made accessible by the camera
micro-processor upon coupling, although it may be made accessible
by a switch that turns on the connection or by opening the shutter
or by other preparation for taking a picture.
[0068] There are many alternatives to the preferred embodiments
described above that may be incorporated into a image processing
method, a digital camera, and/or an image processing system
including a digital camera and an external image processing device
that may be advantageous. For example, an electronic circuit may be
designed to detect maximum and minimum dust detection probability
thresholds while acquiring pixels in an image (see also U.S. Pat.
No. 5,065,257 to Yamada, hereby incorporated by reference). Such a
circuit can produce signals that may be used in processing to
eliminate regions, which lie outside the expected range of signal
values due to the presence of dust particles or similar optical
defects, or alternatively to accept, maintain or eliminate regions
as dust artifact regions based on whether a probability
determination exceeds, high or low, a certain threshold or
thresholds. A technique may be used to detect and provide a remedy
for the effects of dust on a digitally-acquired image (see also
U.S. Pat. No. 5,214,470 to Denber, hereby incorporated by
reference). An image may be recorded within a digital camera or
external processing device such as may be signal coupled with a
digital camera for receiving digital output image information or a
device that acquires or captures a digital image of a film image.
The digital image may be compared with the film image, e.g.,
through a logical XOR operation, which may be used to remove dust
spots or blemishes picked up in the acquisition of the digital
image from the film image.
[0069] Multiple images may be processed and stationary components,
which are common between images, may be detected and assigned a
high probability of being a defect (see also U.S. Pat. No.
6,035,072 to Read, hereby incorporated by reference). Additional
techniques, which may be employed to modify defect probability, may
include median filtering, sample area detection and dynamic
adjustment of scores. This dynamic defect detection process allows
defect compensation, defect correction and alerting an operator of
the likelihood of defects.
[0070] Dark field imaging may be employed to determine the location
of defects in digital images from digital cameras or film scanners
(see U.S. Pat. No. 5,969,372 to Stavely et al., and US patent
application 2001/0035491 to Ochiai et al., each hereby incorporated
by reference). A normal imaging of a object with normal
illumination may be followed by a second imaging using different
wavelengths, e.g., infrared illumination. Dust, fingerprints,
scratches and other optical defects are typically opaque to
infrared light. Thus the second image produces an image with dark
spots indicating the position of dust particles or other
defects.
[0071] A process may involve changing any of a variety of extracted
parameters (see elsewhere herein), angle of sensor relative to
image plane, distance of image plane or sensor from dust specks
(e.g., on window of sensor), etc., and imaging a same object with
the digital camera. A comparison of the images reveals with
enhanced probability the locations of dust artifact. In a camera
application, the unique location of the actual dust relative to the
object and to the image plane provide information about extracted
parameter-dependent characteristics of dust artifact in the images.
The analysis for the digital camera application depends on the
"transmission"-based optical parameters of the system, i.e., the
fact that light travels from a scene through the camera lens and
onto the camera sensor, and not involving any substantial
reflective effects. It is possible to make determinations as to
where the dust actually is in the system by analyzing multiple
images taken with different extracted parameters, e.g., on the
sensor window, or in an image of an original object which itself is
being images such as in film imaging.
[0072] In a scanning application, this technique can be use the
face that a speck of dust will cast a shadow of a different color,
geometry location, etc. with changes in extracted parameters, e.g.,
with a different color with increasing elongation of the shadow for
each parallel row of pixels (a "rainbow" shadow, as it were).
Multiple scans taken from various angles of illumination may be
employed to produce an image which identifies dust defects from
their shadows and the colors thereof (see U.S. Pat. No. 6,465,801
to Gann et al. and US patent applications 2002/0195577 and
2002/0158192 to Gann et al, hereby incorporated by reference). A
linear scanning element moves across a document (or the document is
moved across the scanning element) and an image of the document is
built up as a series of rows of pixels. This differs from the
physical configuration of a camera in which a shutter illuminates a
X-Y sensor array with a single burst of light. In both cases,
though, dust may lie close to the imaging plane of the sensor.
[0073] Technique may be applied as part of a photofinishing process
to eliminate blemishes on a film image obtained by a digital camera
(see also US patent application 2001/0041018 to Sonoda, hereby
incorporated by reference). Such techniques may import previous
acquired information about defects in images from a blemish
detection procedure. A technique for correcting image defects from
a digital image acquisition device such as a digital camera may
involve repeated imaging of an object or other image, where each
successive image-acquisition involves different properties or
extracted parameters or meta-data related properties, such as
variable angles of incidence or variable lighting or contrast
parameters, and the results of these repeated scans may be combined
to form a reference image from which defect corrections are made
(see also US patent application 2003/0118249 to Edgar, hereby
incorporated by reference).
[0074] A decision on whether a defect in a image acquired by a
field-based digital camera is to be corrected or not may be based
on a balancing of considerations. For example, the likely damage to
surrounding defect-free portions of the image may be balanced
against the likelihood of successfully achieving correction of the
defect.
[0075] Image processing means may be employed where the detection
or correction of defects in a digital image may be based solely on
analysis of the digital image, or may employ techniques directly
related to the image acquisition process, or both. Anomalous image
regions may be determined based on the difference between the
gradient of an image at a set of grid points and the local mean of
the image gradient (e.g., see U.S. Pat. No. 6,233,364 to Krainiouk
et al., hereby incorporated by reference). Such technique can
reduce the number of false positives in "noisy" regions of an image
such as those representing leaves in a tree, or pebbles on a beach.
After determining an initial defect list by this means, the
technique may involve culling the list based on a one or more or a
series of heuristic measures based on color, size, shape and/or
visibility measures where these are designed to indicate how much
an anomalous region resembles a dust fragment or a scratch.
[0076] Techniques and means to correct scratches in a digitized
images may employ a binary mask to indicate regions requiring
repair or noise removal, and sample and repair windows to indicate
(i) the region requiring repair and/or (ii) a similar "sample" area
of the image (see also U.S. Pat. No. 5,974,194 to Hirani et al.,
hereby incorporated by reference). Data from a sample window may be
converted to a frequency domain and combined with frequency domain
data of the repair window. When a low-pass filter is applied, it
has the effect to remove the sharp, or high-frequency, scratch
defect.
[0077] Techniques and means of detecting potential defect or
"trash" regions within an image may be based on a comparison of the
quadratic differential value of a pixel with a pre-determined
threshold value (see U.S. Pat. No. 6,125,213 to Morimoto, hereby
incorporated by reference). The technique may involve correcting
"trash" regions within an image by successively interpolating from
the outside of the "trash" region to the inside of this region.
[0078] Techniques and means to automate the removal of narrow
elongated distortions from a digital image may utilize the
characteristics of image regions bordering the distortion (see also
U.S. Pat. No. 6,266,054 to Lawton et al., hereby incorporated by
reference). User input may be used to mark the region of the
defect, or automatic defect detection may be employed according to
a preferred embodiment herein, while the process of delineating the
defect is also preferably also performed automatically.
[0079] Techniques and means to allow automatic alteration of
defects in digital images may be based upon a defect channel having
a signal proportional to defects in the digital image (see also
U.S. Pat. No. 6,487,321 to Edgar et al., hereby incorporated by
reference). This allows areas of strong defect to be more easily
excised without causing significant damage to the area of the image
surrounding the defect.
[0080] Techniques and means may be employed to generate replacement
data values for an image region (see also U.S. Pat. No. 6,587,592
to Georgiev et al., hereby incorporated by reference) Image defect
may be repaired as facilitated by the replacement data. Moreover,
the repairing of the unwanted image region may preserves image
textures within the repaired (or "healed") region of the image.
[0081] Techniques and means may be employed to detect defect pixels
by applying a median filter to an image and subtracting the result
from the original image to obtain a difference image (see also US
patent application 2003/0039402 and WIPO patent application
WO-03/019473, both to Robins et al., each hereby incorporated by
reference). This may be used to construct at least one defect map.
Correction of suspected defect pixels may be achieved by replacing
those pixel values with pixel values from the filtered image and
applying a smoothing operation. User input may or may not be
utilized to further mitigate the effects of uncertainty in defect
identification.
[0082] Techniques and means for retouching binary image data which
is to be presented on a view-screen or display apparatus may be
employed to eliminate local screen defects such as dust and scratch
artifacts (see also US patent application 2002/0154831 to Hansen et
al., hereby incorporated by reference). The production of visible
moire effects in the retouched image data may be avoided by the
replacement of small areas.
[0083] A digital video camera with sensor apparatus may incorporate
a defect detecting mode (see also U.S. Pat. No. 5,416,516 to
Kameyama et al., hereby incorporated by reference). The locations
of detected defect pixels may be retained in the memory of the
camera apparatus and replacement pixel values may be interpolated
by processing algorithms, which convert the sensor data into
digital image pixel values. Techniques may be employed to
automatically detect and compensate for defective sensor pixels
within a video camera (see also U.S. Pat. No. 5,625,413 to Katoh et
al., hereby incorporated by reference). The camera may perform a
dark current measurement on start-up when the camera iris is closed
and by raising the gain can determine pixels which exhibit
abnormally high dark current values. The location of these pixels
is recorded in camera memory as a LUT with associated threshold
brightness values associated with each pixel depending on its dark
current value; defect compensation depends on input image
brightness and ambient temperature.
[0084] An image pickup apparatus, such as a digital camera, may
have a detachable lens (see also US patent application 2003/0133027
to Itoh, hereby incorporated by reference). The camera may
incorporate a defect detecting section and a compensation section
for sensor defects. Further the defect detection operation may
become active when the camera lens is detached so that the user
will not miss an opportunity to take a picture due to the operation
of the defect detection process.
[0085] The techniques of the preferred and alternative embodiments
described herein may be applied to cameras with interchangeable
lens units (see also U.S. Pat. No. 5,003,399 to Masayoshi et. al.,
hereby incorporated by reference). The camera and lens have first
and second computers in the camera body and lens device,
respectively, which are programmed so that, in an initial
communication sequence there between after the selected lens device
is mounted on the camera body, the second microcomputer transmits
optical characteristic data for the respective lens device to the
first microcomputer in response to a data transmission request
signal from the latter.
[0086] The techniques of the preferred and alternative embodiments
described herein may be applied to printers and to imaging devices
such as a digital cameras which incorporate a focusing lens system.
A process may be employed for detecting and mapping dust on the
surface of a photographic element (see also U.S. Pat. No. 5,436,979
to Gray et al., hereby incorporated by reference). This may be
applied in the context of a verification procedure to follow a
cleaning process for a range of photographic elements including
film negatives and slides. Statistical information may be obtained
and presented to an operator to allow control of the cleaning
process. Detailed location information may be also recorded and/or
correction means may be also provided for dust defects on a
photographic element.
[0087] Techniques and means to create a defect map for a digital
camera or similar imaging device may use an all-white reference
background (see also US patent application 2002/0093577 to Kitawaki
et al., hereby incorporated by reference). The location of any dust
or scratch defects may be recorded in the memory of the imaging
apparatus when the camera is in a dust detection mode and when a
dust correction circuit is active any image data co-located with a
defect may be corrected for the presence of dust by elimination,
color correction or interpolation based on the surrounding pixels.
Further, where the position of a dust defect changes with f-stop of
the camera a list of dust locations corresponding to f-stop
settings is pre recorded at the time of manufacturing in a LUT in
the camera memory. Any effect of different focal length may be
simplified to the effect of the change in dust due to magnification
of the lens. In addition, techniques for dynamically obtaining a
defect map based on the processing of a plurality of images may be
employed with this technique.
[0088] Techniques may be also employed involving correcting for
dust defects based on the geometry of said dust or of the camera.
Further techniques may involve utilizing camera metadata to enhance
the detection and correction processes for defects. Further
techniques may involve alerting the user that the camera requires
servicing due to excessive levels of dust contamination, or the
fact that it is not only magnification but the actual lens that is
mounted.
[0089] A method of filtering dust artifacts form an acquired
digital image including multiplicity of pixels indicative of dust,
the pixels forming various shapes in the image, may be employed.
The method may include analyzing image information including
information describing conditions under which the image was
acquired and/or acquisition device-specific information. One or
more regions may be determined within the digital image suspected
as including dust artifact. Based at least in part on said
meta-data analysis, it may be determined whether the regions are
actual dust artifact.
[0090] A method may further include analyzing the images in
comparison to a predetermined dust map to establish the validity of
the dust over progressions of time. The method may further involve
mapping the acquired image to a predetermined default acquisition
condition based on a lens and camera calibration tables, as a
function of the lens type and the focal length that was used at
acquisition.
[0091] A method may further include mapping a dust spec as depicted
in the dust map and the suspected dust specs in the acquired image
based on a calculated transformation of the dust as a function of
the lens and the aperture, or other extracted parameter, used to
acquire the image. The actual removal of dust artifacts from an
image may include a step where missing data as obscured by the dust
specs is regenerated and in-painted based on analysis of the region
in the image surrounding the dust spec. The actual removal of the
dust artifacts from the image may also include a step where
deteriorated regions primarily in the periphery of the dust spec
are enhanced and restored based on knowledge of the deterioration
function. The actual image retouching may include both in-painting
and restoration or either one of these operations, or another image
correction technique as may be understood by those skilled in the
art.
[0092] A method of detecting and removing dust artifacts may be
performed in the acquisition device as a post-processing stage
prior to saving the image. This method may further include an
analysis of the image in its raw format immediately followed by the
acquisition stage. The method of detecting and removing dust
artifacts can be performed on an external device as part of a
download or capture process. Such external device may be a personal
computer, a storage device, and archival device, a display or a
printing device or other device. The method of detecting and
removing dust artifacts can be performed in part in the acquisition
device and the external device.
[0093] A dust detection and/or correction technique may be applied
post priori to a collection of images, or individually to images as
they are added to a collection. The map may be generated a priori
to the introduction of an image, or dynamically and in concurrence
to the introduction of new images. The method may further include
steps of providing a statistical confidence level as to the fact
that a certain region is indeed part of a dust spec. The method may
further provide tools to determine whether the acquisition device
may benefit from some maintenance. A method may be employed that
may be implemented as part of a digitization process, such as
correcting defects on scanning device, whether flat bed or drum,
whether for hard copy documents or for film digitization. A method
may be further applied to other recurring image imperfections such
as dead pixels on the sensor, burnt pixels on the sensor,
scratches, etc. A method of automatically determining whether to
recommend servicing a digital image acquisition system including a
digital camera based on dust analysis may be advantageously
employed. A method of calculating parameters of an optical system
may be based on analysis of the dust.
[0094] While an exemplary drawings and specific embodiments of the
present invention have been described and illustrated, it is to be
understood that that the scope of the present invention is not to
be limited to the particular embodiments discussed. Thus, the
embodiments shall be regarded as illustrative rather than
restrictive, and it should be understood that variations may be
made in those embodiments by workers skilled in the arts without
departing from the scope of the present invention as set forth in
the claims that follow and their structural and functional
equivalents.
[0095] In addition, in methods that may be performed according to
preferred embodiments herein, the operations have been described in
selected typographical sequences. However, the sequences have been
selected and so ordered for typographical convenience and are not
intended to imply any particular order for performing the
operations, unless a particular ordering is expressly provided or
understood by those skilled in the art as being necessary.
[0096] Many references have been cited above herein, and in
addition to that which is described as background, the invention
summary, brief description of the drawings, the drawings and the
abstract, these references are hereby incorporated by reference
into the detailed description of the preferred embodiments, as
disclosing alternative embodiments of elements or features of the
preferred embodiments not otherwise set forth in detail above. A
single one or a combination of two or more of these references may
be consulted to obtain a variation of the preferred embodiments
described in the detailed description above.
* * * * *