U.S. patent application number 11/977378 was filed with the patent office on 2008-05-08 for imaging apparatus, display apparatus, imaging method, and display method.
This patent application is currently assigned to Sony Corporation. Invention is credited to Masamichi Asukai, Taiji Ito, Yoichiro Sako, Masaaki Tsuruta.
Application Number | 20080107361 11/977378 |
Document ID | / |
Family ID | 39247099 |
Filed Date | 2008-05-08 |
United States Patent
Application |
20080107361 |
Kind Code |
A1 |
Asukai; Masamichi ; et
al. |
May 8, 2008 |
Imaging apparatus, display apparatus, imaging method, and display
method
Abstract
Disclosed herein is an imaging apparatus including: imaging
means for imaging a scene that is in a direction in which a user
sees to obtain image data of the scene; biological sensor means for
detecting biological information concerning the user; and control
means for generating metadata based on at least the biological
information obtained by the biological sensor means, and performing
a storage process of storing the image data obtained by the imaging
means so as to be associated with the metadata.
Inventors: |
Asukai; Masamichi;
(Kanagawa, JP) ; Tsuruta; Masaaki; (Tokyo, JP)
; Ito; Taiji; (Kanagawa, JP) ; Sako; Yoichiro;
(Tokyo, JP) |
Correspondence
Address: |
WOLF GREENFIELD & SACKS, P.C.
600 ATLANTIC AVENUE
BOSTON
MA
02210-2206
US
|
Assignee: |
Sony Corporation
Tokyo
JP
|
Family ID: |
39247099 |
Appl. No.: |
11/977378 |
Filed: |
October 24, 2007 |
Current U.S.
Class: |
382/317 |
Current CPC
Class: |
A61B 3/113 20130101;
A61B 5/021 20130101; H04N 13/332 20180501; A61B 5/389 20210101;
A61B 5/0816 20130101; A61B 5/0205 20130101; A61B 5/0531 20130101;
A61B 5/16 20130101; A61B 5/00 20130101; A61B 5/369 20210101; A61B
5/145 20130101; A61B 5/024 20130101; A61B 5/02055 20130101; A61B
5/11 20130101 |
Class at
Publication: |
382/317 |
International
Class: |
G06K 9/20 20060101
G06K009/20 |
Foreign Application Data
Date |
Code |
Application Number |
Nov 7, 2006 |
JP |
JP2006-301601 |
Claims
1. An imaging apparatus, comprising: imaging means for imaging a
scene that is in a direction in which a user sees to obtain image
data of the scene; biological sensor means for detecting biological
information concerning the user; and control means for generating
metadata based on at least the biological information obtained by
said biological sensor means, and performing a storage process of
storing the image data obtained by said imaging means so as to be
associated with the metadata.
2. The imaging apparatus according to claim 1, wherein said control
means adds the metadata to the image data obtained by said imaging
means and performs the storage process when a predetermined storage
condition has been satisfied.
3. The imaging apparatus according to claim 1, further comprising
recording means for recording data on a nonvolatile storage medium,
wherein said control means performs, as the storage process, a
process of controlling said recording means to record the image
data obtained by said imaging means and the metadata on the
nonvolatile storage medium.
4. The imaging apparatus according to claim 1, further comprising
transmission means for transmitting data to an external device,
wherein said control means performs, as the storage process, a
process of controlling said transmission means to transmit the
image data obtained by said imaging means and the metadata to the
external device.
5. The imaging apparatus according to claim 1, wherein the
biological information is at least one of a pulse, heart beats, an
electrocardiogram, electromyographic information, breathing,
perspiration, GSR, blood pressure, a saturation oxygen
concentration in blood, a skin surface temperature, brain waves, a
blood flow change, a body temperature, a motion of a body, a motion
of a head, a center of gravity, rhythm of walking/running, and a
state of an eye.
6. The imaging apparatus according to claim 2, wherein the metadata
generated by said control means includes information of a content
of the satisfied storage condition.
7. The imaging apparatus according to claim 1, wherein the metadata
generated by said control means includes information of a date and
time or a current location.
8. The imaging apparatus according to claim 1, wherein the metadata
generated by said control means includes user identification
information.
9. A display apparatus that acquires image data from a nonvolatile
storage medium having stored therein image data to which metadata
based on biological information is added, and displays the acquired
image data, said display apparatus comprising: display means;
biological sensor means for detecting biological information
concerning a user; and control means for generating a search
condition based on the biological information detected by said
biological sensor means, and controlling image data found in the
nonvolatile storage medium as a result of a search using the search
condition to be displayed on said display means.
10. The display apparatus according to claim 9, further comprising
playback means for playing back data stored in the nonvolatile
storage medium, wherein said control means controls said playback
means to display the image data found as a result of the search
using the search condition on said display means.
11. The display apparatus according to claim 9, further comprising
communication means for performing data communication with an
external device that reads data from the nonvolatile storage
medium, wherein said control means performs control for
transmitting the search condition to the external device via said
communication means to request the external device to perform the
search, acquiring the image data found as a result of the search
performed in the external device via said communication means, and
displaying the acquired image data on said display means.
12. The display apparatus according to claim 9, wherein the
biological information is at least one of a pulse, heart beats, an
electrocardiogram, electromyographic information, breathing,
perspiration, GSR, blood pressure, a saturation oxygen
concentration in blood, a skin surface temperature, brain waves, a
blood flow change, a body temperature, a motion of a body, a motion
of a head, a center of gravity, rhythm of walking/running, and a
state of an eye.
13. The display apparatus according to claim 9, wherein image data
corresponding to a time when a predetermined storage condition was
satisfied is stored in the nonvolatile storage medium together with
metadata including information of a content of the storage
condition, and the search condition generated by said control means
includes the information of the content of the storage
condition.
14. The display apparatus according to claim 9, wherein the image
data is stored in the nonvolatile storage medium together with
metadata including information of a date and time or a current
location, and the search condition generated by said control means
includes the information of the date and time or the current
location.
15. The display apparatus according to claim 9, wherein the image
data is stored in the nonvolatile storage medium together with
metadata including user identification information, and the search
condition generated by said control means includes the user
identification information.
16. The display apparatus according to claim 9, wherein said
display means is capable of causing a whole or a part of a screen
area for image display to enter a transparent or translucent
see-through state.
17. The display apparatus according to claim 16, wherein said
control means controls said display means to cause a part of the
screen area for image display to enter the see-through state, and
display the image data found in and read from the nonvolatile
storage medium in a remaining part of the screen area.
18. The display apparatus according to claim 16, wherein said
control means controls said display means to cause the whole of the
screen area for image display to enter the see-through state, and
display the image data found in and read from the nonvolatile
storage medium in the whole or a part of the screen area.
19. The display apparatus according to claim 16, wherein said
control means controls said display means to switch between the
see-through state and a state in which the image data found in and
read from the nonvolatile storage medium is displayed.
20. The display apparatus according to claim 9, wherein said
display means is arranged in front of an eye of the user to perform
image display.
21. An imaging method comprising the steps of: imaging a scene that
is in a direction in which a user sees to obtain image data of the
scene; detecting biological information concerning the user;
generating metadata based on at least the biological information
obtained in said detecting; and storing the image data obtained in
said imaging so as to be associated with the metadata generated in
said generating.
22. A display method employed in a display apparatus that acquires
image data from a nonvolatile storage medium having stored therein
image data to which metadata based on biological information is
added, and displays the acquired image data, said method comprising
the steps of: detecting biological information concerning a user;
generating a search condition based on the biological information
detected in said detecting; and displaying image data found in the
nonvolatile storage medium as a result of a search using the search
condition generated in said generating.
23. An imaging apparatus, comprising: an imaging section configured
to image a scene that is in a direction in which a user sees to
obtain image data of the scene; a biological sensor section
configured to detect biological information concerning the user;
and a control section configured to generate metadata based on at
least the biological information obtained by said biological sensor
section, and performing a storage process of storing the image data
obtained by said imaging section so as to be associated with the
metadata.
24. A display apparatus that acquires image data from a nonvolatile
storage medium having stored therein image data to which metadata
based on biological information is added, and displays the acquired
image data, said display apparatus comprising: a display section; a
biological sensor section configured to detect biological
information concerning a user; and a control section configured to
generate a search condition based on the biological information
detected by said biological sensor section, and controlling image
data found in the nonvolatile storage medium as a result of a
search using the search condition to be displayed on said display
section.
Description
CROSS REFERENCES TO RELATED APPLICATIONS
[0001] The present invention contains subject matter related to
Japanese Patent Application JP 2006-301601, filed in the Japanese
Patent Office on Nov. 7, 2006, the entire contents of which being
incorporated herein by reference.
BACKGROUND OF THE INVENTION
[0002] 1. Field of the Invention
[0003] The present invention relates to an imaging apparatus and an
imaging method for imaging a scene that is in a direction in which
a user sees. The present invention also relates to a display
apparatus and a display method for retrieving image data obtained
by imaging by and stored in the above imaging apparatus and
displaying the retrieved image data.
[0004] 2. Description of the Related Art
[0005] There has been proposed an apparatus that has a
spectacle-shaped or head-worn wearing unit to which a small camera
is attached so as to be able to image a scene that is in the
direction in which the user sees (see, for example,
JP-A-2005-172851).
[0006] JP-A-2006-146630 discloses a technique that uses biological
information to select content, such as a tune, that is suited to a
situation of the user, and presents the selected content to the
user.
SUMMARY OF THE INVENTION
[0007] The capability to image the scene that is in the direction
in which the users sees allows, for example, a scene that the user
sees in his or her daily life to be recorded as image data thereof,
but no known apparatus is capable of allowing the image data
obtained by imaging to be used in a mode desirable or enjoyable for
the user.
[0008] As such, the present invention has been devised to enable
appropriate storage of a photographed image and interesting use of
image data of the photographed image.
[0009] According to one embodiment of the present invention, there
is provided an imaging apparatus including: imaging means for
imaging a scene that is in a direction in which a user sees to
obtain image data of the scene; biological sensor means for
detecting biological information concerning the user; and control
means for generating metadata based on at least the biological
information obtained by the biological sensor means, and performing
a storage process of storing the image data obtained by the imaging
means so as to be associated with the metadata.
[0010] According to another embodiment of the present invention,
there is provided a display apparatus that acquires image data from
a nonvolatile storage medium having stored therein image data to
which metadata based on biological information is added, and
displays the acquired image data, the display apparatus including:
display means; biological sensor means for detecting biological
information concerning a user; and control means for generating a
search condition based on the biological information detected by
the biological sensor means, and controlling image data found in
the nonvolatile storage medium as a result of a search using the
search condition to be displayed on the display means.
[0011] According to yet another embodiment of the present
invention, there is provided an imaging method including the steps
of: imaging a scene that is in a direction in which a user sees to
obtain image data of the scene; detecting biological information
concerning the user; generating metadata based on at least the
biological information obtained in the detecting step; and storing
the image data obtained in the imaging step so as to be associated
with the metadata generated in the generating step.
[0012] According to yet another embodiment of the present
invention, there is provided a display method employed in a display
apparatus that acquires image data from a nonvolatile storage
medium having stored therein image data to which metadata based on
biological information is added, and displays the acquired image
data, the method including the steps of: detecting biological
information concerning a user; generating a search condition based
on the biological information detected in the detecting step; and
displaying image data found in the nonvolatile storage medium as a
result of a search using the search condition generated in the
generating step.
[0013] According to the above-described embodiments of the present
invention, the user wears the imaging apparatus using, for example,
a spectacle-shaped or head-worn wearing unit so that the imaging
means is able to image a scene that is in a direction in which the
user sees. The metadata generated using at least the biological
information concerning the user is added to the image data obtained
by imaging by the imaging means, and the image data is stored
together with the added metadata.
[0014] Meanwhile, the display apparatus detects at least the
biological information concerning the user, and generates the
search condition using at least the biological information. Then,
the display apparatus acquires and displays image data that has
been found as a result of a search performed across the stored
image data using the search condition.
[0015] Thus, for example, an image of a scene at an earlier time
when the user was in a similar situation (for example, have similar
feelings) to the current one is displayed for the user. Thus,
automatic display of a memorable scene is realized, for
example.
[0016] Note that the storage process performed in the imaging
apparatus and the imaging method refers to: a process of causing,
for example, recording means within the imaging apparatus to record
the image data obtained by imaging on a nonvolatile storage medium
such as a hard disk drive (HDD), an optical disk, a magneto-optical
disk, a magnetic disk, or a flash memory; or a process of causing
transmission means to transmit the image data obtained by imaging
to an external device so that the external device can store the
image data in an HDD or the like.
[0017] In the display apparatus and the display method, a search is
performed across the image data recorded on and stored in such a
nonvolatile storage medium, and image data retrieved is
displayed.
[0018] In the imaging apparatus and the imaging method according to
the present invention, a scene that the user sees in his or her
daily life is stored together with the metadata generated based on
the biological information concerning the user. Thus, an image of
the scene that the user sees in his or her daily life can be stored
so as to be associated with a situation of the user, which provides
convenience when performing a search for subsequent playback or the
like.
[0019] In the display apparatus and the display method according to
the present invention, a search is performed across the stored
image data using the search condition generated using the
biological information, and the retrieved image data is displayed.
Thus, an image of a past scene (e.g., a past scene that the user
himself or herself saw, a past scene that another person saw, etc.)
that is suited to the current situation of the user is presented to
the user. Thus, very entertaining image presentation is achieved,
such as presentation of an image that reminds the user of a memory,
presentation of an image that allows the user to watch a similar
scene in the past, or presentation of an image of an impressive
event.
BRIEF DESCRIPTION OF THE DRAWINGS
[0020] FIG. 1 is an illustration of an exemplary appearance of an
imaging/display apparatus according to one embodiment of the
present invention;
[0021] FIG. 2 is an illustration of an exemplary appearance of
another imaging/display apparatus according to one embodiment of
the present invention;
[0022] FIGS. 3A to 3C are diagrams illustrating relationships
between an imaging/display apparatus according to one embodiment of
the present invention and an external device;
[0023] FIG. 4 is a block diagram of an imaging/display apparatus
according to one embodiment of the present invention;
[0024] FIG. 5 is a block diagram of another imaging/display
apparatus according to one embodiment of the present invention;
[0025] FIG. 6 is a block diagram of yet another imaging/display
apparatus according to one embodiment of the present invention;
[0026] FIGS. 7A to 7C are illustrations for describing a
see-through state, an image obtained by regular imaging, a
telephoto image, respectively, according to one embodiment of the
present invention;
[0027] FIGS. 8A and 8B are illustrations for describing a magnified
image according to one embodiment of the present invention;
[0028] FIGS. 9A and 9B are illustrations for describing an image
photographed with increased infrared sensitivity according to one
embodiment of the present invention;
[0029] FIGS. 10A and 10B are illustrations for describing an image
photographed with increased ultraviolet sensitivity according to
one embodiment of the present invention;
[0030] FIGS. 11A and 11B are diagrams illustrating a temporary
storage section and a replay operation according to one embodiment
of the present invention;
[0031] FIGS. 12A and 12B are illustrations for describing
displaying of replay images according to one embodiment of the
present invention;
[0032] FIG. 13 is a flowchart illustrating a control procedure
according to one embodiment of the present invention;
[0033] FIG. 14 is a flowchart illustrating a replay procedure
according to one embodiment of the present invention;
[0034] FIG. 15 is a flowchart illustrating an imaging system
control procedure according to one embodiment of the present
invention;
[0035] FIG. 16 is a flowchart illustrating a procedure for
generating metadata according to one embodiment of the present
invention;
[0036] FIG. 17 is a flowchart illustrating a procedure for
displaying a stored image according to one embodiment of the
present invention;
[0037] FIGS. 18A and 18B are illustrations of how the stored image
is displayed according to one embodiment of the present invention;
and
[0038] FIGS. 19A to 19C are illustrations of how the stored image
is displayed according to one embodiment of the present
invention.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0039] Hereinafter, an imaging apparatus, an imaging method, a
display apparatus, and a display method according to embodiments of
the present invention will be described in the following order,
with reference to an exemplary case of an imaging/display
apparatus. [0040] [1. Exemplary appearances of imaging/display
apparatus] [0041] [2. Exemplary structures of imaging/display
apparatus] [0042] [3. Exemplary photographed images] [0043] [4.
Replay operation] [0044] [5. Exemplary manners of selecting image
to be stored] [0045] [6. Imaging and exemplary storage procedure]
[0046] [7. Exemplary procedure for displaying stored image] [0047]
[8. Effects of embodiments, exemplary variants, and exemplary
expansions]
1. Exemplary Appearances of Imaging/Display Apparatus
[0048] FIG. 1 shows an exemplary appearance of an imaging/display
apparatus 1, which is a spectacle-shaped display camera, according
to one embodiment of the present invention. The imaging/display
apparatus 1 has a wearing unit having a frame structure that
extends halfway around a head from both temporal regions to an
occipital region, for example, and is worn by a user with the
wearing unit placed over ears as illustrated in this figure.
[0049] The imaging/display apparatus 1 has a pair of display
sections 2 designed for left and right eyes, and the display
sections 2 are arranged in front of the eyes of the user (i.e., at
positions where lenses of common spectacles would be located) when
the imaging/display apparatus 1 is worn by the user in a manner as
illustrated in FIG. 1. Liquid crystal panels, for example, are used
for the display sections 2, and the display sections 2 are capable
of entering a see-through state, i.e., a transparent or translucent
state, as illustrated in this figure by transmissivity control. The
capability of the display sections 2 to enter the see-through state
allows the user to wear the imaging/display apparatus 1 at all
times as he or she wears spectacles, with no interference occurring
in his or her daily life.
[0050] In addition, the imaging/display apparatus 1 has a
photographic lens 3a arranged to face forward so as to image a
scene that is in a direction in which the user sees while the
imaging/display apparatus 1 is worn by the user.
[0051] In addition, the imaging/display apparatus 1 has a pair of
earphone speakers 5a that can be inserted into right and left
earholes of the user when the imaging/display apparatus 1 is worn
by the user. Note that only the left earphone speaker 5a is shown
in the figure.
[0052] In addition, the imaging/display apparatus 1 has microphones
6a and 6b for collecting external sounds. The microphones 6a and 6b
are arranged to the right of the right display section 2 and to the
left of the left display section 2, respectively.
[0053] Note that FIG. 1 shows one example, and that various
structures are possible for the user to wear the imaging/display
apparatus 1. In general, a requirement for the wearing unit is that
it be in the shape of spectacles or of a head-worn type. At least,
a requirement for the present embodiment is that the display
sections 2 be arranged in front of and close to the eyes of the
user, and that the direction in which the photographic lens 3a
photographs is a direction in which the eyes of the user are
directed, i.e., in a forward direction. Also note that, instead of
having the pair of display sections 2 provided for both eyes, the
imaging/display apparatus 1 may have only one of the display
sections 2 provided for one eye.
[0054] Also note that the imaging/display apparatus 1 need not have
the left and right stereo speakers 5a, but may have only one of the
earphone speakers 5a to be inserted into only one earhole. Also
note that the number of microphones may be one. That is, the
imaging/display apparatus 1 may have only one of the microphones 6a
and 6b. Also note that the imaging/display apparatus 1 need not
have any microphone or earphone speaker.
[0055] Also note that the imaging/display apparatus may have a
lighting section that provides illumination in a direction of
photographing by the photographic lens 3a. The lighting section is
formed by a light emitting diode (LED), for example.
[0056] In the imaging/display apparatus 1 illustrated in FIG. 1, a
component for imaging and the display sections 2 for monitoring an
image obtained by imaging are integrated in one unit. However, as
with an imaging/display apparatus 1 as illustrated in FIG. 2, a
case containing the display section may be provided separately from
a case containing the component for imaging.
[0057] The imaging/display apparatus 1 as illustrated in FIG. 2
includes an imaging apparatus section 40 and a display apparatus
section 30, which are separate from each other.
[0058] The imaging apparatus section 40 is worn on a head of the
user via a predetermined wearing frame. The imaging apparatus
section 40 has the photographic lens 3a, which is arranged to face
forward so as to image a scene that is in the direction in which
the user sees while the imaging apparatus section 40 is worn by the
user. In addition, the imaging apparatus section 40 has the
microphone 6a for collecting the external sounds.
[0059] In this case, as described below, the imaging apparatus
section 40 contains a communication section for transmitting the
image data obtained by imaging and so on to the display apparatus
section 30, which is provided separately from the imaging apparatus
section 40.
[0060] The display apparatus section 30 is, for example, a small
and light device that the user is able to carry.
[0061] The display apparatus section 30 contains a communication
section internally for performing data communication with the
imaging apparatus section 40, and performs an operation of allowing
the image data supplied from the imaging apparatus section 40 to be
displayed on a display screen 31.
[0062] The user is able to use the imaging/display apparatus 1 in
which the imaging apparatus section 40 and the display apparatus
section 30 are separate from each other as described above.
[0063] Here, a portable display apparatus has been cited as the
display apparatus section 30. However, the display apparatus
section 30 may be a stationary display apparatus, a computer
apparatus, a television receiver, a mobile phone, a personal
digital assistant (PDA), or the like, for example. In short, in the
case of the imaging/display apparatus 1 as illustrated in FIG. 2,
which does not have a monitor display capability (and even in the
case of the imaging/display apparatus 1 as illustrated in FIG. 1,
which has the monitor display capability), the image data can be
monitored with any external display apparatus.
[0064] Note that examples of external devices to which the image
data obtained by imaging can be transmitted by the imaging/display
apparatus 1 via its communication capability include, in addition
to the various display devices as mentioned above, a video storage
device, a computer apparatus, and a server apparatus. That is, it
is conceivable that the photographed image data is stored in or
delivered by such an external device.
[0065] Also note that the appearance of the imaging/display
apparatus 1 is not limited to the examples of FIGS. 1 and 2, but
that various types of variations are possible.
[0066] The above examples of the imaging/display apparatus 1 have
the spectacle-shaped and head-worn wearing units, respectively.
Note, however, that the wearing unit used for the user to wear the
imaging/display apparatus may be of any type, such as a headphone
type, a neckband type, a behind-the-ear type, or the like. Further,
the imaging/display apparatus may be attached to common spectacles,
visor, headphone, or the like via a fixing device, such as a clip,
so that the imaging/display apparatus can be worn by the user. Also
note that it is not necessary that the imaging/display apparatus be
worn on the head of the user.
[0067] While exemplary internal structures of the imaging/display
apparatus 1 will be described later, the imaging/display apparatus
1 is capable of storing the image data obtained by imaging inside
itself (e.g., in a storage section 25 described below with
reference to FIG. 4), and also capable of transmitting the image
data obtained by imaging to the external device via a communication
section (e.g., a communication section 26 described below with
reference to FIG. 5) so that the image data can be stored in the
external device.
[0068] In addition, this imaging/display apparatus 1 is capable of
retrieving the image data stored therein (i.e., in the storage
section 25) and plays back the retrieved image data, or receiving,
from the external device (via the communication section 26), image
data found as a result of a search performed across image data
stored in the external device and displaying the received image
data.
[0069] FIGS. 3A to 3C show exemplary modes of use of the
imaging/display apparatus 1 in relation to external devices.
[0070] FIG. 3A shows a case in which the imaging/display apparatus
1 is used singly. In this case, the imaging/display apparatus 1
stores the image data obtained by imaging in the internal storage
section 25. In addition, the imaging/display apparatus 1 performs a
search across the image data stored in the storage section 25, and
plays back and displays the image data found as a result of the
search on the display section 2.
[0071] FIG. 3B shows a case where the imaging/display apparatus 1
has a communication capability (i.e., the communication section 26)
and communicates with an external storage device 70. In this case,
the imaging/display apparatus 1 transmits the image data obtained
by imaging to the storage device 70 via the communication section
26 so that the image data can be stored in the storage device 70.
The imaging/display apparatus 1 is also capable of transmitting a
search condition to the storage device 70 via the communication
section 26, and requesting the storage device 70 to perform a
search and transmit image data found as a result of the search, so
that the imaging/display apparatus 1 can receive the retrieved
image data. Then, the imaging/display apparatus 1 displays the
received image data, i.e., the image data found as a result of the
search, on the display section 2.
[0072] FIG. 3C shows a case where the communication section 26 in
the imaging/display apparatus 1 has a capability to access a
network 73, such as the Internet, to communicate with an external
server apparatus 71 connected to the imaging/display apparatus 1
via the network 73. In this case, the imaging/display apparatus 1
is capable of transmitting image data to be stored or the search
condition to the server apparatus 71 via the network 73, and also
capable of receiving image data found as a result of a search from
the server apparatus 71 via the network 73.
[0073] Various types of devices are conceivable as the storage
device 70 in FIG. 3B or the server apparatus 71 in FIG. 3C, such as
a device possessed by the user of the imaging/display apparatus 1,
a device possessed by an acquaintance of the user of the
imaging/display apparatus 1, and a device possessed by a public
institution, a service company, or the like that provides images
and which is capable of communicating with the imaging/display
apparatus 1.
[0074] Further, such devices may be of various types, such as a
video storage device, a home server device, a personal computer, a
personal digital assistant (PDA), a mobile phone, and a large-scale
server device.
2. Exemplary Structures of Imaging/Display Apparatus
[0075] Here, three exemplary structures of the imaging/display
apparatus 1 according to embodiments of the present invention will
be described with reference to FIGS. 4, 5, and 6. Exemplary
structures illustrated in FIGS. 4 and 5 correspond to the case
where the imaging/display apparatus 1 is the spectacle-shaped
display camera as illustrated in FIG. 1, which has both an imaging
capability and a display capability. Meanwhile, an exemplary
structure illustrated in FIG. 6 corresponds to the case where the
imaging apparatus section 40 and the display apparatus section 30
are provided separately as illustrated in FIG. 2.
[0076] First, the exemplary structure of FIG. 4 will now be
described below.
[0077] A system controller 10 is formed by a microcomputer that
includes a central processing unit (CPU), a read only memory (ROM),
a random access memory (RAM), a nonvolatile memory section, and an
interface section, for example, and controls an overall operation
of the imaging/display apparatus 1. The system controller 10
controls each part of the imaging/display apparatus 1 to perform a
predetermined operation based on a program held in the internal ROM
or the like.
[0078] The imaging/display apparatus 1 includes, as units for
imaging the scene that is in the direction in which the user sees,
an imaging section 3, an imaging control section 11, and an imaging
signal processing section 15.
[0079] The imaging section 3 includes: a lens system formed by the
photographic lens 3a illustrated in FIG. 1, an aperture, a zoom
lens, a focus lens, and the like; a driving system for allowing the
lens system to perform a focusing operation, a zoom operation, and
the like; and a solid-state imaging device array for detecting
light for imaging obtained by the lens system, and subjecting the
detected light to photoelectric conversion to generate an imaging
signal. The solid-state imaging device array is, for example, a
charge coupled device (CCD) sensor array or a complementary metal
oxide semiconductor (CMOS) sensor array.
[0080] The imaging signal processing section 15 includes a
sample-hold/automatic gain control (AGC) circuit for subjecting the
signal obtained by the solid-state imaging device in the imaging
section 3 to gain control and waveform shaping, and a video A/D
converter, and obtains an imaging signal in digital form.
[0081] In addition, the imaging signal processing section 15
performs white balancing processing, brightness processing, color
signal processing, blur correction processing, and the like on the
imaging signal. Further, the imaging signal processing section 15
is also capable of performing processes such as: brightness level
control, color correction, contrast control, and sharpness (edge
enhancement) control for the imaging signal; generation of a
magnified image in which a part of the imaging signal is magnified;
generation of a reduced image in which a part of the imaging signal
is reduced; application of image effects such as mosaicking,
brightness reversal, soft focus, highlighting of a part of the
image, and varying of the overall color atmosphere of the image;
generation of a character image or a conceptual image; and
combination of a generated image with the photographed image. In
short, the imaging signal processing section 15 is capable of
performing various processes on a digital video signal as the
imaging signal.
[0082] Based on an instruction issued from the system controller
10, the imaging control section 11 controls an imaging operation
achieved by operations of the imaging section 3 and the imaging
signal processing section 15. For example, the imaging control
section 11 controls activation and deactivation of the operations
of the imaging section 3 and the imaging signal processing section
15. In addition, the imaging control section 11 exercises control
(motor control) for allowing the imaging section 3 to perform an
operation such as autofocusing, automatic exposure adjustment,
aperture adjustment, zooming, or focus change.
[0083] The imaging control section 11 includes a timing generator,
and uses a timing signal generated by the timing generator to
control signal processing operations performed by the solid-state
imaging device, and the sample-hold/AGC circuit and the video A/D
converter in the imaging signal processing section 15. Moreover,
such timing control enables adjustment of an imaging frame
rate.
[0084] In addition, the imaging control section 11 controls imaging
sensitivity and signal processing in the solid-state imaging device
and the imaging signal processing section 15. For example, as
control of the imaging sensitivity, the imaging control section 11
is capable of performing the gain control on the signal read from
the solid-state imaging device, and black level setting, control of
various coefficients in processing the imaging signal in digital
form, control of a correction value in the blur correction
processing, and the like. Regarding the control of the imaging
sensitivity, overall sensitivity adjustment with no regard to any
particular wavelength range, and sensitivity adjustment of
adjusting imaging sensitivity of a particular wavelength range such
as an infrared range or an ultraviolet range (for example, imaging
that involves cutting off the particular wavelength range) are
possible, for example. Sensitivity adjustment in accordance with
the wavelength is achieved by insertion of a wavelength filter in a
photographic lens system or a wavelength filter operation process
performed on the imaging signal. In these cases, the imaging
control section 11 achieves the sensitivity control by controlling
the insertion of the wavelength filter, specification of a filter
operation coefficient, or the like.
[0085] Further, based on an instruction issued from the system
controller 10, the imaging control section 11 controls the
above-described image processing operation performed by the imaging
signal processing section 15.
[0086] A temporary storage section 23 is a storage section that
uses a solid-state memory, such as a dynamic random access memory
(D-RAM) or a static random access memory (S-RAM), as a storage
medium. Note, however, that the temporary storage section 23 may be
constructed as a unit for recording and reproducing data onto or
from a storage medium such as a flash memory, an optical disk, a
magnetic disk, or a memory card containing the flash memory.
Further, the temporary storage section 23 may be formed by an
HDD.
[0087] The temporary storage section 23 stores the image data
constantly obtained by imaging performed by the above-described
imaging system (hereinafter, the imaging section 3, the imaging
signal processing section 15, and the imaging control section 11
will be referred to collectively as the "imaging system"). That is,
when the image data has been supplied, the temporary storage
section 23 performs a predetermined encoding process for storage on
the image data and stores the encoded image data in the storage
medium. In addition, under control of the system controller 10, the
temporary storage section 23 is capable of reading the stored image
data from the storage medium, and decoding and outputting the image
data.
[0088] FIG. 11A is a schematic diagram of a storage area of the
memory, such as the D-RAM, in the temporary storage section 23. The
storage area, which extends from a top address AdST to an end
address AdED, is used in a ring memory manner. Specifically, the
image data is written to an address indicated by a write address
pointer W-Ad, which moves from the top address AdST toward the end
address AdED and, when it has reached the end address AdED, returns
to the top address AdST. In such a manner, the image data is
recorded on the storage area continuously.
[0089] Therefore, image data obtained in a period from a
predetermined time ago up to the present is stored temporarily in
the temporary storage section 23. Naturally, how long the image
data is stored therein depends on the capacity of the temporary
storage section 23 and a rate (e.g., a frame rate, a data amount
per frame, etc.) of the image data.
[0090] The frame rate at which the imaging system takes a video to
be stored in the temporary storage section 23 may be set at any
value. The image data obtained by imaging performed by the imaging
system and stored in the temporary storage section 23 may have
either a normal frame rate (e.g., 30 frames per second, for
example) or a higher frame rate. It is also possible to extract
several frames from the photographed image data in each second, for
example, and allow the temporary storage section 23 to store
resultant pseudo-moving image data. Moreover, it is also possible
to extract one frame every one to several seconds, for example, and
allow the temporary storage section 23 to store image data in the
form of still images obtained at intervals of one to several
seconds.
[0091] The imaging signal (i.e., the image data obtained by
imaging) obtained by imaging by the imaging section 3 and
processing by the imaging signal processing section 15 is supplied
to an image input/output control section 27.
[0092] Under control of the system controller 10, the image
input/output control section 27 controls transfer of the image
data. Specifically, the image input/output control section 27
controls the transfer of the image data among the imaging system
(i.e., the imaging signal processing section 15), the temporary
storage section 23, a display image processing section 12, and a
storage section 25.
[0093] In the imaging/display apparatus 1 according to the present
embodiment, the imaging system basically performs the imaging
operation constantly, and the image data obtained by imaging is
transferred to the temporary storage section 23 via the image
input/output control section 27 and temporarily stored in the
temporary storage section 23. Thus, the image data constantly
obtained by imaging is constantly recorded in the temporary storage
section 23 in the above-described manner, so that image data
obtained by imaging in a period from a certain time before up to
the present is stored therein constantly.
[0094] Under control of the system controller 10, the image
input/output control section 27 is also capable of supplying the
imaging signal (i.e., the image data) processed by the imaging
signal processing section 15 to the display image processing
section 12.
[0095] In addition, the image input/output control section 27 is
also capable of supplying image data read from the storage section
25, for example, to the display image processing section 12.
[0096] Further, under control of the system controller 10, the
image input/output control section 27 is capable of supplying image
data read from the temporary storage section 23 (i.e., some of the
image data temporarily stored in the temporary storage section 23)
to the display image processing section 12 or the storage section
25.
[0097] The storage section 25 is a unit for recording and
reproducing data onto or from a predetermined storage medium (e.g.,
a nonvolatile storage medium). The storage section 25 is formed by
a hard disk drive (HDD), for example. Needless to say, as the
nonvolatile storage medium, various types of storage media are
adoptable such as: a solid-state memory such as a flash memory, a
memory card containing a fixed memory, an optical disk, a
magneto-optical disk, and a hologram memory. A requirement for the
storage section 25 is to be capable of recording and reproducing
the data in accordance with the adopted storage medium.
[0098] Unlike the above-described temporary storage section 23, the
storage section 25 is provided for storing the image data not
temporarily but semi-permanently.
[0099] In particular, in the case where some of the image data
temporarily stored in the temporary storage section 23 has been
selected as a subject of a storage process, the selected image data
is read from the temporary storage section 23 and supplied to the
storage section 25 via the image input/output control section 27.
Under control of the system controller 10, the storage section 25
encodes the supplied image data so that it can be recorded on the
storage medium, and records the encoded image data on the storage
medium.
[0100] That is, the storage section 25 performs a process of, when
a predetermined condition for storage (hereinafter referred to as a
"storage condition") has been satisfied, storing, in the HDD, image
data that has been determined to be an object to be stored among
the image data temporarily stored in the temporary storage section
23.
[0101] When controlling the storage section 25 to perform the above
process of storing the image data, the system controller 10
generates metadata and controls the storage section 25 to record
the generated metadata together with the image data.
[0102] In addition, under control of the system controller 10, the
storage section 25 is capable of reproducing the recorded image
data. The reproduced image data is supplied to the display image
processing section 12 via the image input/output control section
27.
[0103] In particular, the system controller 10 generates a search
condition concerning the image data stored in the storage section
25, and allows the storage section 25 to perform a search using the
search condition. The search condition is automatically generated
by the system controller 10 based on a situation of the user and so
on. A content of the search condition is targeted at the metadata
added to the image data.
[0104] The storage section 25 performs a search using the given
search condition, and if any piece of image data that matches the
search condition is found, the system controller 10 allows the
storage section 25 to read the found image data. The read image
data is supplied to the display image processing section 12 via the
image input/output control section 27 and displayed on the display
section 2.
[0105] Although not shown in the figures, the reproduced image data
may also be output to an external device via a predetermined
interface section so as to be displayed by an external monitoring
device or stored in an external storage device, for example.
[0106] The imaging/display apparatus 1 includes, as units for
presenting a display to the user, the display section 2, the
display image processing section 12, a display driving section 13,
and a display control section 14.
[0107] Under control of the system controller 10, the image data
read from the temporary storage section 23 (i.e., the image data
obtained by imaging in the recent past), the image data obtained by
imaging by the imaging section 3 and processed by the imaging
signal processing section 15 (i.e., the image data that is being
currently obtained by imaging), or the image data read from the
storage section 25 (i.e., the stored image data) is supplied to the
display image processing section 12 via the image input/output
control section 27.
[0108] The display image processing section 12 performs signal
processing (e.g., the brightness level control, the color
correction, the contrast control, the sharpness (edge enhancement)
control, etc., for example) for displaying the supplied image data
on the display section 2, a split screen process, synthesis of the
character image, or the like.
[0109] The display driving section 13 is formed by a pixel driving
circuit for allowing an image signal supplied from the display
image processing section 12 to be displayed on the display section
(e.g., a liquid crystal display) 2. That is, the display driving
section 13 applies driving signals based on a video signal to
pixels arranged in a matrix in the display section 2 with specified
horizontal/vertical driving timing for displaying. In addition, the
display driving section 13 is capable of controlling transmissivity
of each of the pixels in the display section 2 to allow the pixel
to enter the see-through state.
[0110] Based on an instruction issued from the system controller
10, the display control section 14 controls a processing operation
of the display image processing section 12 and an operation of the
display driving section 13. Specifically, the display control
section 14 controls the display image processing section 12 to
perform the aforementioned various processes. Also, the display
control section 14 controls the display driving section 13 to
switch between the see-through state and an image displaying
state.
[0111] The imaging/display apparatus 1 further includes an audio
input section 6, an audio signal processing section 16, and an
audio output section 5.
[0112] The audio input section 6 includes the microphones 6a and 6b
illustrated in FIG. 1, a microphone amplifier section for
amplifying audio signals obtained by the microphones 6a and 6b, and
an A/D converter, and outputs audio data.
[0113] The audio data obtained by the audio input section 6 is
supplied to an audio input/output control section 28.
[0114] Under control of the system controller 10, the audio
input/output control section 28 controls transfer of the audio
data. Specifically, the audio input/output control section 28
controls transfer of the audio data among the audio input section
6, the audio signal processing section 16, the temporary storage
section 23, and the storage section 25.
[0115] Basically, the audio input/output control section 28
constantly supplies the audio data obtained by the audio input
section 6 to the temporary storage section 23. As a result, the
temporary storage section 23 constantly stores the audio data
obtained by the collecting of sound by the microphones 6a and 6b
together with the image data obtained by imaging performed by the
imaging system.
[0116] In addition, the audio input/output control section 28
performs a process of supplying the audio data obtained by the
audio input section 6 to the audio signal processing section
16.
[0117] Further, in the case where the temporary storage section 23
reads out data, the audio input/output control section 28 performs
a process of supplying audio data read out by the temporary storage
section 23 to the audio signal processing section 16 or the storage
section 25.
[0118] Still further, in the case where the storage section 25
reads out data, the audio input/output control section 28 performs
a process of supplying audio data read out by the storage section
25 to the audio signal processing section 16.
[0119] The audio signal processing section 16 is formed by a
digital signal processor, a D/A converter, and the like, for
example. Under control of the system controller 10, the audio
signal processing section 16 performs a process such as volume
control, tone control, or application of a sound effect on the
audio data supplied via the audio input/output control section 28.
Then, the audio signal processing section 16 converts the processed
audio data into an analog signal, and supplies the analog signal to
the audio output section 5. Note that the audio signal processing
section 16 is not limited to a unit that performs digital signal
processing, but may be a unit that performs signal processing using
an analog amplifier, an analog filter, or the like.
[0120] The audio output section 5 includes the pair of earphone
speakers 5a illustrated in FIG. 1 and an amplifier circuit for the
earphone speakers 5a.
[0121] The audio input section 6, the audio signal processing
section 16, and the audio output section 5 enable the user to
listen to an external sound, audio reproduced by the temporary
storage section 23, and audio reproduced by the storage section
25.
[0122] Note that the audio output section 5 may be formed by a
so-called bone conduction speaker.
[0123] The imaging/display apparatus 1 is provided with an
operation input section 20 for user operation.
[0124] The operation input section 20 may include an operation unit
such as a key, a dial, or the like, and be configured to detect a
user operation such as a key operation. Alternatively, the
operation input section 20 may be configured to detect a deliberate
behavior of the user.
[0125] In the case where the operation input section 20 is provided
with the operation unit, the operation input section 20 may be
provided with an operation unit for a replay operation, which will
be described later, an operation unit for an operation (e.g., the
zoom operation, signal processing, etc.) performed by the imaging
system, and the like, for example.
[0126] In the case where the operation input section 20 is
configured to detect a user behavior, the operation input section
20 may be provided with an acceleration sensor, an angular velocity
sensor, a vibration sensor, a pressure sensor, or the like.
[0127] For example, the user's act of tapping the imaging/display
apparatus 1 from the side may be detected with the acceleration
sensor, the vibration sensor, or the like. Thus, the system
controller 10 may determine that a user operation has occurred when
lateral acceleration has exceeded a predetermined value, for
example. For example, when the user has tapped the imaging/display
apparatus 1 once, the system controller 10 may determine that the
user has performed a replay start operation, whereas when the user
has tapped the imaging/display apparatus 1 twice, the system
controller 10 may determine that the user has performed a replay
termination operation. Moreover, the acceleration sensor, the
angular velocity sensor, or the like may be used to detect whether
the user has tapped a side (which corresponds to a sidepiece of
spectacles) of the imaging/display apparatus 1 from the right side
or from the left side, and the system controller 10 may regard each
of these acts of the user as a predetermined operation.
[0128] Further, the user's act of turning or shaking his or her
head may be detected with the acceleration sensor, the angular
velocity sensor, or the like. The system controller 10 may regard
each of these acts of the user as a user operation.
[0129] Still further, the pressure sensor may be provided on each
of left and right sides (which correspond to the sidepieces of the
spectacles) of the imaging/display apparatus 1, for example. Then,
the system controller 10 may determine that the user has performed
an operation for telephoto zooming when the user has pushed the
right side of the imaging/display apparatus 1, and determine that
the user has performed an operation for wide-angle zooming when the
user has pushed the left side of the imaging/display apparatus
1.
[0130] The operation input section 20, which acquires information
by functioning as the operation unit, the acceleration sensor, the
angular velocity sensor, the vibration sensor, the pressure sensor,
or the like as described above, supplies the acquired information
to the system controller 10, and the system controller 10 detects
the user operation based on the supplied information.
[0131] Next, the biological sensor 21 will now be described below.
The system controller 10 may recognize information detected by the
biological sensor 21 as an operation input by the user. One example
of deliberate behaviors of the user is a motion of the eyes (e.g.,
a change in the direction in which the eyes of the user are
directed, winking, etc.). A visual sensor, which will be described
later, may be used to detect winking. For example, the system
controller 10 may regard the user's act of winking three times as a
specific operation input.
[0132] A power operation unit may be provided for turning the power
of the imaging/display apparatus 1 on and off. Alternatively, the
system controller 10 may automatically turn the power on when the
biological sensor 21 has detected that the user has put on the
imaging/display apparatus 1, and automatically turn the power off
when the user has taken off the imaging/display apparatus 1.
[0133] The biological sensor 21 detects biological information
concerning the user. Examples of the biological information include
a pulse rate, a heart rate, electrocardiogram information,
electromyographic information, breathing information (e.g., a rate
of breathing, a depth of breathing, the amount of ventilation,
etc.), perspiration, galvanic skin response (GSR), blood pressure,
a saturation oxygen concentration in the blood, a skin surface
temperature, brain waves (e.g., information of alpha waves, beta
waves, theta waves, and delta waves), a blood flow change, and the
state of the eyes.
[0134] In order to detect the galvanic skin response, a body
temperature, the skin surface temperature, an electrocardiographic
response, the electromyographic information, heart beats, a pulse,
a blood flow, the blood pressure, the brain waves, the
perspiration, or the like, a sensor that is attached to an inside
of the wearing frame as illustrated in FIG. 1, for example, so as
to be in contact with the temporal region or the occipital region
of the user or a sensor that is separate from the wearing frame and
attached to another body part of the user may be used.
[0135] Further, an imaging section for imaging the skin of the user
may be used. This imaging section is a sensor capable of detecting
a change in skin color, for example.
[0136] A visual sensor including an imaging section that is
arranged near the display section 2, for example, to photograph the
eye of the user may be used as a sensor for detecting vision of the
user. In this case, an image of the eye of the user taken by this
imaging section may be subjected to image analysis to detect the
direction in which the eye is directed, a focal distance, the
degree of dilation of a pupil of the eye, a fundus pattern, opening
and closing of an eyelid, and the like. Further, a lighting section
that is arranged near the display section 2 to emit light to the
eye of the user and a light-receiving section for receiving the
light reflected from the eye may be used. In this case, it is
possible to detect the thickness of a crystalline lens of the user
based on the received reflected light, for example.
[0137] The biological sensor 21 supplies information of such
detection by such a sensor to the system controller 10.
[0138] The acceleration sensor, the angular velocity sensor, the
vibration sensor, and so on have been cited as the sensors that may
be included in the operation input section 20. Such sensors are
capable of detecting a motion of a body of the user, a motion of
the head, the center of gravity, the rhythm of walking/running, or
the like. In the above description of the operation input section
20, the deliberate behaviors (i.e., "operations" by the user) have
been mentioned as a motion to be detected by such sensors. However,
it may be so arranged that a behavior which the user does not
intend as an "operation", such as the motion of the body of the
user, the motion of the head, the center of gravity, or the rhythm
of walking/running, is detected by the acceleration sensor or the
like, and that such a behavior is treated as one piece of
biological information.
[0139] The imaging/display apparatus 1 further includes an audio
analysis section 24.
[0140] The audio analysis section 24 analyzes the audio data of the
external sound obtained by the audio input section 6. For example,
the audio analysis section 24 performs frequency analysis,
amplitude level evaluation, voiceprint analysis, or the like, and
supplies resultant analysis information to the system controller
10.
[0141] The imaging/display apparatus 1 further includes a time/date
calculation section 18 and a GPS receiver section 19.
[0142] The time/date calculation section 18 serves as a so-called
clock section to calculate a date and time (year, month, day, hour,
minute, second), and outputs information of the current date and
time to the system controller 10.
[0143] The GPS receiver section 19 receives a radio wave from a
global positioning system (GPS) satellite, and outputs information
of a latitude and longitude of a current location to the system
controller 10.
[0144] In the imaging/display apparatus 1, the imaging system
constantly performs imaging, and the image data obtained by imaging
is temporarily stored in the temporary storage section 23 in the
ring memory manner, and when the predetermined storage condition
has been satisfied, the system controller 10 extracts, from the
image data temporarily stored in the temporary storage section 23
at the time, image data to be stored and transfers the extracted
image data to the storage section 25 so as to be recorded therein
on a permanent basis.
[0145] Here, the image data to be stored on a permanent basis in
the storage section 25 is, among the images obtained by constant
imaging, an image that the system controller 10 has determined to
be an image of a scene that has some meaning for the user, such as
an image of a scene that has interested the user, an image of a
scene at a time when a change has occurred in the user's feelings,
an image of a scene that is likely to remain in the user's memory,
or an image of a scene that the user is likely to desire to see
again later.
[0146] The storage condition is satisfied when the system
controller 10 infers that such an image is currently stored in the
temporary storage section 23. For example, the system controller 10
determines that the storage condition has been satisfied in the
following cases.
[0147] First, an image that the user has caused to be replayed and
an image concerning which the user has issued an instruction
related to an operation of the imaging system, such as zoom
photographing, can be regarded as an image (of a scene) that has
interested the user. Therefore, the system controller 10 may
determine that the storage condition has been satisfied when
replaying has been performed in response to the user operation or
when an operation concerning the imaging system, such as zooming,
has been performed.
[0148] When the user has become interested in a scene within his or
her field of vision or when a change has occurred in his or her
feelings, corresponding biological information concerning the user
is obtained. Therefore, the system controller 10 may determine that
the storage condition has been satisfied when a predetermined
detection value has been obtained as the biological information.
For example, the system controller 10 may determine that the
storage condition has been satisfied when a predetermined situation
has been detected by the biological sensor 21. Examples of such
predetermined situations include: a situation in which the heart
rate has exceeded its normal range; a situation in which the
perspiration, the blood pressure, or the like has deviated from its
normal state; and a situation in which a line of sight of the user
has moved abruptly.
[0149] Further, it is possible to identify feelings of the user
based on the biological information. Accordingly, the system
controller 10 may determine that the storage condition has been
satisfied when the user is identified as being in any of the
following states of feelings, for example: cheerful, amused,
pleased, sad, frightened, calm, nostalgic, moved, alarmed, excited,
and tense.
[0150] Further, in the present embodiment, the external sound is
analyzed by the audio analysis section 24. Thus, the system
controller 10 may determine that the storage condition has been
satisfied when a loud sound has occurred suddenly or when a
situation in which loud cheers or the like are being shouted or
voice of a specific person has been detected.
[0151] As described above, the system controller 10 performs a
control process in accordance with the user operation inputted from
the operation input section 20, and a process of determining that
the storage condition has been satisfied based on the information
detected by the biological sensor 21 or the analysis information
obtained by the audio analysis section 24. When the storage
condition has been satisfied, the system controller 10 performs, as
the storage process, a process of controlling the storage section
25 to record the image data extracted from the temporary storage
section 23 on the storage medium. When performing the storage
process, the system controller 10 generates the metadata in
accordance with the storage condition, and adds the generated
metadata to the image data so that the generated metadata will be
recorded together with the image data.
[0152] The metadata includes at least information based on the
biological information. For example, a numerical value itself
detected as the biological information (e.g., the heart rate, a
respiratory rate, the blood pressure, the galvanic skin response,
the body temperature, or other numerical values as the detected
biological information) may be included in the metadata. Also,
feeling information (e.g., information such as "cheerful",
"pleased", or "sad" as mentioned above) of the user that can be
determined based on such a piece of biological information may be
included in the metadata.
[0153] Further, in addition to the information based on the
biological information, the information of the date and time
obtained by the time/date calculation section 18, the information
of the current location (i.e., the latitude and longitude thereof)
obtained by the GPS receiver section 19, information of a content
of the satisfied storage condition, identification information of
the user, or the like may also be included in the metadata.
[0154] The system controller 10 also performs a process of
generating the search condition based on the information detected
by the biological sensor 21. The search condition is a condition
used when performing a search for selecting an image to be played
back from among the image data stored in the storage section 25.
For example, the numerical value itself detected as the biological
information (e.g., the heart rate, the respiratory rate, the blood
pressure, the galvanic skin response, the body temperature, or
other numerical values as the detected biological information) may
be included in the search condition. Also, feeling information
(e.g., the information such as "cheerful", "pleased", or "sad" as
mentioned above) of the user that can be determined based on such a
piece of biological information may be included in the search
condition.
[0155] Further, in addition to the information based on the
biological information, the information of the date and time, the
information of the current location (i.e., the latitude and
longitude thereof), the information of the content of the storage
condition satisfied when performing the storage process, the
identification information of the user, or the like may also be
included in the search condition. That is, the search condition is
generated using information corresponding to the metadata generated
in the above-described manner.
[0156] The system controller 10 generates the search condition, for
example, when a change has occurred in the feelings of the user or
when a change in the numerical value of the biological information
has been detected, and causes the storage section 25 to perform the
search. The storage section 25 performs the search by collating the
search condition with the metadata added to the stored image data.
If a certain piece of image data is found as a result of the
search, the system controller 10 performs a process of playing back
and displaying this piece of image data on the display section
2.
[0157] Next, an exemplary structure of the imaging/display
apparatus 1 illustrated in FIG. 5 will now be described below. Note
that, in FIG. 5, components that have their counterparts in FIG. 4
are assigned the same reference numerals as those of their
counterparts in FIG. 4, and descriptions thereof will be omitted.
The structure of FIG. 5 is the same as the structure of FIG. 4
except that a communication section 26 is included in place of the
storage section 25.
[0158] The communication section 26 transmits and receives data to
and from the external device. The external device refers to a
device corresponding to the storage device 70 illustrated in FIG.
3B or the server apparatus 71 illustrated in FIG. 3C, and various
electronic devices, such as a computer device, the PDA, the mobile
phone, and the video storage device, are conceivable as the
external device. In particular, any device that has a capability to
store the image data transmitted by the imaging/display apparatus 1
via the communication section 26 in a storage medium such as an
HDD, a capability to search for the image data using the search
condition transmitted from the imaging/display apparatus 1, and a
capability to transmit the image data found as a result of a search
to the imaging/display apparatus 1 can function as the external
device.
[0159] The communication section 26 may be configured to perform
network communication via short-range wireless communication for a
network access point, for example, in accordance with a system such
as a wireless LAN, Bluetooth, or the like. Alternatively, the
communication section 26 may perform wireless communication
directly with the external device having a corresponding
communication capability.
[0160] The image data (and the audio data) read from the temporary
storage section 23 is supplied to the communication section 26 via
the image input/output control section 27 (and the audio
input/output control section 28). In addition, the metadata is
supplied from the system controller 10 to the communication section
26.
[0161] Under control of the system controller 10, the communication
section 26 adds the metadata to the supplied image data (and the
supplied audio data), encodes and modulates resultant data for
transmission, and transmits resultant data to the external
device.
[0162] For example, when the storage condition has been satisfied,
as in the case of the structure of FIG. 4 described above, the
system controller 10 performs, as the storage process, a process of
causing the image data to be stored to be transmitted, together
with the metadata, to the external device via the communication
section 26. Upon receipt of the image data and the metadata
transmitted from the communication section 26, the external device
(e.g., the storage device 70 or the server apparatus 71) records
the received image data and metadata on the storage medium.
[0163] In addition, when the system controller 10 has generated the
search condition, as in the case of the structure of FIG. 4
described above, the communication section 26 transmits the
generated search condition to the external device. Upon receipt of
the search condition, the external device performs the search using
the received search condition. If a certain piece of image data is
found as a result of the search, the external device transmits this
piece of image data to the imaging/display apparatus 1.
[0164] The communication section 26 receives the image data found
as a result of the search performed in the external device and
transmitted from the external device. The received image data is
supplied to the display image processing section 12 via the image
input/output control section 27 and displayed on the display
section 2.
[0165] In the case of the exemplary structure of FIG. 4, the system
controller 10 performs, as the storage process for storing the
image temporarily stored in the temporary storage section 23 on a
permanent basis, the process of controlling the storage section 25
to record the image therein. In contrast, in the case of the
structure of FIG. 5, the system controller 10 performs, as the
storage process for storing the image temporarily stored in the
temporary storage section 23 on a permanent basis, the process of
controlling the communication section 26 to transmit the image data
to the external device.
[0166] That is, with the exemplary structure of FIG. 5, the
imaging/display apparatus 1 does not have a capability to store
data on a permanent basis within itself. The exemplary structure of
FIG. 5 depends on the assumption that the image data is stored in
the external device.
[0167] Next, FIG. 6 illustrates an exemplary structure of the
imaging/display apparatus 1 in which the imaging apparatus section
40 and the display apparatus section 30 are provided separately as
illustrated in FIG. 2.
[0168] In the case of this structure, the imaging apparatus section
40 includes the system controller 10, the imaging system (i.e., the
imaging section 3, the imaging signal processing section 15, and
the imaging control section 11), the temporary storage section 23,
the storage section 25, the operation input section 20, the
biological sensor 21, the image input/output control section 27,
the audio input section 6, and the audio analysis section 24.
[0169] However, the imaging apparatus section 40 does not include
the audio input/output control section 28, the audio signal
processing section 16, or the audio output section 5. Thus, a
signal of the external sound obtained by the audio input section 6
is used only in a process performed by the audio analysis section
24, i.e., determination concerning the storage condition,
generation of the metadata, and generation of the search condition.
Therefore, in this exemplary structure, the audio data is neither
temporarily stored in the temporary storage section 23 nor stored
in the storage section 25.
[0170] The image input/output control section 27 performs a process
of transferring the image data constantly obtained by the imaging
system to the temporary storage section 23. The image input/output
control section 27 also performs a process of transferring the
image data read from the temporary storage section 23 to the
communication section 26 or the storage section 25. The image
input/output control section 27 also performs a process of
transferring the image data read from the storage section 25 to the
communication section 26.
[0171] The communication section 26 performs an encoding process on
the supplied image data for transmission to the display apparatus
section 30. Then, the communication section 26 transmits the
encoded image data to the display apparatus section 30.
[0172] The display apparatus section 30 includes a communication
section 34, a display section 31, a display image processing
section 32, a display driving section 33, a display control section
35, and an operation section 36.
[0173] The communication section 34 performs data communication
with the communication section 26 in the imaging apparatus section
40. The communication section 34 receives the image data
transmitted from the imaging apparatus section 40, and performs a
decoding process on the received image data.
[0174] The image data decoded by the communication section 34 is
supplied to the display image processing section 32. The display
image processing section 32 performs signal processing for
displaying the image data, the split screen process, synthesis of
the character image, or the like.
[0175] The display driving section 33 is formed by a pixel driving
circuit for allowing an image signal supplied from the display
image processing section 32 to be displayed on the display section
(e.g., a liquid crystal display) 31. That is, the display driving
section 33 applies driving signals based on a video signal to
pixels arranged in a matrix in the display section 31 with
specified horizontal/vertical driving timing for displaying.
[0176] The display control section 35 controls a processing
operation of the display image processing section 32 and an
operation of the display driving section 33. For example, in
accordance with a user operation inputted from the operation
section 36, the display control section 35 controls activation and
deactivation of a display operation, switching of the form of an
area on a screen, or the like.
[0177] In the case where instruction information is transmitted
from the system controller 10 via communication between the
communication sections 26 and 34, the display control section 35
may control the activation and deactivation of the display
operation, switching of the form of the area on the screen, or the
like in accordance with the instruction information transmitted
from the system controller 10.
[0178] While the exemplary structures of the imaging/display
apparatus 1 have been described above with reference to FIGS. 4, 5,
and 6, various other structures of the imaging/display apparatus 1
are possible.
[0179] For example, the structures of FIGS. 4 and 5 may be modified
so as not to include a system for recording or outputting the audio
data (i.e., the audio input/output control section 28, the audio
signal processing section 16, and/or the audio output section 5) as
is the case with the structure of FIG. 6. Conversely, the structure
of FIG. 6 may be modified so as to additionally include the system
for recording or outputting the audio data.
[0180] The imaging/display apparatus 1 need not include the audio
analysis section 24. The imaging/display apparatus 1 need not
include the time/date calculation section 18. The imaging/display
apparatus 1 need not include the GPS receiver section 19.
[0181] The structure of FIG. 6 may be modified so as to include the
communication section 26 for transmitting the image data to be
stored to the external device in place of the storage section
25.
[0182] The imaging/display apparatus may include both the storage
section 25 and the communication section 26.
3. Exemplary Photographed Images
[0183] Here, with reference to FIGS. 7A to 10B, examples of the
image data that are obtained by imaging performed by the imaging
system and temporarily stored in the temporary storage section 23
will now be described below.
[0184] However, FIG. 7A illustrates not an image obtained by
imaging but an example of a scene that the user sees when the
entire screen of the display section 2 as illustrated in FIG. 1 is
in the see-through state. At this time, the display section 2 is in
a state of being simply a transparent plate, and the user is
viewing the scene within his or her field of vision through the
transparent display section 2.
[0185] It is assumed that when the imaging section 3 photographs in
a regular manner, an image equivalent to a scene that the user
would see if the display section 2 were in the see-through state is
obtained by imaging.
[0186] FIG. 7B illustrates an example of image data that is
obtained when the imaging system photographs in the regular manner.
This image data represents a scene that is nearly the same as the
scene that the user would see if the display section 2 were in the
see-through state. While the imaging system is imaging in the
regular manner, images representing scenes that are approximately
equivalent to scenes that the user would regularly see are obtained
by imaging as described above, and image data of such images are
sequentially stored in the temporary storage section 23.
[0187] FIG. 7C illustrates an example of an image obtained by
imaging when the system controller 10 has instructed the imaging
control section 11 to control the imaging section 3 to take a
telephoto in accordance with the user operation via the operation
input section 20. Taking a telephoto allows image data of such a
telephoto image to be obtained so that the image data of the
telephoto image can be stored in the temporary storage section
23.
[0188] Note that although the example of telephotography has been
described above, the imaging section 3 may be caused to perform
wide-angle zooming to obtain image data of a wide-angle image of a
scene nearby.
[0189] Acquisition by the imaging system of the
telephoto/wide-angle images is achieved by drive control of the
zoom lens in the imaging section 3 as well as by signal processing
in the imaging signal processing section 15.
[0190] Although not shown in the figures, the system controller 10
may instruct the imaging section 3 to perform adjustment of a focal
point instead of the so-called zoom operation to take an image of a
scene nearby or a distant scene so that image data thereof can be
stored in the temporary storage section 23.
[0191] FIG. 8A illustrates an example of an image obtained by
regular imaging, and FIG. 8B illustrates an example of a magnified
image.
[0192] In accordance with the user operation, for example, the
system controller 10 may instruct the imaging signal processing
section 15 to perform a magnification process on the image obtained
from the imaging section 3 to obtain image data of the magnified
image as illustrated in FIG. 8B, and store the image data of the
magnified image in the temporary storage section 23. Although not
shown in the figures, the system controller 10 is also capable of
instructing the imaging signal processing section 15 to perform a
reduction process on the image, and storing a resultant reduced
image in the temporary storage section 23.
[0193] FIG. 9A illustrates an image obtained by regular imaging
when the user is in a dark room where a child is sleeping, for
example. Because the user is in the dark room, this image obtained
by regular imaging does not show the child and so on clearly.
[0194] In this case, the system controller 10 is able to instruct
the imaging control section 11 (i.e., the imaging section 3 or the
imaging signal processing section 15) to increase infrared imaging
sensitivity to obtain image data of an infrared image as
illustrated in FIG. 9B, in which a face of the child sleeping in
the dark room and so on are recognizable.
[0195] FIG. 10A illustrates an image obtained by regular imaging.
In this case, the system controller 10 is able to instruct the
imaging control section 11 (i.e., the imaging section 3 or the
imaging signal processing section 15) to increase ultraviolet
imaging sensitivity to obtain image data of an image as illustrated
in FIG. 10B in which ultraviolet components are shown.
[0196] The imaging system is capable of obtaining the above types
of image data including the image data of the image obtained by
regular imaging. The image data in various states obtained by the
imaging system are temporarily stored in the temporary storage
section 23.
[0197] Needless to say, the image data that can be obtained by the
imaging system is not limited to the above types of image data.
Various other types of image data can also be obtained in various
imaging modes by controlling processes performed by and the
operations of the imaging section 3 and the imaging signal
processing section 15.
[0198] A great variety of forms of photographed images are
conceivable, such as: a telephoto image; a wide-angle image; an
image photographed while zooming in or zooming out is performed
within a range between a telephoto extreme and a wide-angle
extreme; a magnified photographed image; a reduced photographed
image; an image photographed with a varied frame rate (e.g.,
photographed with a high frame rate, or photographed with a low
frame rate); a photographed image with increased brightness; a
photographed image with reduced brightness; a photographed image
with varied contrast; a photographed image with varied sharpness;
an image photographed with increased imaging sensitivity; an image
photographed with increased infrared imaging sensitivity; an image
photographed with increased ultraviolet imaging sensitivity; an
image photographed with a particular wavelength range cut off;
effect-applied photographed images, such as a mosaic image, a
brightness-reversed image, a soft-focus image, an image with a part
of the image highlighted, and an image with varied overall color
atmosphere; and a still photographed image.
4. Replay Operation
[0199] The replay operation will now be described below. Note that
the replay operation described herein refers to a replay operation
using the image data stored in the temporary storage section 23
instead of the image data stored in the storage section 25 or the
external device.
[0200] In the present embodiment, the image data obtained by
constant imaging is stored in the temporary storage section 23, and
the image data stored in the temporary storage section 23 can be
used to present a replay display of a scene in the recent past.
That is, it is possible to replay a past scene within a range of
the image data stored in the temporary storage section 23.
[0201] In the case where the user witnessed a traffic accident by
chance, for example, the user is able to watch a replay image of a
scene of the accident. Further, while watching a sport game, the
user is able to watch a replay image of a play in the immediate
past.
[0202] Examples of displays presented on the display section 2 at
the time of the replay operation are illustrated in FIGS. 12A and
12B. At the time of the replay operation, the system controller 10
allows the image data to be read from the temporary storage section
23, and allows the read image data to be supplied to the display
image processing section 12 via the image input/output control
section 27. Then, the display image processing section 12 splits
the screen of the display section 2, for example, and causes the
replay image (i.e., the read image data) to be displayed on a part
of the screen.
[0203] FIG. 12A illustrates an exemplary case where the display
image processing section 12 has set a child-screen area AR2 within
a parent-screen area AR1, and is displaying the replay image in the
area AR2 while allowing the area AR1 to stay in the see-through
state. In this case, the user is able to watch a replay image of a
past scene with the area AR2 while seeing a current scene with the
area AR1, which is in the see-through state.
[0204] FIG. 12B illustrates an exemplary case where the display
image processing section 12 has split the screen into upper and
lower areas AR1 and AR2, and is displaying the replay image in the
area AR2 while allowing the area AR1 to stay in the see-through
state. In this case also, the user is able to watch the replay
image of the past scene with the area AR2 while seeing the current
scene with the area AR1, which is in the see-through state.
[0205] As described above, the screen of the display section 2 is
divided into the parent and child screens or split into two parts,
for example, and the display of the replay image is presented while
a part of the screen of the display section 2 is caused to stay in
the see-through state. Needless to say, the position of the child
screen within the screen and the size of the child screen may be
changeable in accordance with the user operation. It is also
possible that the screen of the display section 2 be split into
left and right areas AR1 and AR2. It is also possible that the area
of the area AR1 and the area of the area AR2 be set to be unequal
by the user operation. Further, switching of display positions may
be carried out in accordance with the user operation. For example,
the parent screen and the child screen may be replaced by each
other. Also, the areas obtained by screen splitting may be replaced
by each other.
[0206] Further, instead of being caused to enter the see-through
state, the area AR1 may be employed to display the image that is
currently obtained by imaging by the imaging system (i.e., the
image of the current scene, which is equivalent to the scene that
the user would see if the screen were in the see-through
state).
[0207] Still further, at the time of replaying, the replay image
may be displayed on the entire screen of the display section 2.
That is, the entire screen may be shifted from the see-through
state (or a state in which the image obtained by regular imaging is
being displayed) to a state in which the replay image is
displayed.
[0208] Exemplary control exercised by the system controller 10 over
the temporary storage section 23 when a request for replaying has
been issued by the user operation will now be described below with
reference to FIG. 11B.
[0209] FIG. 11B illustrates the storage area of the temporary
storage section 23. As noted previously, the image data obtained by
the imaging system constantly is sequentially stored between the
top address AdST and the end address AdED of the storage area in
the ring memory manner, while the write address pointer W-Ad is
moving.
[0210] Suppose that the user has performed an operation to issue
the request for replaying when the write address pointer W-Ad is
pointing to an address AdX and the image data is being stored at
the address AdX.
[0211] In this case, the system controller 10 continues
incrementing of the write address pointer W-Ad and writing of the
image data obtained by imaging, leaving the write address pointer
W-Ad to continue to move forward from the address AdX. At the same
time, the system controller 10 causes a read address pointer R-Ad
to move backward from the address AdX. The temporary storage
section 23 performs a process of reading image data at an address
indicated by the read address pointer R-Ad, and the read image data
is supplied to the display image processing section 12 via the
image input/output control section 27 to be displayed in the area
AR2 of the display section 2 as illustrated in FIGS. 12A and 12B,
for example.
[0212] At this time, the speed of the read address pointer R-Ad may
be changed (decremented) to 1.5 times speed, double speed, triple
speed, or the like, so that images will be displayed on the display
section 2 from the current scene to progressively earlier scenes in
a fast reverse mode.
[0213] While watching the images played back in the fast reverse
mode, the user, searching for the top of a scene that the user
desires to replay, performs the replay start operation at a point
at which the user desires replaying to start.
[0214] Suppose, for example, that the user has performed the replay
start operation at a time when the read address pointer R-Ad has
moved backward up to an address AdY. In this case, at the time when
the user has performed the replay start operation, the system
controller 10 causes the read address pointer R-Ad to start to be
incremented in a normal moving direction and at a normal speed. As
a result, the temporary storage section 23 starts to read the image
data at the normal speed, starting with the address AdY, and
accordingly, a series of replay images beginning with one stored at
the address AdY starts to be displayed on the display section
2.
[0215] Suppose, for example, that the user has thereafter performed
the replay termination operation when the read address pointer R-Ad
has reached an address AdZ. In this case, the system controller 10
terminates replaying at this time. That is, the system controller
10 issues an instruction to terminate the reading in the temporary
storage section 23 and the displaying of the replay images in the
display section 2.
[0216] In this case, the images stored between the addresses AdY
and AdZ correspond to the replay images which the user has desired
to watch again.
[0217] While a simple example has been described above for purposes
of illustration, it may happen that while the user is watching the
images played back in the fast reverse mode, searching for the
point at which the user desires replaying to start, fast reversing
goes too far, and the user accordingly desires to fast-forward
conversely. Also, the user may desire to watch the replay images
not at the normal speed but at a low speed. Also, the user may
desire to pause replaying or play some or all of the replay images
repeatedly. Therefore, it is preferable that it be possible to
change whether the read address pointer R-Ad is incremented or
decremented in accordance with the user operation. It is also
preferable that it be possible to change the rate of incrementing
or decrementing or to pause replaying in accordance with the user
operation.
[0218] Regarding the user operation, an operation unit related to
replaying may be provided. Also, the system controller 10 may
recognize a user behavior detected by the acceleration sensor or
the like as the user operation, for example.
[0219] For example, the system controller 10 may regard the user's
act of tapping the imaging/display apparatus 1 twice as the replay
request operation and the replay termination operation. Also, the
system controller 10 may regard the user's act of tapping the
imaging/display apparatus 1 once as the replay start operation, for
example.
[0220] Further, the user's act of tapping the imaging/display
apparatus 1 from the left or right side may be regarded as an
operation for fast-forwarding, an operation for fast-reversing, an
operation for fast playback, an operation for slow playback, or the
like. For example, the user's act of tapping the right side of the
imaging/display apparatus 1 may be regarded as an operation for
forward playback, and the user's act of tapping the left side of
the imaging/display apparatus 1 as an operation for reverse
playback, and the speed (slow, normal, double, etc.) of playback
may be determined based on the number of times of tapping.
[0221] Needless to say, various other examples are conceivable. For
example, the user's act of shaking his or her head may be regarded
as an operation for fast-forwarding/fast-reversing.
[0222] In FIG. 11B, the user searches first the image of the
current scene and then the images of the progressively earlier
scenes to find the point at which the user desires replaying to be
started. However, it may be so arranged that, when the user has
performed a replay operation, replaying is started with an image of
a scene a specified time ago. Also, it may be so arranged that the
user is able to specify, by the number of times of tapping or the
like, the point at which replaying is started, such as thirty
seconds ago, one minute ago, three minutes ago, five minutes ago,
and so on.
5. Exemplary Manners for Selecting Image to be Stored
[0223] As described above, the system controller 10 makes a
determination concerning the storage condition and, when the
storage condition has been satisfied, extracts the image data from
the temporary storage section 23 to perform the storage process
(i.e., the recording of the extracted image data in the storage
section 25, or the transmission of the extracted image data to the
external device via the communication section 26). Here, examples
of the storage condition and the range of images that are extracted
in response to the satisfaction of such storage conditions will be
described.
Execution of Replaying
[0224] The user will perform the replay operation when the user has
become interested in a scene that he or she viewed in the immediate
past or when he or she desires to view the scene again for some
reason. Therefore, the system controller 10 may determine that the
storage condition has been satisfied when the replay operation has
been performed in accordance with the user operation. In this case,
the images that the user has watched again during the replay
operation may be extracted as images to be stored. In the case of
the example of FIG. 11B described above, for example, the image
data stored in the range between the addresses AdY and AdZ is
extracted as the images to be stored.
Operation Related to Imaging System
[0225] The user is able to perform an operation to instruct the
imaging system to perform telephoto/wide-angle zooming, image
magnification/reduction, imaging with increased infrared
sensitivity, imaging with increased ultraviolet sensitivity, change
of the frame rate, application of the image effect, or the like. It
is reasonable to think that the user performs such an operation
when he or she is viewing a scene that interests him or her.
Therefore, the system controller 10 may determine that the storage
condition has been satisfied when the user has performed such an
operation related to the imaging system, so that an operation such
as the telephoto/wide-angle zooming or the like or signal
processing has been performed in the imaging system. In this case,
image data obtained during a period in which the operation such as
the telephoto/wide-angle zooming or the like or the signal
processing continues to be performed may be extracted as images to
be stored.
Determination Based on Biological Information (Occurrence of
Biological Trigger)
[0226] The biological information detected by the biological sensor
21 can be used to detect a change in the situation of the user or
to infer the feelings of the user. Thus, the system controller 10
may determine that the storage condition has been satisfied when it
is determined that a predetermined situation has occurred based on
the change in the numerical value detected as the biological
information or the feelings of the user inferred from the
biological information.
[0227] For example, the system controller 10 may determine that the
storage condition has been satisfied when the change in the
numerical value of the biological information that accompanies
occurrence of a tense state, an excited state, a comfortable state,
or the like is detected, or when a feeling (e.g., cheerful, amused,
pleased, sad, frightened, calm, nostalgic, moved, alarmed, excited,
tense, etc.) of the user has been inferred based on the biological
information.
[0228] This is because a scene that the user is viewing at such a
time can be considered as giving some stimulus to senses of the
user. A reaction of the user indicated by the biological
information detected when he or she has got excited or gone wild
with excitement while watching a sport game, when he or she has
witnessed an interesting scene, when he or she has been moved, when
he or she has witnessed a traffic accident or the like, when he or
she has encountered a favorite person, celebrity, or the like, when
he or she has got uneasy or afraid, or when he or she has been
surprised, for example, may be regarded as a trigger for the
satisfaction of the storage condition. This makes it possible to
store a scene important for the user.
[0229] In the case where the user has been excited continuously for
a certain period, image data obtained during that period, in which
biological information indicating that the user is in the excited
state continues to be obtained, may be extracted as images to be
stored. In the case where biological information indicating that
the user has been surprised for a moment has been obtained, image
data obtained during a certain period around that moment may be
extracted as images to be stored.
[0230] While the excited state can be detected by the biological
information concerning the user, such as the brain waves, the blood
pressure, or the galvanic skin response, the state of the user can
also be determined based on the state of the pupil or movement of
the line of sight detected by the visual sensor. Moreover, the
motion of the body of the user detected by the acceleration sensor,
the vibration sensor, or the like may be used to determine the
state of the user.
[0231] Increase of the pulse or the like is sometimes caused by
tension or excitement and other times by exercise such as running.
In order to discriminate between such causes, information obtained
by the acceleration sensor or the like may be additionally referred
to.
Determination Based on External Sound (Occurrence of Sound
Trigger)
[0232] The system controller 10 may determine that the storage
condition has been satisfied when the audio analysis section 24,
while analyzing the external sound, has detected occurrence of a
very loud sound, such as a great cheer, a sound of an accident, or
a warning sound, or occurrence or a specific sound. That is, the
detection of the occurrence of such a sound may be regarded as a
trigger for the satisfaction of the storage condition.
[0233] It may be so arranged that a specific sound, such as a voice
of an acquaintance, an electronic sound, a cry of a pet, or a
natural sound, is registered in the audio analysis section 24, and
that the system controller 10 determines that the storage condition
has been satisfied when the registered specific sound has been
detected.
[0234] In such cases also, image data obtained during a period in
which such a sound that causes the storage condition to be
satisfied continues to be detected or image data obtained during a
certain period around a time at which it is determined that the
storage condition has been satisfied may be extracted as images to
be stored.
[0235] Examples of the determination of the scene that has
interested the user or the scene that the user desires to watch
again, i.e., the determination concerning the storage condition,
have been cited above. It should be noted, however, that there are
many other manners for making a determination concerning the
storage condition.
[0236] Note that, in the present embodiment, the image data
obtained by imaging is constantly stored in the temporary storage
section 23 once, and then, when the storage condition has been
satisfied, the process of extracting the image data from the
temporary storage section 23 and storing the extracted image data
on a permanent basis is performed. Therefore, the determination
concerning the storage condition is performed in the present
embodiment. However, in the case where the storage section 25 or
the external device has so large a storage capacity that every
piece of image data obtained by imaging can be stored on a
permanent basis, there is no need to perform the process of
determining whether the storage condition has been satisfied to
store the extracted image data.
6. Imaging and Exemplary Storage Procedure
[0237] Exemplary control procedures performed by the system
controller 10 for achieving imaging and storage of the image data
in the imaging/display apparatus 1 according to the present
embodiment will now be described below. It is assumed here that the
imaging/display apparatus 1 that has the display section 2 arranged
in front of the eyes of the user as illustrated in FIG. 1 is
used.
[0238] FIG. 13 illustrates a procedure performed by the system
controller 10 between turn-on and turn-off of the imaging/display
apparatus 1, for example. This procedure may also be considered as
a procedure to be performed between a time when the user has
performed an operation for starting the operation after the turn-on
of the imaging/display apparatus 1 and a time when the user has
performed an operation for terminating the operation.
[0239] In the case where the operation is started as a result of
the turn-on of the imaging/display apparatus 1 or the like, the
system controller 10 first starts imaging and the temporary storage
of the image data obtained by imaging at step F101. Specifically,
the system controller 10 controls the imaging system to start a
regular imaging operation, allows the image data obtained by
imaging to be supplied to the temporary storage section 23, and
controls the temporary storage section 23 to start a storage
operation in the ring memory manner.
[0240] Thereafter, this imaging and the storage of the image data
obtained by imaging in the temporary storage section 23 are
continued until the operation is terminated as a result of the
turn-off of the imaging/display apparatus 1 or the like.
[0241] At step F102, the system controller 10 instructs the display
control section 14 to cause the entire screen of the display
section 2 to enter the see-through state.
[0242] As a result of the processes of steps F101 and F102, the
user becomes able to view the scene within his or her field of
vision in a regular manner through the display section 2, which is
in the see-through state, and the scenes within the user's field of
vision are constantly imaged and temporarily stored.
[0243] After the operation is started in the above-described
manner, the system controller 10 performs monitoring processes in a
monitoring process loop at steps F103, F104, F105, F106, F107, and
F108.
[0244] At step F103, the system controller 10 monitors whether the
user has performed an operation for requesting replaying.
[0245] At step F104, the system controller 10 monitors whether the
user has performed an imaging-related operation. The term
"imaging-related operation" as used herein refers to an operation
for switching the image data obtained by the imaging system from
the image data obtained by regular imaging to another type of image
data. Examples of such operations include a telephoto/wide-angle
zoom operation, an operation for image magnification/reduction, an
operation for adjusting the imaging frame rate, an operation for
changing the imaging sensitivity, an operation for increasing the
infrared imaging sensitivity, an operation for increasing the
ultraviolet imaging sensitivity, and an operation for image
processing such as the application of the image effect. Further, an
operation for shifting the state of imaging from such a non-regular
state back to a regular state is also one example of the
imaging-related operations.
[0246] At step F105, the system controller 10 monitors based on the
information obtained from the audio analysis section 24 whether a
sound trigger (the satisfaction of the storage condition as a
result of audio analysis) has occurred.
[0247] At step F106, the system controller 10 monitors based on the
information obtained from the biological sensor 21 whether a
biological trigger (i.e., the satisfaction of the storage condition
based on the biological information) has occurred.
[0248] At step F107, the system controller 10 determines whether an
internal storage flag is on.
[0249] At step F108, the system controller 10 monitors whether the
operation should be terminated as a result of a turn-off operation
or the operation for terminating the operation being performed by
the user, for example.
[0250] When the user has performed the replay request operation,
the system controller 10 proceeds from step F103 to step F109, and
performs a replay procedure. This replay procedure is a procedure
for executing the operation described above with reference to FIG.
11B, and is illustrated in FIG. 14.
[0251] First, at step F201, the system controller 10 performs
control for starting the displaying of the images in the fast
reverse mode. Specifically, the system controller 10 controls the
temporary storage section 23 to read the image data while
decrementing the read address pointer R-Ad so as to move backward
from a current location of the write address pointer W-Ad
approximately at the double speed, for example. In addition, the
system controller 10 instructs the display control section 14 to
allow the image data read from the temporary storage section 23 to
be displayed on a part of the screen, such as the area AR2 as
illustrated in FIG. 12A or 11B. Note that the images played back in
the fast reverse mode may be displayed on the entire screen of the
display section 2.
[0252] As a result of the process of step F201, the user becomes
able to watch the images played back in the fast reverse mode
(i.e., the image of the current scene and the images of the
progressively earlier scenes). While watching the images played
back in the fast reverse mode, the user searches for a start point
of the scene that the user desires to watch again, and performs the
replay start operation at the start point.
[0253] Upon detection of the replay start operation, the system
controller 10 proceeds from step F202 to step F203, and performs
control for starting the displaying of the replay images.
Specifically, the system controller 10 controls the temporary
storage section 23 to change the mode of the read address pointer
R-Ad so that the read address pointer R-Ad starts to be incremented
(i.e., move in the normal direction in which time progresses) at
the normal speed, and read the image data. As a result, the replay
images are played back in a normal manner and displayed on the
display section 2, and the user becomes able to watch the scene in
the recent past again. Note that although not shown in FIG. 14, at
this time, the replay images may be played back at a low speed or
played back at a high speed such as the 1.5 times speed in
accordance with the user operation.
[0254] When it is detected thereafter that the user has performed
the replay termination operation or when replaying has been
completed thereafter, the system controller 10 proceeds from step
F204 to step F205. Replaying is completed, for example, when
replaying has progressed so far as to reach the image that was
obtained at the time when the user performed the replay request
operation (i.e., the address that was indicated by the write
address pointer W-Ad at the time when the user requested replaying)
or when, in the case where replaying is performed at a high speed,
replaying has progressed still further to reach an image that is
obtained at the current time (i.e., an address that is indicated by
the write address pointer W-Ad at the current time).
[0255] When the replay termination operation has been performed or
when replaying has been completed, the system controller 10
performs a replay termination process at step F205. Specifically,
the system controller 10 controls the temporary storage section 23
to terminate the reading of the image data, and instructs the
display control section 14 to return the entire screen of the
display section 2 to the see-through state.
[0256] As noted previously, the system controller 10 regards the
performance of the replay operation as the satisfaction of the
storage condition. Therefore, at step F206, the system controller
10 determines a storage range of the image data to be stored as an
accompaniment to the performance of the replay operation. In
addition, the system controller 10 generates the metadata. This
metadata includes information based on the biological information
detected at the time of replaying, information indicating that the
image data to be stored has been determined because of the
performance of the replay operation, and so on. A process of
generating the metadata will be described later with reference to
FIG. 16.
[0257] Further, the system controller 10 turns on the storage
flag.
[0258] After performing the above processes, the system controller
10 returns to the monitoring loop at steps F103 to F108 in FIG.
13.
[0259] Immediately after the above replay procedure is performed,
the system controller 10 proceeds from step F107 to step F113 in
FIG. 13 because the storage flag has been turned on. Then, at step
F113, the system controller 10 performs the storage process.
[0260] As the storage process, the system controller 10 controls
the storage section 25 to record the image data within the storage
range and the metadata on the storage medium. That is, in this
case, the system controller 10 allows the image data within the
storage range determined at step F206 in FIG. 14, e.g., the image
data that was displayed in replaying, to be transferred from the
temporary storage section 23 to the storage section 25, and
transmits the metadata generated at step F206 to the storage
section 25. Then, the system controller 10 controls the storage
section 25 to add the metadata to the image data, encode the
resultant data for recording, and record the resultant data on the
storage medium.
[0261] Note that the above control is performed in the case where
the imaging/display apparatus 1 includes the storage section 25 as
illustrated in FIG. 4, and that in the case where the
imaging/display apparatus 1 includes the communication section 26
in place of the storage section 25 as illustrated in FIG. 5, the
system controller 10 performs control for allowing the image data
within the storage range and the metadata to be encoded for
transmission, and transmitting the resultant data to the external
device (e.g., the storage device 70, the server apparatus 71, etc.)
via the communication section 26.
[0262] After the storage process at step F113 is completed, the
system controller 10 turns off the storage flag at step F114 and
returns to the monitoring loop at steps F103 to F108.
[0263] When the user has performed the imaging-related operation,
the system controller 10 proceeds from step F104 to step F110, and
performs an imaging system control procedure. The imaging system
control procedure is illustrated in FIG. 15.
[0264] First, at step F301, the system controller 10 performs
control related to the imaging system in accordance with the
imaging-related operation. That is, the system controller 10 causes
an operation requested by the user to be performed.
[0265] In the case where the user has performed the telephoto or
wide-angle zoom operation, for example, the system controller 10
instructs the imaging control section 11 to perform the zoom
operation and drive the zoom lens in the imaging section 3.
[0266] In the case where the user has performed the operation for
image magnification or reduction, the system controller 10
instructs the imaging control section 11 to perform an image
magnification or reduction process, thereby causing the imaging
signal processing section 15 to perform the magnification process
or the reduction process on the photographed image data.
[0267] In the case where the user has performed the operation for
adjusting the imaging frame rate, the system controller 10
instructs the imaging control section 11 to change the frame rate,
thereby changing the frame rate in the imaging section 3 and the
imaging signal processing section 15.
[0268] In the case where the user has performed the operation for
changing the imaging sensitivity, the operation for increasing the
infrared imaging sensitivity, or the operation for increasing the
ultraviolet imaging sensitivity, the system controller 10 instructs
the imaging control section 11 to change the imaging sensitivity,
thereby changing sensitivity of signals obtained from an imaging
device in the imaging section 3 (e.g., changing a gain of the
signals read from the imaging device).
[0269] In the case where the user has performed the operation for
the application of the image effect, the system controller 10
instructs the imaging control section 11 to perform an effect
process on the image, thereby causing the imaging signal processing
section 15 to perform the image effect process on the photographed
image data.
[0270] In the case where the user has performed an operation for
shifting the state of imaging from any of the non-regular states
such as, a telephoto/wide-angle zoom state, an image
magnification/reduction state, a frame rate changing state, a
sensitivity changing state, or an image effect state, back to the
regular state, the system controller 10 instructs the imaging
control section 11 to perform regular imaging, thereby returning
the operations of the imaging section 3 and the imaging signal
processing section 15 to the state in which regular imaging is
performed.
[0271] As described above, the system controller 10 controls the
operation of the imaging system in accordance with the
imaging-related operation performed by the user. When, at step
F301, the system controller 10 has controlled the operation of the
imaging system, which has been imaging in the regular manner, so
that the telephoto/wide-angle zooming, the image
magnification/reduction, the change of the frame rate, the change
of the sensitivity, the application of the image effect, or the
like is performed, the system controller 10 proceeds from step F302
to step F303, and, in this case, the system controller 10 causes
the photographed image to be displayed on the display section 2.
That is, the system controller 10 controls the image input/output
control section 27 to supply the image data obtained from the
imaging system to the display image processing section 12 while
continuing to supply the image data obtained from the imaging
system to the temporary storage section 23 as before, and also
instructs the display control section 14 to display the
photographed image data.
[0272] Then, the system controller 10 returns to the monitoring
loop at steps F103 to F108.
[0273] Hereinafter, in order to facilitate description, a state of
the imaging operation in which the telephoto/wide-angle zooming,
the image magnification/reduction, the change of the frame rate,
the change of the sensitivity, the application of the image effect,
or the like is performed will be referred to as a "special imaging
state", and thus be differentiated from a regular imaging state.
The regular imaging state refers to a state of the imaging
operation in which the image equivalent to the image viewed through
the display section 2 in the see-through state is obtained, as
illustrated in FIG. 7B.
[0274] When the user has performed an operation for shifting the
state of the imaging operation to the special imaging state, such
as the zoom operation, the process of step F303 causes the display
section 2 to switch from the see-through state to a state in which
a zoom image or the like is displayed, so that the user becomes
able to view the photographed image. That is, if the user performs
the operation for the telephoto/wide-angle zooming, the image
magnification/reduction, the change of the frame rate, the change
of the imaging sensitivity, imaging with increased infrared
sensitivity, imaging with increased ultraviolet sensitivity, the
application of the image effect, or the like, the user becomes able
to view a corresponding photographed image (e.g., one of the images
as described above with reference to FIGS. 7A to 10B) obtained in
the special imaging state with the display section 2.
[0275] Note that, in this case, the photographed image may be
displayed on the entire screen of the display section 2, or as is
the case with the replay images as illustrated in FIGS. 12A and
12B, the photographed image may be displayed in a part of the
screen such as the area AR2 while the area AR1 is in the
see-through state.
[0276] In the case where the imaging-related operation detected at
step F103 is an operation for shifting the state of the imaging
operation from the special imaging state back to the regular
imaging state, the system controller 10 controls the imaging system
to return to the regular imaging state at step F301 in FIG. 15 as
described above. In this case, the system controller 10 proceeds
from step F302 to step F304.
[0277] At step F304, the system controller 10 instructs the display
control section 14 to control the entire screen of the display
section 2 to return to the see-through state.
[0278] The system controller 10 regards the shifting of the state
of the imaging operation to the special imaging state as a result
of the user operation as the satisfaction of the storage condition.
Therefore, at step F305, the system controller 10 determines the
storage range of the image data to be stored in accordance with
special imaging that had been performed up to the immediate past.
In addition, the system controller 10 generates the metadata. This
metadata includes: information based on the biological information
detected during a period of the special imaging state; information
indicating that the image data to be stored has been determined
because of the special imaging state, such as information that
indicates, as special imaging that had been performed, telephoto
zooming, wide-angle zooming, image magnification, image reduction,
the change of the frame rate, the change of the imaging
sensitivity, imaging with increased infrared sensitivity, imaging
with increased ultraviolet sensitivity, the application of the
image effect, or the like; and so on.
[0279] Further, the system controller 10 turns on the storage
flag.
[0280] After performing the above processes, the system controller
10 returns to the monitoring loop at steps F103 to F108 in FIG.
13.
[0281] Immediately after the state of the imaging operation is
shifted from the special imaging state back to the regular imaging
state, the system controller 10 proceeds from step F107 to step
F113 in FIG. 13 because the storage flag has been turned on. Then,
at step F113, the system controller 10 performs the storage
process.
[0282] As the storage process, the system controller 10 controls
the storage section 25 to record the image data within the storage
range and the metadata on the storage medium. That is, in this
case, the system controller 10 allows the image data within the
storage range determined at step F305 in FIG. 15, e.g., the image
data that was obtained by imaging in the special imaging state, to
be transferred from the temporary storage section 23 to the storage
section 25, and transmits the metadata generated at step F305 to
the storage section 25. Then, the system controller 10 controls the
storage section 25 to add the metadata to the image data, encode
the resultant data for recording, and record the resultant data on
the storage medium.
[0283] Note that in the case where the imaging/display apparatus 1
includes the communication section 26, the system controller 10 may
perform control for allowing the image data within the storage
range and the metadata to be encoded for transmission, and
transmitting the resultant data to the external device via the
communication section 26.
[0284] After the storage process at step F113 is completed, the
system controller 10 turns off the storage flag at step F114 and
returns to the monitoring loop at steps F103 to F108.
[0285] Incidentally, the special imaging state may continue for a
long time. For example, a zoom state or the like may continue
longer than a period of time for which the data is stored in the
temporary storage section 23. Note that this period of time depends
on a storage capacity of the temporary storage section 23.
Therefore, if the storage process is performed at step F113 after
the state of imaging is returned to the regular imaging state, for
example, part of the image data to be stored may have already been
lost.
[0286] Therefore, it may be necessary to modify the above
procedures so that when a certain period of time has elapsed after
the state of imaging was shifted to the special imaging state, the
processes of steps F305 and F113 will be performed
interruptively.
[0287] Moreover, although not shown in FIG. 15, it may happen that
the state of imaging is switched from a certain special imaging
state to another special imaging state or that a compound operation
is requested. For example, the state of imaging may be switched
from a telephoto zoom state to a state in which imaging is
performed with a varied frame rate. Also, imaging with increased
infrared sensitivity may be requested while the telephoto zoom
state should be maintained. It is preferable that the processes of
steps F305 and F113 be performed interruptively when the mode of
the operation has been altered while the state of imaging is
maintained in the special imaging state.
[0288] When the system controller 10 has determined that the sound
trigger has occurred in the monitoring loop at steps F103 to F108,
the system controller 10 proceeds from step F105 to step F111.
Then, the system controller 10 determines the storage range of the
image data to be stored as an accompaniment to the occurrence of
the sound trigger. In addition, the system controller 10 generates
the metadata. This metadata includes: information based on the
biological information detected at the time of the occurrence of
the sound trigger; information indicating that the image data to be
stored has been determined because of the occurrence of the sound
trigger; a content of the sound trigger (i.e., the analysis
information obtained by the audio analysis section 24); and so on.
Further, the system controller 10 turns on the storage flag.
[0289] After performing the above processes, the system controller
10 returns to the monitoring loop at steps F103 to F108.
Immediately after this, the system controller 10 proceeds from step
F107 to step F113 because the storage flag has been turned on, and
performs the storage process.
[0290] As the storage process, the system controller 10 controls
the storage section 25 to record the image data within the storage
range and the metadata on the storage medium. That is, in this
case, the system controller 10 allows the image data within the
storage range determined at step F111 to be transferred from the
temporary storage section 23 to the storage section 25, and
transmits the metadata generated at step F111 to the storage
section 25. Then, the system controller 10 controls the storage
section 25 to add the metadata to the image data, encode the
resultant data for recording, and record the resultant data on the
storage medium.
[0291] In the case where the imaging/display apparatus 1 includes
the communication section 26, the system controller 10 may perform
control for allowing the image data within the storage range and
the metadata to be encoded for transmission, and transmitting the
resultant data to the external device via the communication section
26.
[0292] After the storage process at step F113 is completed, the
system controller 10 turns off the storage flag at step F114 and
returns to the monitoring loop at steps F103 to F108.
[0293] When the system controller 10 has determined that the
biological trigger has occurred in the monitoring loop at steps
F103 to F108, the system controller 10 proceeds from step F106 to
step F112. Then, the system controller 10 determines the storage
range of the image data to be stored as an accompaniment to the
occurrence of the biological trigger. In addition, the system
controller 10 generates the metadata. This metadata includes
information representing that the image data to be stored has been
determined because of the occurrence of the biological trigger, and
a content of the biological trigger (e.g., the information detected
by the biological sensor 21, a content of judgment about the state
of the user based on the detected information, etc.). Further, the
system controller 10 turns on the storage flag.
[0294] After performing the above processes, the system controller
10 returns to the monitoring loop at steps F103 to F108.
Immediately after this, the system controller 10 proceeds from step
F107 to step F113 because the storage flag has been turned on, and
performs the storage process.
[0295] As the storage process, the system controller 10 controls
the storage section 25 to record the image data within the storage
range and the metadata on the storage medium. That is, in this
case, the system controller 10 allows the image data within the
storage range determined at step F112 to be transferred from the
temporary storage section 23 to the storage section 25, and
transmits the metadata generated at step F112 to the storage
section 25. Then, the system controller 10 controls the storage
section 25 to add the metadata to the image data, encode the
resultant data for recording, and record the resultant data on the
storage medium.
[0296] In the case where the imaging/display apparatus 1 includes
the communication section 26, the system controller 10 may perform
control for allowing the image data within the storage range and
the metadata to be encoded for transmission, and transmitting the
resultant data to the external device via the communication section
26.
[0297] After the storage process at step F113 is completed, the
system controller 10 turns off the storage flag at step F114 and
returns to the monitoring loop at steps F103 to F108.
[0298] When the power is turned off or the operation is completed,
the system controller 10 proceeds from step F108 to step F115, and
terminates the imaging operation in the imaging system and the
storage of the image data in the temporary storage section 23,
thereby finishing the series of processes.
[0299] According to the above-described procedures, the constant
imaging and the temporary storage of the image data obtained by
imaging are performed while, out of the image data temporarily
stored, the image data of the image that has interested the user or
which the user desires to watch again later is stored in the
storage section 25 on a permanent basis (or transmitted to the
external device via the communication section 26 and stored therein
on a permanent basis).
[0300] Moreover, the user is able to watch the scene that he or she
viewed in the immediate past by the replay operation.
[0301] Note that it has been assumed in the above-described
exemplary procedures that the images are displayed on a part or the
whole of the display section 2 when the imaging system has entered
the special imaging state or when the displaying of the replay
images is performed, while otherwise the entire screen of the
display section 2 is caused to stay in the see-through state.
However, instead of causing the entire screen of the display
section 2 to stay in the see-through state, it is possible to
display the image data obtained by regular imaging on the entire
screen of the display section 2.
[0302] Also note that in the case where, as illustrated in FIG. 2,
the imaging/display apparatus 1 is composed of the display
apparatus section 30 and the imaging apparatus section 40, which
are separate from each other, a process of causing the display
apparatus section 30 to enter the see-through state does not need
to be performed. In this case, it may be so arranged that, at
normal times, the screen of the display apparatus section 30 stays
inactive or the image data obtained by regular imaging is displayed
thereon.
[0303] As described above, the system controller 10 performs the
process of generating the metadata at steps F111 and F112 in FIG.
13, step F206 in FIG. 14, and step F305 in FIG. 15. The process of
generating the metadata is performed in accordance with a procedure
illustrated in FIG. 16, for example.
[0304] First, at step F401, the system controller 10 generates
storage condition information as information to be included in the
metadata. In the case of step F111 in FIG. 13, the storage
condition information refers to the information indicating that the
image data to be stored has been determined because of the
occurrence of the sound trigger, and information concerning a
result of the audio analysis. In the case of step F112 in FIG. 13,
the storage condition information refers to the information
indicating that the image data to be stored has been determined
because of the occurrence of the biological trigger. In the case of
step F206 in FIG. 14, the storage condition information refers to
the information indicating that the image data to be stored has
been determined because of the performance of the replay operation.
In the case of step F305 in FIG. 15, the storage condition
information refers to information indicating the operation that had
been performed in the special imaging state.
[0305] Next, at step F402, the system controller 10 generates
biological sensor detection information and feeling information,
which are also to be included in the metadata. The biological
sensor detection information refers to information of the value
detected by the biological sensor. The biological sensor detection
information is, for example, information representing a detection
value of the biological information to be detected, such as the
heart rate or the blood pressure.
[0306] The feeling information refers to information representing
the feeling of the user determined based on the biological
information detected by the biological sensor 21. The feeling
information is, for example, information that represents
"cheerful", "amused", "pleased", "sad", "frightened", "calm",
"nostalgic", "moved", "alarmed", "excited", "tense", or the
like.
[0307] At step F403, the system controller 10 receives the
information of the current date and time obtained by the time/date
calculation section 18, and generates date/time information as
information to be included in the metadata.
[0308] At step F404, the system controller 10 receives the
information of the current location (i.e., the latitude and
longitude thereof) obtained by the GPS receiver section 19, and
generates location information as information to be included in the
metadata.
[0309] At step F405, the system controller 10 generates user
identification information as information to be included in the
metadata. Various types of user identification information are
conceivable. In the case where the user has set an ID number
previously, for example, the ID number may be used as the user
identification information. Also, a serial number or the like
assigned to the imaging/display apparatus 1 may be considered as
identification information specific to the user and thus be used as
the user identification information.
[0310] As an example of the user identification information that
serves to identify an individual more accurately, information
detected by the biological sensor 21 can be used. For example, one
or more pieces of information, such as the pulse, breathing
activity, the fundus pattern, and the thickness of the crystalline
lens, can be used individually or collectively to generate
identification information of the user who is actually using the
imaging/display apparatus 1.
[0311] Then, at step F406, the system controller 10 generates the
metadata including all pieces of information generated at steps
F401 to F405.
[0312] This metadata is added to the image data and then subjected
to the storage process at step F113 in FIG. 13. Thus, the storage
condition, the biological information (e.g., the detection value or
the feeling information), the date/time information, the location
information, and the user identification information are, as added
information, associated with the image data to be stored.
[0313] The metadata as described above is used when performing a
search across the stored image data.
7. Exemplary Procedure for Displaying Stored Image
[0314] As described above, the imaging/display apparatus 1
according to the present embodiment images the scenes within the
user's field of vision and stores image data of scenes that are
meaningful for the user.
[0315] Then, a search is automatically performed across the stored
image data as described below, so that an image that reminds the
user, for example, of a past memory or feeling is displayed and
thus presented to the user.
[0316] An exemplary procedure for displaying the image data stored
in the storage section 25 or the external device (e.g., the storage
device 70 or the server apparatus 71) will now be described below
with reference to FIG. 17.
[0317] FIG. 17 illustrates a procedure performed by the system
controller 10 for displaying a stored image. This procedure is
performed continuously all the time while the power of the
imaging/display apparatus 1 is on, for example. Alternatively, this
procedure may be started when operation start is requested by a
user operation or the like after the power of the imaging/display
apparatus 1 was turned on.
[0318] Note that the procedure for displaying the stored image as
illustrated in FIG. 17 is performed in parallel with the
above-described procedures of FIGS. 13 to 16, i.e., procedures
related to imaging and the storage of the image data. The parallel
processing is achieved, for example, by the procedure of FIG. 17
being regularly handled as an interrupt while the system controller
10 is performing the procedures of FIGS. 13 to 16. A program for
the procedure of FIG. 17 may either be incorporated in a program
for executing the procedures of FIGS. 13 to 16, or be a separate
program that is called up regularly. That is, the form of the
program(s) is not limited.
[0319] After the power is turned on, for example, the system
controller 10 performs a process of identifying a situation at step
F501 in FIG. 17. The situation refers to the user situation, the
date and time, the location, the user operation, or the like. The
user situation is, for example, information of the detection value
of the biological information concerning the body of the user
detected by the biological sensor 21 or the feeling information of
the user, which can be determined based on the biological
information.
[0320] At step F502, based on the user situation, the date and
time, the location, the user operation, or the like identified at
step F501, the system controller 10 determines whether a search
should be performed now in order to display a stored image of a
past scene.
[0321] For example, regarding the user situation, the feeling
information, the date and time, the location, or the like, a
situation in which a search should be performed may be specified
beforehand. In this case, the system controller 10 determines
whether a current situation corresponds to any of such previously
specified situations in which a search should be performed.
[0322] Specifically, a time when the feeling information such as
"cheerful", "sad", "pleased", "alarmed", or the like has been
detected may be specified as a time when a search should be
performed. In this case, the system controller 10 determines
whether the user currently has such a feeling.
[0323] Also, a time when the detection value of the biological
information has entered a predetermined situation in terms of the
numerical value, such as "the heart rate being greater than x",
"the amount of perspiration being greater than x", or "the level of
the alpha waves being higher than x", may be specified as a time
when a search should be performed. In this case, the system
controller 10 determines whether a current detection value of the
biological information falls within such a predetermined
situation.
[0324] Moreover, the date and time falling within a specific
season, month, day, or time (time period), the user being at a
specific location, or the like may also be taken into account when
determining whether a search should be performed now.
[0325] If it is determined that the current situation is not a
situation in which a search should be performed, the system
controller 10 returns to step F501 through step F507. Meanwhile, if
it is determined that the current situation is a situation in which
a search should be performed, the system controller 10 proceeds to
step F503, and sets the search condition and performs control for
performing the search.
[0326] The search condition is a search condition used when
performing a search across the above-described metadata. A content
of the information added to the image data as the metadata is
specified as the search condition.
[0327] For example, the value of the biological information or the
feeling information detected at the time of the situation
identification at step F501 is specified as the search condition.
Specifically, in the case where a feeling "cheerful" has been
detected as the current feeling of the user, information "cheerful"
may be specified as the search condition. Also, in the case where
the current heart rate of the user is "greater than x", information
"the heart rate being greater than x" may be specified as the
search condition.
[0328] Further, the current date and time or the current location
(i.e., the latitude and longitude thereof) may also be included in
the search condition. The user identification information may also
be included in the search condition. The user identification
information is the identification information described above when
describing the generation of the metadata.
[0329] Still further, information of a content of a current user
operation (e.g., an operation related to the imaging system, such
as the zoom operation, or an operation for replaying) may also be
included in the search condition.
[0330] After specifying the search condition, the system controller
10 supplies the search condition to the storage section 25 and
causes the storage section 25 to perform a search across the image
data stored therein.
[0331] Alternatively, the system controller 10 may transmit the
search condition to the external device via the communication
section 26 and request the external device to perform a search.
[0332] The storage section 25 (or the external device) searches the
metadata of the stored image data to determine whether there is any
metadata that matches the search condition, and notifies the system
controller 10 whether any image data that matches the search
condition has been found.
[0333] If no image data that matches the search condition has been
found, the system controller 10 returns from step F504 to step F501
through step F507.
[0334] If a certain piece of image data that matches the search
condition is found, the system controller 10 proceeds from step
F504 to step F505, and performs a process of displaying the image
data found as a result of the search.
[0335] Specifically, the system controller 10 instructs the storage
section 25 to read the found image data, and controls the read
image data to be supplied to the display image processing section
12 via the image input/output control section 27. Meanwhile, the
system controller 10 instructs the display control section 14 to
perform a process of displaying an image of the image data read
from the storage section 25. As a result, the image found as a
result of the search is displayed on the display section 2.
[0336] In the case where the search is performed in the external
device, the communication section 26 receives the found image data
transmitted from the external device. The system controller 10
controls the found image data read in and transmitted from the
external device and received by the communication section 26 to be
supplied to the display image processing section 12 via the image
input/output control section 27. Meanwhile, the system controller
10 instructs the display control section 14 to perform a process of
displaying this image data. As a result, an image found as a result
of the search in the external device is displayed on the display
section 2.
[0337] Various modes of displaying the image (i.e., a playback
image) found as a result of the search on the display section 2 are
conceivable. The various modes of displaying of the playback image
are achieved, for example, by, based on an instruction issued from
the system controller 10, the display control section 14 specifying
a display mode to the display image processing section 12 and the
display driving section 13.
[0338] For example, as illustrated in FIG. 18A, in a screen area of
the display section 2, a child-screen area AR2 may be set within a
parent-screen area AR1, and then, the playback image may be
displayed in the area AR2 while the area AR1 is allowed to stay in
the see-through state. In this case, the user ifs able to watch the
automatically retrieved image of a past scene with the area AR2
while seeing a current scene with the area AR1, which is in the
see-through state.
[0339] FIG. 18B illustrates an exemplary case where the screen area
is split into upper and lower areas AR1 and AR2, and the playback
image is displayed in the area AR2 while the area AR1 is allowed to
stay in the see-through state. In this case also, the user is able
to watch the automatically retrieved image of the past scene with
the area AR2 while seeing the current scene with the area AR1,
which is in the see-through state.
[0340] As described above, the screen of the display section 2 is
divided into the parent and child screens or split into two parts,
for example, and the displaying of the playback image found as a
result of the search is performed while a part of the screen of the
display section 2 is allowed to stay in the see-through state.
Needless to say, the position of the child screen within the screen
and the size of the child screen may be changeable in accordance
with the user operation. It is also possible that the screen of the
display section 2 be split into left and right areas AR1 and AR2.
It is also possible that the area of the area AR1 and the area of
the area AR2 be set to be unequal by the user operation. Further,
switching of display positions may be carried out in accordance
with the user operation. For example, the parent screen and the
child screen may be replaced by each other in accordance with the
user operation. Also, the areas obtained by screen splitting may be
replaced by each other in accordance with the user operation.
[0341] Further, instead of being allowed to stay in the see-through
state, the area AR1 may be employed to display the image that is
currently obtained by imaging by the imaging system (i.e., the
image of the current scene, which is equivalent to the scene that
the user would see if the screen were in the see-through
state).
[0342] Also, the playback image found as a result of the search may
be displayed once or repeatedly across the entire screen area while
the whole or a part of the screen area is allowed to stay in the
see-through state.
[0343] For example, while the entire screen area is allowed to stay
in the see-through state, the playback image may be displayed
across the entire screen area for an instant. Alternatively, the
instant displaying of the playback image may be repeated.
[0344] Also, since the pair of display sections 2 are prepared for
the pair of eyes, an entire screen of one of the two display
sections 2 may be allowed to stay in the see-through state while
the playback image is displayed on an entire screen of the other
one of the two display sections 2.
[0345] Further, while the entire screen area of the display section
2 is allowed to stay in the see-through state, the playback image
may be displayed in the whole or a part of the screen area.
[0346] For example, FIG. 19A illustrates a scene that the user sees
when the entire screen is in the see-through state, whereas FIG.
19B illustrates a case where, at this time, a playback image P is
displayed in a lower right corner of the screen area. Further, FIG.
19C illustrates a case where while the entire screen is in the
see-through state, the playback image P is displayed across the
entire screen.
[0347] When the playback image is displayed in an area that is in
the see-through state because of increased transmittance, the
playback image appears within the user's field of vision as if it
loomed up indistinctly. This results in a display that reminds the
user of a memory or a past experience, for example.
[0348] Note that, in the case where the display apparatus section
30 is provided separately as illustrated in FIG. 2, there is no
need to cause the screen to enter the see-through state, and
therefore, the playback image may be displayed in the display
apparatus section 30 in a regular manner.
[0349] At step F505, the retrieved image data is presented to the
user in any of the above-described manners. When the displaying of
the retrieved image data is completed, the system controller 10
proceeds from step F506 to step F507, and, if the operation should
not be ended yet, returns to step F501.
[0350] The procedure of FIG. 17 as described above is continued
until it is determined at step F507 that the operation should be
ended. The end of the operation is determined, for example, when
the power of the imaging/display apparatus 1 has been turned off or
when the user has performed an operation for stopping the above
display procedure.
[0351] Note that although the above procedure of FIG. 17 has been
assumed to be performed continuously after the power of the
imaging/display apparatus 1 is turned on or after the operation
start is requested by the user operation or the like after the
power of the imaging/display apparatus 1 was turned on, the user
may be allowed to perform an operation for requesting the
displaying of the playback image. In this case, when the user has
performed that operation, the procedure of FIG. 17 is started. In
this case, the determination at step F502 of whether the current
situation is a situation in which a search should be performed may
be omitted.
[0352] Moreover, user authentication may be carried out as a
condition for performing the procedure of FIG. 17. For example, the
information detected by the biological sensor 21 can be used to
identify the individual user. Accordingly, the display procedure of
FIG. 17 may be performed only when the imaging/display apparatus 1
is worn by a specific user. Needless to say, not only the procedure
of FIG. 17 but also the procedures of FIGS. 13 to 16 may be
performed only when the imaging/display apparatus 1 is worn by the
specific user.
8. Effects of Embodiments, Exemplary Variants, and Exemplary
Expansions
[0353] As described above, in the imaging/display apparatus 1
according to the above-described embodiments of the present
invention, the image data of the scenes that the user sees in his
or her daily life is stored together with the metadata generated
based on the biological information concerning the user. Thus,
images of daily scenes can be stored so as to be associated with
the situation of the user, so that the images can be retrieved
conveniently for subsequent playback or the like.
[0354] Further, by searching for the stored image data using the
search condition generated by use of the biological information and
displaying the retrieved image data, it is possible to present an
image of a past scene that is suited to the current situation of
the user. Thus, very entertaining image presentation is achieved,
such as presentation of an image that reminds the user of a memory,
presentation of an image that allows the user to watch a similar
scene in the past, or presentation of an image of an impressive
event in the past.
[0355] More specifically, the following is achieved by the
imaging/display apparatus 1 performing operations such as imaging,
temporary storage, and storage process in accordance with the
control procedures of FIGS. 13 to 16 performed by the system
controller 10, and performing automatic search and the display
operation in accordance with the control procedure of FIG. 17
performed by the system controller 10.
[0356] First, because every scene that the user sees is stored
temporarily, the user is able to watch the scene in the immediate
past again by replaying.
[0357] In the case where the user is watching a sport game in a
sports ground, a soccer stadium, a ballpark, or the like, for
example, the user is normally unable to watch a replay as when the
user is watching a sport game with a television broadcast. However,
if the user wears the imaging/display apparatus 1, the user is able
to watch a replay of a play of a player arbitrarily, for
example.
[0358] Further, the following situations may occur in the user's
daily life: the user is absentminded and misses a certain scene;
the user happens to pass an interesting person; and the user
witnesses a traffic accident or the like. In such situations, as
well as in various other situations, the user is able to watch a
scene in the immediate past by replaying.
[0359] Because the temporary storage section 23 is assumed to be
used for temporary storage, the temporary storage section 23 can be
used for storing the images constantly obtained by imaging, using
its storage area in the ring memory manner, and does not need to
have enormous storage capacity. The storage capacity of the
temporary storage section 23 may be determined in a design stage
based on how long ago the image data that can be replayed should
extend, for example.
[0360] Further, by performing the imaging-related operation, the
user is able to watch the image obtained by imaging in the special
imaging state with the display section 2, such as the telephoto
image, the wide-angle image, the magnified image, the image
photographed with increased infrared imaging sensitivity, the image
photographed with increased ultraviolet imaging sensitivity, or the
image photographed with a high frame rate. Thus, the user is able
to arbitrarily watch a scene that cannot be seen with a normal
vision.
[0361] Further, when the user has performed an operation to issue
an instruction to perform replaying or imaging in the special
imaging state, a scene that is a subject of replaying or imaging is
probably an interesting scene for the user. Since the storage
process of causing the image data of such a scene to be stored in
the storage section 25 (or the external device with which the
communication section 26 communicates) on a permanent basis is
performed, the image of the interesting scene for the user is
stored. Therefore, the user is able to play the image of the scene
at a later date to watch the scene again. The user is also able to
compile such stored images into a video album or the like that
records an action history or memories of the user, for example.
[0362] Similarly, the biological trigger, the sound trigger, and
the like also initiate the storage process of storing the image
data. Therefore, without the need for the user to perform any
particular operation, the image data of the scene that interests
the user or which is important for the user can be stored on a
permanent basis.
[0363] Thus, an imaging/display apparatus that is capable of
recording the image of the important scene in the user's daily life
is achieved.
[0364] Still further, the image of the past scene is automatically
retrieved and displayed in accordance with the feeling or body
condition of the user, the date and time, the location, or the
like. Thus, an image that is suited to the situation of the user or
the like is presented to the user.
[0365] When the user is cheerful, for example, a past scene that
the user saw when he or she was similarly cheerful is displayed.
When the user feels sad, a past scene that the user saw when he or
she was sad is displayed.
[0366] For example, FIGS. 18A and 18B illustrate cases where when
the user is currently moved by a certain view, an image of a past
view that similarly moved the user in the past is being
displayed.
[0367] Meanwhile, FIGS. 19B and 19C illustrate cases where when the
user, filled with nostalgic sweetness, is reminded of a past
acquaintance, girlfriend, or the like, a scene at an earlier time
that the user spent with her is being displayed.
[0368] As exemplified by the above, a display operation that causes
a scene at a time when the user had a similar feeling in the past
to be displayed as if it loomed up is performed in accordance with
the current feeling of the user or the like.
[0369] Further, in the case where the date and time, the location,
or the like is also included in the search condition, it is
possible to retrieve image data that is associated with a specific
date and time, a specific location, or the like as well. For
example, when the user is enjoying a Christmas Day, an image of a
scene of a past Christmas Day that the user enjoyed can be played
back. Also, when the user is excited watching a sport game in a
sport stadium, an image of a scene at an earlier time when the user
was excited in the same place can be displayed, for example.
[0370] In the case where the image data is stored in the storage
section 25 within the imaging/display apparatus 1, and the image
data stored in the storage section 25 is retrieved and then played
back, a scene reproduced is basically a scene that the user himself
or herself saw in the past. In contrast, in the case where the
imaging/display apparatus 1 has the communication section 26, not
only the scenes that the user himself or herself saw but also
scenes that other people saw can be displayed for the user to
see.
[0371] Suppose, for example, that many users each wear the
imaging/display apparatus 1 and the image data obtained is stored
in the server apparatus 71. In this case, image data of scenes
viewed by each of the many users is stored in the server apparatus
71. When such image data is searched as a search target, a scene
viewed by another person and which matches the current feeling of
the user, the date and time, the location, or the like can be
retrieved.
[0372] That is, scenes associated with feelings or the like of many
people can be shared among them. This enables the user, when
feeling cheerful, to watch a scene that another person viewed when
he or she was cheerful, for example.
[0373] Meanwhile, because the image data to be stored is image data
of scenes that the individual user viewed, it may be improper to
permit every individual to watch the images of the stored image
data from the standpoint of privacy. Accordingly, the user
identification information may be added to the search condition so
that only the scenes that the user himself or herself saw in the
past can be retrieved.
[0374] Further, identification information may be set for a group
of specific users, and this identification information may be
included in the metadata to be stored together with the image data
and also added to the search condition. This makes it possible for
the users belonging to the same group to share the image data of
scenes viewed by each of the users. This enables a group of
acquaintances to share scenes that they saw when they were moved,
scenes that they saw when they were cheerful, and so on.
[0375] The retrieved image is displayed while a part of the screen
area stays in the see-through state as illustrated in FIGS. 18A and
18B, or the retrieved image is displayed so as to be superimposed
on the current scene viewed through the screen in the see-through
state as illustrated in FIGS. 19B and 19C. Therefore, the
displaying of the retrieved image does not result in obstructing
the user's field of vision. That is, appearance of the image of the
past scene does not interfere with the user's activity in his or
her daily life.
[0376] The displaying of the playback image such that it looms up
indistinctly as illustrated in FIGS. 19B and 19C provides a display
effect suitable for the memorable scene.
[0377] Note that a variety of variants are conceivable about the
structure and operation of the imaging/display apparatus 1.
[0378] In the above-described embodiments of the present invention,
the image data constantly obtained by imaging is stored in the
temporary storage section 23 and when the storage condition is
satisfied, relevant image data is stored in the storage section 25
or the like on a permanent basis. However, in the case where the
storage section 25 within the imaging/display apparatus 1 or the
server apparatus 71 can use a nonvolatile storage medium with very
large storage capacity, for example, all pieces of photographed
image data may be stored on a permanent basis. In this case, the
value of the biological information constantly detected, the
feeling information, the information of the date and time, the
information of the location, or the like may be added as the
metadata.
[0379] It is not essential that imaging and the storage (or
temporary storage) of the image data be performed all the time
while the user is wearing the imaging/display apparatus 1. Instead,
a period in which such an operation is performed may be specified
by the user with the user operation. In other words, the procedures
of FIGS. 13 to 16 may be performed in a period between when the
user performs a starting operation and when the user performs a
termination operation.
[0380] Similarly, the procedure of FIG. 17 may be performed in a
period between when the user performs a starting operation and when
the user performs a termination operation.
[0381] It should be noted that the procedures of FIGS. 13 to 16 and
the procedure of FIG. 17 have been described by way of example.
There are various procedures that can be adopted by an imaging
apparatus for imaging and storing the image data. Also, there are
various procedures that can be adopted by a display apparatus for
retrieving and displaying the stored image data in accordance with
the current situation.
[0382] As described above, in the case where the image data is
stored in the external device such as the storage device 70 or the
server apparatus 71, the search condition may be transmitted to the
external device so that the search is performed using the search
condition in the external device in the procedure of FIG. 17.
However, it is also possible, for example, that all pieces of image
data that match a certain condition (e.g., the user identification
information, a date/time range, etc.) and are stored in the
external device be downloaded to the storage section 25 within the
imaging/display apparatus 1 and then a search be performed across
the image data in the storage section 25.
[0383] Such an operation is possible when the imaging/display
apparatus 1 has both the communication section 26 and the storage
section 25.
[0384] In the exemplary manners of displaying the playback image as
described above with reference to FIGS. 18A, 18B, 19B and 19C, at
least a part of the screen area is allowed to stay in the
see-through state so that the user can see the current scene.
However, instead of allowing at least a part of the screen area to
stay in the see-through state, image data currently obtained by
imaging by the imaging system may be displayed. This is because it
also enables the user to see the current scene.
[0385] While retrieval and displaying of the image of the past
scene has been described above, it is also conceivable that the
audio data is stored together with the metadata and audio data of a
past sound is searched for and retrieved in accordance with the
current situation so that the retrieved audio data is played back
for the user.
[0386] This achieves, for example, an operation of, when the user
is moved, playing back music that the user listened to when he or
she was moved in the past.
[0387] While the imaging/display apparatus 1 has been described
above as one embodiment of the present invention, it should be
noted that an imaging apparatus for imaging and storing the image
data and a display apparatus for retrieving and displaying the
stored image data in accordance with the current situation are also
conceivable as embodiments of the present invention.
[0388] Also note that an imaging apparatus that does not have a
capability to perform replaying described above is also conceivable
as one embodiment of the present invention.
[0389] It should be understood by those skilled in the art that
various modifications, combinations, sub-combinations and
alterations may occur depending on design requirements and other
factors insofar as they are within the scope of the appended claims
or the equivalents thereof.
* * * * *