U.S. patent application number 15/583323 was filed with the patent office on 2017-08-17 for image pickup apparatus, information processing apparatus, and information processing method.
The applicant listed for this patent is CANON KABUSHIKI KAISHA. Invention is credited to Hiroshi Tojo.
Application Number | 20170236010 15/583323 |
Document ID | / |
Family ID | 43879001 |
Filed Date | 2017-08-17 |
United States Patent
Application |
20170236010 |
Kind Code |
A1 |
Tojo; Hiroshi |
August 17, 2017 |
IMAGE PICKUP APPARATUS, INFORMATION PROCESSING APPARATUS, AND
INFORMATION PROCESSING METHOD
Abstract
An information processing apparatus comprises: an object
detection unit to detect an object included in a frame image based
on a feature amount of the frame image, and generate and output
object information concerning the detected object; an event
detection unit to detect an event of the object based on the object
information output by the object detection unit, and generate and
output event concern information concerning the detected event of
the object; and a transmission unit to transmit the frame image,
the event concern information, concerning the frame image, output
by the event detection unit, and time information concerning the
frame image, as associating them with others. Thus, even in a case
where event detection timing and event occurrence timing are
different from each other, it is possible to perform a display by
which a user can easily confirm the event.
Inventors: |
Tojo; Hiroshi; (Fuchu-shi,
JP) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
CANON KABUSHIKI KAISHA |
Tokyo |
|
JP |
|
|
Family ID: |
43879001 |
Appl. No.: |
15/583323 |
Filed: |
May 1, 2017 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
14953559 |
Nov 30, 2015 |
9679202 |
|
|
15583323 |
|
|
|
|
12902639 |
Oct 12, 2010 |
9237266 |
|
|
14953559 |
|
|
|
|
Current U.S.
Class: |
382/103 |
Current CPC
Class: |
G06K 9/00744 20130101;
G06K 9/00771 20130101; H04N 5/23293 20130101; G06K 2009/00738
20130101; G06K 9/00718 20130101; H04N 5/232 20130101; H04N 5/232945
20180801; H04N 7/18 20130101; H04N 5/23218 20180801; G06T 7/246
20170101; G06T 7/73 20170101; H04N 5/232941 20180801; H04N 5/23206
20130101 |
International
Class: |
G06K 9/00 20060101
G06K009/00; H04N 5/232 20060101 H04N005/232 |
Foreign Application Data
Date |
Code |
Application Number |
Oct 19, 2009 |
JP |
2009-240655 |
Claims
1.-15. (canceled)
16. An image processing apparatus comprising: a receiving unit
configured to receive a plurality of frame images; a storage
control unit configured to store in a storage apparatus the
plurality of frame images in association with time information
which indicates receiving times at which respective frame images
were received; an object detection unit configured to detect an
object from a frame image in the plurality of frame images; an
event detection unit configured to detect a leaving event in which
the object has been left behind based on a status of the object and
a duration time of the status; a determination unit configured to
determine an event occurrence time indicating occurrence of the
leaving event based on a detection time at which the leaving event
was detected and the existing time of the object; an obtaining unit
configured to obtain from the storage apparatus a frame image
corresponding to the determined event occurrence time; and a
display control unit configured to control a display apparatus to
display the frame image obtained by the obtaining unit.
17. The image processing apparatus according to claim 16, wherein
the event detection unit detects the leaving event if the status of
the object is a stationary state and the duration time of the
stationary state is equal to or longer than a threshold.
18. The image processing apparatus according to claim 16, wherein
the duration time of the status is represented as a number of
frames in which the object is in the status, and the event
occurrence time is represented as a frame number.
19. The image processing apparatus according to claim 16, wherein
the determination unit determines the event occurrence time by
subtracting the duration time of the status from the detection time
at which the leaving event was detected.
20. The image processing apparatus according to claim 16, wherein
the obtaining unit obtains a sequence of frame images including a
frame image corresponding to the event occurrence time, and the
display control unit controls the display apparatus to display the
sequence of frame images including the frame image corresponding to
the event occurrence time.
21. The image processing apparatus according to claim 16, wherein
the display control unit controls the display apparatus to display
the frame image corresponding to the determined event occurrence
time in a sub window.
22. The image processing apparatus according to claim 16, wherein
the display control unit controls the display apparatus to display
the frame image corresponding to the determined event occurrence
time with a message indicating the event occurrence.
23. An image processing method comprising: receiving a plurality of
frame images; storing in a storage apparatus the plurality of frame
images in association with time information which indicates
receiving times at which respective frame images were received;
detecting an object from a frame image in the plurality of frame
images; detecting a leaving event in which the object has been left
behind based on a status and a duration time of the status;
determining an event occurrence time indicating occurrence of the
leaving event based on a detection time at which the leaving event
was detected and the existing time of the object; obtaining from
the storage apparatus a frame image corresponding to the determined
event occurrence time; and controlling a display apparatus to
display the obtained frame image.
24. A non-transitory computer-readable storage medium which stores
a program to cause a computer to function as: a receiving unit
configured to receive a plurality of frame images; a storage
control unit configured to store in a storage apparatus the
plurality of frame images in association with time information
which indicates receiving times at which respective frame images
were received; an object detection unit configured to detect an
object from a frame image in the plurality of frame images; an
event detection unit configured to detect a leaving event in which
the object has been left behind based on a status and a duration
time of the status; a time determination unit configured to
determine an event occurrence time indicating occurrence of the
leaving event based on a detection time at which the leaving event
was detected and the existing time of the object; an obtaining unit
configured to obtain from the storage apparatus a frame image
corresponding to the determined event occurrence time; and a
display control unit configured to control a display apparatus to
display the frame image obtained by the obtaining unit.
Description
BACKGROUND OF THE INVENTION
[0001] Field of the Invention
[0002] The present invention relates to an image pickup apparatus,
an information processing apparatus, and an information processing
method.
[0003] Description of the Related Art
[0004] Conventionally, a digital monitoring system using a network
camera and a computer has been rapidly introduced. That is, the
digital monitoring system like this is constituted by the network
camera, a video recording apparatus of recording a video produced
by the network camera, and a viewer of reproducing a live video
produced by the network camera and a recorded video recoded by the
video recording apparatus. Here, the viewer has a function of
displaying a phenomenon of the system and a phenomenon occurred in
the video, as events.
[0005] For example, in Japanese Patent Application Laid-Open
2007-025950, abnormality is detected by analyzing a video produced
by a camera, and, if the abnormality is detected, an informing
signal or a message is output. Further, in Japanese Patent
Application Laid-Open 2007-049378, abnormality is detected by a
sensor, a camera related to the sensor is specified based on layout
information indicating the relation between the sensor and the
camera, and a current video produced by the specified camera is
displayed.
[0006] However, for example, in a case where a phenomenon that a
certain person misplaced or mislaid a bag or the like is detected
as an event, such a problem as described below occurs. That is,
only if the person goes away from a place where the bag has been
put on, a user of the system can actually detect such misplacement
of the bag. In this connection, even if the video obtained at the
time when the misplacement of the bag is detected can be displayed
by such conventional techniques as described above, it may be
difficult for the user to discriminate which person misplaced the
bag. According to circumstances, there is a possibility that the
person who is the owner of the misplaced bag has left. Like this,
if there is a gap between timing when the event can be detected and
timing when the event occurs (in this example, at a moment that the
person leaves from the bag), it is impossible in the conventional
technique to perform a display which enables the user to confirm
occurrence of the event.
[0007] The present invention has been completed in consideration of
the above-described conventional problem, and aims to perform the
display which enables the user to confirm the occurrence of the
event even if the timing when the event is detected is different
from the timing when the event occurs.
SUMMARY OF THE INVENTION
[0008] Consequently, an image pickup apparatus according to the
present invention is characterized by comprising: an object
detection unit configured to detect an object included in a frame
image on the basis of a feature amount of the frame image, and
generate and output object information concerning the detected
object; an event detection unit configured to detect an event of
the object on the basis of the object information output by the
object detection unit, and generate and output event concern
information concerning the detected event of the object; and a
transmission unit configured to transmit the frame image, the event
concern information, concerning the frame image, output by the
event detection unit, and time information concerning the frame
image, as associating the frame image, the event concern
information and the time information with others.
[0009] Further, an information processing apparatus according to
the present invention is characterized by comprising: a reception
unit configured to receive a frame image, event concern
information, associated with the frame image, concerning an event
of an object included in the frame image, and time information
concerning the frame information; a storage control unit configured
to store in a storage apparatus the frame image and the time
information received by the reception unit, as associating the
frame image and the time information with each other; an obtaining
unit configured to determine, on the basis of the event concern
information received by the reception unit, an event occurrence
time concerning occurrence of the event included in the
corresponding frame image, and obtain from the storage apparatus
the frame image corresponding to the determined event occurrence
time; and a display control unit configured to display on a display
apparatus the frame image obtained by the obtaining unit.
[0010] Furthermore, an information processing apparatus according
to the present invention is characterized by comprising: an object
detection unit configured to detect an object included in a frame
image on the basis of a feature amount of the frame image, and
generate and output object information concerning the detected
object; an event detection unit configured to detect an event of
the object on the basis of the object information output by the
object detection unit, and generate and output event concern
information concerning the detected event of the object; a storage
control unit configured to store in a storage apparatus the frame
image and time information concerning the frame image, as
associating the frame image and the time information with each
other; an obtaining unit configured to determine, on the basis of
the event concern information output by the event detection unit,
an event occurrence time concerning occurrence of the event
included in the corresponding frame image, and obtain from the
storage apparatus the frame image corresponding to the determined
event occurrence time; and a display control unit configured to
display on a display apparatus the frame image obtained by the
obtaining unit.
[0011] Further features and aspects of the present invention will
become apparent from the following detailed description of
exemplary embodiments with reference to the attached drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
[0012] FIG. 1 is a block diagram illustrating an example of a
hardware constitution of a PC (personal computer).
[0013] FIG. 2 is a block diagram illustrating an example of a
system configuration of an image processing system according to the
first embodiment, and functional constitutions of apparatuses
included in the image processing system.
[0014] FIG. 3 is a flow chart for describing a process to be
performed by a camera according to the first embodiment.
[0015] FIG. 4 is a flow chart indicating an example of a process to
be performed by an object detection unit.
[0016] FIG. 5 is a diagram for describing various modes.
[0017] FIG. 6 is a flow chart indicating an example of a process of
detecting misplacement as an event.
[0018] FIG. 7 is a diagram indicating an example of meta data.
[0019] FIG. 8 is a flow chart for describing a process to be
performed by the PC according to the first embodiment.
[0020] FIGS. 9A and 9B are diagrams respectively illustrating
screens.
[0021] FIG. 10 is a block diagram illustrating an example of a
system configuration of an image processing system according to the
second embodiment, and functional constitutions of apparatuses
included in the image processing system.
[0022] FIG. 11 is a flow chart for describing a process to be
performed by the PC according to the second embodiment.
[0023] FIG. 12 is a block diagram illustrating an example of a
system configuration of an image processing system according to the
third embodiment, and functional constitutions of apparatuses
included in the image processing system.
[0024] FIG. 13 is a diagram for describing a specific entrance and
exit event.
[0025] FIG. 14 is a flow chart for describing a process to be
performed by a camera according to the third embodiment.
[0026] FIG. 15 is a flow chart indicating an example of a process
of detecting specific entrance and exit as an event.
[0027] FIG. 16 is a flow chart for describing a process to be
performed by a PC according to the third embodiment.
DESCRIPTION OF THE EMBODIMENTS
[0028] Hereinafter, exemplary embodiments of the present invention
will be described with reference to the attached drawings.
[0029] An image processing system according to the first embodiment
of the present invention includes, as illustrated in FIG. 2, a
camera 100, and a PC 110 which is an example of an information
processing apparatus. Hereinafter, an example of a hardware
constitution of the PC 110 will be described with reference to FIG.
1. That is, FIG. 1 is the block diagram illustrating the example of
the hardware constitution of the PC.
[0030] A CPU (central processing unit) 801 performs various control
processes of the PC 110. A ROM (read only memory) 802 stores
therein a boot program to be performed when activating the PC 110,
and various data. A RAM (random access memory) 803 stores therein
control programs when the CPU 801 performs these programs, and also
provides a working area when the CPU 801 performs the various
control processes. Further, a keyboard 804 and a mouse 805 provide
a user with an operation environment for inputting various
information.
[0031] An external storage apparatus 806 is constituted by a hard
disk, a flexible disk, an optical disk, a magnetic disk, a
magnetooptical disk, a magnetic tape, or the like. However, if the
ROM 802 is set to store therein all of the control program and the
various data, the external storage apparatus 806 is not necessarily
an essential component as the hardware of the PC 110. A display 807
is constituted by a display unit or the like to display a processed
result and the like to the user. A network interface (indicated as
an NIC (network interface card) in FIG. 1) 808 enables to perform
communication between the camera 100 and the PC 110 on a network
through, e.g., a LAN (local area network) cable or the like.
Further, a bus 810 is used to mutually connect the respective
components as described above.
[0032] If the programs (control programs) stored in the external
storage apparatus 806 or the like are read to the RAM 803 or the
like and then performed by the CPU 801, the functions of the PC 110
according to the present embodiment and the processes to be
performed by the PC 110 as indicated by later-described flow charts
are achieved.
[0033] Incidentally, a figure illustrating in detail a hardware
constitution of the camera 100 in the present embodiment is
omitted. However, it should be noted that, as well as the PC 110,
the camera 100 is constituted by a CPU, a RAM, a ROM, an network
interface and the like. In addition, the camera 100 is equipped
with an imaging lens (image pickup lens), an imaging sensor (image
pickup sensor), and the like. If the programs stored in the ROM or
the like of the camera 100 are read to the RAM of the camera 100
and then performed by the CPU of the camera 100, the functions of
the camera 100 according to the present embodiment and the
processes to be performed by the camera 100 as indicated by
later-described flow charts are achieved.
[0034] FIG. 2 is a block diagram illustrating an example of a
system configuration of the image processing system according to
the first embodiment, and functional constitutions of the
apparatuses included in the image processing system.
[0035] In the image processing system as illustrated in FIG. 2, the
camera 100 and the PC 110 can communicate with each other through a
network.
[0036] The camera 100 includes, as its functional constitution, an
image pickup unit 101, a video compression encoding unit 102, an
object detection unit 103, an event detection unit 104, and a video
transmission unit 105.
[0037] Further, the image pickup unit 101 is constituted by the
imaging lens, and the imaging sensor such as a CCD (charge coupled
device), a CMOS (complementary metal-oxide semiconductor) sensor or
the like.
[0038] The video compression encoding unit 102 performs, frame by
frame, a compression encoding process to the image data obtained by
the image pickup unit 101, at predetermined time intervals.
[0039] The object detection unit 103 detects an object on the basis
of a feature amount generated by the video compression encoding
unit 102 during the compression.
[0040] The event detection unit 104 detects an event by analyzing
the result detected by the object detection unit 103, and outputs
the detected event as meta data.
[0041] The video transmission unit 105 adds the meta data obtained
as the result of the detection by the event detection unit 104 to
the frame image subjected to the compression encoding process by
the video compression encoding unit 102, and transmits the obtained
frame image. That is, the frame image is transmitted from the
camera 100 to the PC 110 as packet data of an http protocol through
the network such as the LAN cable or the like. Here, it should be
noted that the protocol to be used when transmitting the packet
data is not limited to the http protocol.
[0042] The PC 110 includes, as its functional constitution, a video
reception unit 111, a video recording unit 112, a video
accumulation unit 113, an event occurrence time point determination
unit 114, and a display unit 115.
[0043] The video reception unit 111 receives the packet data of the
frame image, through the network interface 808 on the PC 110. Then,
the video reception unit 111 generates the frame image based on the
received packet data, and outputs the generated frame image to the
video recording unit 112 together with a given time stamp.
[0044] The video recording unit 112 sequentially stores, together
with the time stamp, the received frame image in the video
accumulation unit 113 constituted by the external storage apparatus
806 or the RAM 803.
[0045] The event occurrence time point determination unit 114
obtains the meta data from the frame image received by the video
reception unit 111, and determines an event occurrence time point
of the event on the basis of the obtained meta data. Further, the
event occurrence time point determination unit 114 obtains the
frame image from the video accumulation unit 113 on the basis of
the determined event occurrence time point, and outputs the
obtained frame image to the display unit 115.
[0046] The display unit 115 displays, on the display 807, the frame
image which is being recorded by the video recording unit 112, and
the frame image at the time when the event occurred input from the
event occurrence time point determination unit 114. Thus, the user
is provided with these images through the display 807.
[0047] Subsequently, the process to be performed by the camera 100
will be described with reference to the flow chart illustrated in
FIG. 3. Namely, FIG. 3 is the flow chart for describing the process
to be performed by the camera according to the first
embodiment.
[0048] Initially, in a step S200, the image pickup unit 100
performs imaging to obtain the frame image.
[0049] Next, in a step S201, the video compression encoding unit
102 performs the compression encoding process to the frame image in
a proper compression encoding method. Here, the video compression
encoding unit 102 may select, as the compression encoding method,
any one of a JPEG (Joint Photographic Experts Group) method, an
MPEG-1 (Motion Picture Experts Group 1) method, an MPEG-2 method,
an MPEG-4 method, and the like. However, it is assumed that, in the
present embodiment, the video compression encoding unit 102
performs the compression encoding process in the JPEG method.
[0050] Next, in a step S202, the object detection unit 103 detects
the object based on the feature amount generated by the video
compression encoding unit 102 during the compression, and then
outputs the detected object as object information. Incidentally,
the detail of such an object detection process will be described
with reference to later-described FIG. 4.
[0051] Next, in a step S203, the event detection unit 104 detects
the phenomenon (event) such as intrusion, misplacement,
carrying-away or the like, by analyzing the object information
obtained as the result of the detection by the object detection
unit 103, and then outputs the detected result as the meta
data.
[0052] Next, in a step S204, the video transmission unit 105 adds
the meta data, obtained as the result of the process by the event
detection unit 104, to the frame image subjected to the compression
encoding process, and transmits the obtained frame image to the
network. For example, the video transmission unit 105 adds the meta
data to a header area of the frame image, and then transmits the
obtained frame image. Incidentally, in such a case, a
later-described frame number is used as an example of time
information.
[0053] How to add the meta data to the frame image is not limited
to such a method as described above. It is possible to also adopt a
method capable of adding the meta data to the frame image as
synchronizing the frame image and the meta data with each other
(that is, as obtaining correspondence between the frame image and
the meta data). For example, the video transmission unit 105 may
separately transmit the frame image and the meta data after giving
the time stamp to each of the frame image and the meta data. Thus,
the PC 110 at the reception side can obtain the meta data suitably
corresponding to the frame image by referring to the given time
stamp. In such a case, the time stamp is used as an example of the
time information.
[0054] The detail of the process to be performed by the object
detection unit 103 will be described with reference to FIGS. 4 and
5. That is, FIG. 4 is the flow chart indicating the process to be
performed by the object detection unit 103, and FIG. 5 is the
diagram for describing various modes of the object detection unit
103.
[0055] The object detection unit 103 sets plural initial frame
images as backgrounds and then obtains differences between the
initial frame images and a current frame image, thereby detecting
an object. In the present embodiment, the camera 100 has stored the
feature amounts of the plural frame images respectively as plural
states so that such a change in a case where trees tremble in
breeze can be set as the background. Here, it should be noted that
each of these states is called a mode. That is, the camera 100
holds the plural modes, and compares the plural held modes with the
feature amount of the current frame image, thereby judging whether
the detected object is a newly appeared object or the
background.
[0056] Initially, in a step S300, the object detection unit 103
obtains the feature amount for each block from the video
compression encoding unit 102, so as to process the images in units
of block. Here, in the JPEG method, color information of the image
is converted into data in a YUV color space. Then, the object
detection unit 103 divides the overall image into blocks each
consisting of 8 pixels.times.8 pixels (see a block 1000 in FIG. 5).
Further, the object detection unit 103 performs DCT (discrete
cosine transform: i.e., frequency analysis) to each block. Then,
the object detection unit 103 performs quantization so that an
obtained value decreases while becoming a higher frequency
component. After then, Huffman encoding is performed. Here, the
object detection unit 103 obtains a DCT coefficient quantized
before the Huffman encoding, as the feature amount.
[0057] Next, in a step S301, the object detection unit 103 compares
the modes (a block 1001 illustrated in FIG. 5) which are the
feature amounts already stored as the backgrounds with the obtained
feature amount. As described above, since the plural modes can be
held, the object detection unit 103 compares the obtained feature
amount with all of the already held modes. Here, a method of
performing such comparison is not specifically limited. For
example, the object detection unit 103 may perform the comparison
by using the sum of the differences of the luminance components.
More specifically, if it is assumed that a k-th coefficient of the
luminance component is Cy k, a current frame is i, and a frame in
the mode is m, then the summation of the differences Yk from k=0 to
k=N (i.e., from a low frequency to a high frequency) can be
expressed by the following equation (1).
K = 0 N Y k = C y _ k i - C y _ km ( 1 ) ##EQU00001##
[0058] Here, if this value is equal to or higher than a
predetermined threshold, the object detection unit 103 judges that
the features are not similar. Of course, the object detection unit
103 may further obtain the sums of the differences also for the
color components U and V respectively as expressed by the following
equations (2) and (3) and judge that the features are not similar
if at least one of these values is equal to or higher than the
predetermined threshold.
K = 0 N U k = C U _ ki - C U _ km ( 2 ) K = 0 N V k = C V _ ki - C
V _ km ( 3 ) ##EQU00002##
[0059] Next, in a step S302, the object detection unit 103 judges
whether or not a similar mode exists, based on the comparison
result in the step S301.
[0060] Next, in a step S303, if the similar mode does not exist,
the object detection unit 103 judges that the target mode is
equivalent to a new state, and thus adds it as a new mode. Here, as
indicated in the block 1001 illustrated in FIG. 5, each mode
includes a feature amount (DCT coefficient), a mode creation hour,
an active mode, and the number of times of appearance.
Incidentally, the modes have been stored in, e.g., the ROM, the RAM
or the like of the camera 100.
[0061] Then, in a step S304, the object detection unit 103 adds, to
the added mode, the hour when the mode was created (although the
current frame number is used in the present embodiment, an ordinary
hour expression such as "xx hours, yy minutes, zz seconds" may of
course be used). In a step S305, the object detection unit 103 sets
"1" as an initial value of the number of times of appearance to the
added mode. Further, in a step S306, the object detection unit 103
sets the active mode to "ON" as an initial value. Here, the state
that the mode is active implies that this mode corresponds to the
current frame.
[0062] After then, the object detection unit 103 returns the
process to the step S300.
[0063] Next, in a step S307, if the similar mode exists, the object
detection unit 103 sets the active mode of the relevant mode to
"ON". Then, in a step S308, the object detection unit 103 performs
increment of the number of times of appearance.
[0064] Then, in a step S309, the object detection unit 103 judges
whether or not the processes in the steps S300 to S308 have been
performed to all the blocks in the current frame image.
[0065] Next, in a step S310, the object detection unit 103 obtains,
from the results of the above-described processes, all the modes
that the active modes are "ON", with respect to each block.
[0066] Next, in a step S311, the object detection unit 103 performs
an area merging process to the block being the obtained active
mode. That is, if existing times of the modes for the adjacent
blocks are close to each other, the object detection unit 103
performs the area merging process by merging (integrating) the
relevant areas into the single area. The object detection unit 103
can obtain the existing time (although the number of the frames is
used in the present embodiment, an ordinary time expression such as
"xx seconds" may of course be used) by "the current hour (frame
number)"-"the mode creation hour (frame number) of each block". The
object detection unit 103 determines whether to perform the area
merging process based on whether or not the difference between the
existing times of the active modes of the respective block is
within a predetermined threshold.
[0067] Next, in a step S312, the object detection unit 103
discriminates for each of the areas merged in the step S311 whether
the relevant area is a background or a foreground (corresponding to
a newly appeared object). More specifically, the object detection
unit 103 calculates an average value of the existing times of the
active modes corresponding to the blocks included in each area.
Then, the object detection unit 103 sets the calculated average
value of the existing times which is less than the threshold, to
the foreground. That is, the since the background always exists,
the average existing time of the area is long. On the contrary, the
average existing time of the area concerning the foreground is
short.
[0068] Next, in a step S313, the object detection unit 103 obtains
one area which has been judged as the foreground. Further, the
object detection unit 103 performs a process of integrating the
information concerning the foreground area and outputting the
integrated information as object information (steps S314 to S318).
Here, it should be noted that the object information includes
coordinates (x1, y1) of the upper left point of a circumscribed
rectangle of the object, coordinates (x2, y2) of the lower right
point of the circumscribed rectangle, a square measure of the
object, an existing time of the object, and an activity ratio of
the object.
[0069] Next, in the step S314, the object detection unit 103 sets
the circumscribed rectangle of the foreground area, and obtains the
coordinates of the two points, i.e., the upper left point and the
lower right point, of the set circumscribed rectangle.
[0070] Next, in the step S315, the object detection unit 103 counts
the number of the blocks included in the foreground area, and
obtains the square measure of the object based on the counted
number.
[0071] Next, in the step S316, the object detection unit 103 sets
the average existing time of the foreground area as the existing
time of the object.
[0072] Next, in the step S317, the object detection unit 103
obtains an average of the numbers of times of appearance of the
corresponding modes of all the blocks in the foreground area. Then,
the object detection unit 103 sets the activity ratio by dividing
the obtained average by the existing time. That is, if the object
is stationary, it can be considered that the same block resultingly
exists for a long time. For this reason, the number of times of
appearance (the unit is the number of frames) approaches to the
existing time, whereby the activity ratio approaches to "1". Of
course, in a case where the object crosses, the existing time and
the number of times of appearance approach to each other, whereby
the activity ratio approaches to "1". However, the existing time is
very short in this case. Therefore, the object detection unit 103
can discriminate the state that the object crosses from the state
that the object is stationary. If the object moves about, the
existing time is long although the number of times of appearance is
small, whereby the activity ratio approaches to "0".
[0073] Next, in the step S318, the object detection unit 103 judges
whether or not the object information could be created by
performing the processes to all the foreground areas. Then, if
judged that the object information could be created by performing
the processes to all the foreground areas, the object detection
unit 103 ends the process illustrated in FIG. 4. On the other hand,
if judged that the object information cannot be created by
performing the processes to all the foreground areas, the object
detection unit 103 returns the process to the step S313 to obtain
one unprocessed foreground area, and then repeat the
above-described processes.
[0074] Subsequently, the detail of the process to be performed by
the event detection unit 104 will be described with reference to
FIG. 6. That is, FIG. 6 is the flow chart indicating an example of
the process of detecting misplacement as an event.
[0075] Initially, in a step S400, the event detection unit 104
obtains one object information.
[0076] Next, in a step S401, the event detection unit 104 judges
whether or not the square measure of the object is within a range
of a set size (set value). Then, if judged that the square measure
of the object is within the range of the set size, the event
detection unit 104 considers the object as a misplaced (or mislaid)
object. Here, the set value may previously be set in the apparatus.
However, since a condition of the size by which the object can be
considered as the misplaced object is different according to a set
condition or the like of the camera, it is possible to cause a user
to freely set the condition of the size.
[0077] Next, in a step S402, the event detection unit 104 judges
whether or not the existing time in the object information is equal
to or longer than a threshold T1. This is because, since the
misplaced object is the stationary object, the misplaced object has
to exist for a long time.
[0078] Next, in a step S403, the event detection unit 104 judges
whether or not the activity ratio in the object information is
equal to or larger than a threshold R1. This is because, since the
misplaced object is the stationary object, the activity ratio has
to approach to "1". Incidentally, if the misplaced object is always
imaged, "existing time"="threshold T1" is obtained, and also
"activity ratio"="1" is obtained. However, for example, if a person
passes in front of the misplaced object, the number of times of
appearance of the mode corresponding to the misplaced object
reduces for the time during which the misplaced object is hidden by
the person, whereby the activity ratio resultingly reduces. After
then, if the misplaced object is viewed for a sufficiently long
time, the number of times of appearance and the existing time
approach to each other, whereby "activity ratio">"threshold R1"
is obtained. At this time, since relation "existing
time">"threshold T1" is given, the event detection unit 104
comes to be able to detect the misplacement.
[0079] Next, if all the conditions in the steps S401 to S403 are
satisfied, then, in a step S404, the event detection unit 104
detects the misplacement of the object, and generates object meta
data on the RAM of the camera 100. Here, the object meta data is
one example of the event concern information. Incidentally, the
meta data which includes the object meta data and frame meta data
may be one example of the event concern information.
[0080] Here, the meta data which is output by the event detection
unit 104 will be described.
[0081] FIG. 7 is a diagram indicating one example of the meta
data.
[0082] Here, the meta data, which is the meta data for each frame,
includes the frame meta data and the object meta data for each
object.
[0083] Further, the frame meta data includes a current frame number
of the frame, and the number of objects included in the frame.
[0084] Furthermore, the object meta data includes an event ID which
is the information of the event to which the object is relevant,
coordinates of the circumscribed rectangle of the object, the
existing time, and the like.
[0085] Here, the event ID is an ID which has previously been
allocated to each classification of event. For example, "ID=1" is
defined as "intrusion", "ID=2" is defined as "misplacement", and
"ID=3" is defined as "carrying-away". In other words, the ID of the
phenomenon (event) detected by the event detection unit 104 is
described as the event ID. The coordinates of the circumscribed
rectangle of the object, and the existing time are copied from the
object information which is the target for the current process.
Incidentally, the event ID is one example of event discrimination
information, the existing time is one example of existing time
information, and the coordinates of the circumscribed rectangle are
one example of coordinate information.
[0086] The description is turned back again to FIG. 6. In a step
S405, the event detection unit 104 judges whether or not the
judgment process (i.e., the step S400 to the step S404) for judging
the misplaced object was performed to the overall object
information. If judged that the judgment process was performed to
the overall object information, the event detection unit 104
advances the process to a step S406. On the other hand, if judged
that the judgment process is not performed to the overall object
information, the event detection unit 104 returns the process to
the step S400.
[0087] In the step S406, the event detection unit 104 generates, as
no event, the object meta data on the RAM of the camera 100 for all
the objects other than the object judged as the misplaced object.
Incidentally, the event detection unit 104 adds a specific ID such
as "0" or the like as the event ID so as to indicate that there is
no event. On the other hand, as well as the step S404, the
coordinates of the circumscribed rectangle and the existing time
are copied from the object information.
[0088] Next, in a step S407, the event detection unit 104 generates
the frame meta data on the RAM of the camera 100. Further, the
event detection unit 104 adds the current hour (frame number) and
the number of the objects, as the number of objects.
[0089] Next, in a step S408, the event detection unit 104
integrates the frame meta data and all the object meta data, and
outputs the integrated data as the meta data.
[0090] One example of the process to be performed by the event
detection unit 104 is explained as described above. Thus, the event
detection unit 104 detects the event by judging whether or not the
conditions in the steps S401 to S403 are satisfied, with respect to
the coordinates of the circumscribed rectangle, the square measure,
the existing time, and the activity ratio being output as the
object information. Here, another example is given. That is, in
case of the intrusion, the event detection unit 104 causes the user
to designate an area such as a door or the like in the screen at
which the user wishes to detect the intrusion. Then, the event
detection unit 104 detects the intrusion according to a condition
as to whether or not the circumscribed rectangle of the object is
included in the designated area. Further, the camera 100 may be
constituted so that the user can freely set whether or not to
perform a process of judging whether or not the above condition is
satisfied.
[0091] Next, the process to be performed by the PC 110 will be
described with reference to FIG. 8. That is, FIG. 8 is the flow
chart for describing the process to be performed by the PC
according to the first embodiment.
[0092] Initially, in a step S600, the video reception unit 111
receives the frame image transmitted from the camera 100 through
the network.
[0093] Next, in a step S601, the video recording unit 112 stores
and accumulates the received frame image in the video accumulation
unit 113.
[0094] Next, in a step S602, the video recording unit 112 displays
the received frame image on the display unit 115. Thus, the user
can confirm the video which is currently imaged by the camera.
Here, an example of the screen at that time is illustrated in FIG.
9A. Namely, FIGS. 9A and 9B are the diagrams respectively
illustrating examples of the screens. More specifically, the
example illustrated in FIG. 9A includes a main window 300, and an
area 301 in which the current frame image obtained by the camera
100 is displayed.
[0095] Next, in a step S603, the event occurrence time point
determination unit 114 obtains the meta data from the header
portion of the frame image received by the video reception unit
111.
[0096] Next, in a step S604, the event occurrence time point
determination unit 114 sets the value of the number of objects in
the frame meta data of the meta data to "N".
[0097] Next, in a step S605, the event occurrence time point
determination unit 114 obtains one object meta data of the meta
data in rank order.
[0098] Next, in a step S606, the event occurrence time point
determination unit 114 judges whether or not there is an event, by
referring to the event ID. Then, if judged that the event ID is "0"
(i.e., no event), the event occurrence time point determination
unit 114 directly advances the process to a step S613 by skipping
over steps S607 to S612.
[0099] On the other hand, if judged that the event ID is not "0",
in the step S607, the event occurrence time point determination
unit 114 judges whether or not the event is an event of a
detection-delayed type. Here, the event of the detection-delayed
type is the event that the detection is performed after the hour
when the event occurred. For example, "misplacement" and
"carrying-away" are included in the event of the detection-delayed
type. The PC 110 has previously stored the event ID of the event of
the detection-delayed type in the form of a table in the RAM 803,
the ROM 802, the external storage apparatus 806 or the like. Thus,
the event occurrence time point determination unit 114 can detect
whether or not the detected event is the event of the
detection-delayed type, by referring to the table. If judged that
the detected event is the event of the detection-delayed type, the
event occurrence time point determination unit 114 advances the
process to a step S608. On the other hand, if judged that the
detected event is not the event of the detection-delayed type, the
event occurrence time point determination unit 114 advances the
process to a step S609.
[0100] In the step S608, the event occurrence time point
determination unit 114 reads, from the video accumulation unit 113,
the frame image at the hour (frame number) obtained by subtracting
the existing time (the number of frames) from the current hour
(current frame number). Thus, the read frame image is the frame
image at the time when the event occurs. If the event is the
misplacement, the read frame image is the frame image at the time
when the misplacement occurs. On the other hand, as in the
intrusion, if timing of the occurrence of the event corresponds to
timing of the detection of the event, in the step S609, the event
occurrence time point determination unit 114 obtains the frame
image of the current hour (current frame number) from the video
accumulation unit 113.
[0101] Next, in a step S610, the display unit 115 displays, on the
display 807 or the like, a window which includes the frame image at
the time when the event occurs.
[0102] FIG. 9B illustrates one example of the window to be
displayed as a popup at the time of warning.
[0103] When the event occurs, the display unit 115 displays a sub
window 700 as a popup, independently of the main window 300. Here,
the sub window 700 includes an area 702 for displaying the frame
image at the time when the event occurs, a bar 702 for displaying a
message indicating a classification of the event as described
later, and a frame 702 which indicates the circumscribed rectangle
of the object area with which the event is associated as described
later and is superposed on the frame image. Incidentally, FIG. 9B
illustrates the example of the window to be displayed when the
misplacement is detected.
[0104] The description is turned back again to FIG. 8. In a step
S611, the display unit 115 displays the frame 703 indicating the
circumscribed rectangle as superposing it on the frame image at the
time when the event occurs, by referring to the circumscribed
rectangle included in the object meta data currently processed.
[0105] Next, in the step S612, the display unit 115 displays on the
bar 702 the message indicating the classification of the event,
such as "MISPLACEMENT IS DETECTED!", "MISPLACEMENT EVENT DETECTION
OCCURS!", or the like.
[0106] Next, in the step S613, the event occurrence time point
determination unit 114 performs decrement of "N". Then, in a step
S614, the event occurrence time point determination unit 114 judges
whether or not "N" becomes "0". That is, the PC 110 performs the
processes from the step S605 to the step S612 for all the object
meta data.
[0107] Incidentally, it should be noted that the purpose of
displaying the frame in the step S611 and the message in the step
S612 is to enable the user to easily understand the displayed
contents. Therefore, the PC 110 may only display the frame image at
the time when the event occurs, by omitting the processes in the
steps S611 and S612. In such a case, the coordinates of the
circumscribed rectangle of the object meta data is unnecessary.
[0108] As just described, the camera 100 detects the existing time
of the object and the event, and outputs them as the meta data.
Thus, in the case where the event is the event of the
detection-delayed type, the PC 110 can display the video at the
time of the event occurrence on the basis of the existing time and
the current hour. Therefore, the user can easily grasp the aspect
at the time when the event occurred. For example, if the event is
the misplacement, since it is possible to display the video at the
time when the person misplaced the object, the user can easily
confirm who misplaced the object.
[0109] FIG. 10 is a block diagram illustrating an example of a
system configuration of an image processing system according to the
second embodiment, and functional constitutions of apparatuses
included in the image processing system.
[0110] Here, in the second embodiment, the constituent elements
same as those in the first embodiment are added with the same
numerals respectively, and the descriptions thereof will be
omitted.
[0111] However, in the second embodiment, the event detection unit
104 considers, as an event, the time point when an object appears
in the frame image, and detects "appearance" as the event. In the
object detection unit 103, when the object appears in the frame
image, "the number of times of appearance"="1" is given, and "the
existing time"="0" is given, whereby it is possible to easily
detect such an appearance event. Incidentally, in the second
embodiment, the object existing time in the meta data is
unnecessary, because it is not used.
[0112] In the second embodiment, a meta data accumulation unit 1116
is additionally provided in the PC 110. Here, the meta data
accumulation unit 1116, which is constituted by the external
storage apparatus 806 or the RAM 803, sequentially accumulates the
meta data analyzed by the event occurrence time point determination
unit 114.
[0113] Subsequently, the process to be performed by the PC 110 will
be described with reference to FIG. 11. That is, FIG. 11 is the
flow chart for describing the process to be performed by the PC
according to the second embodiment.
[0114] Here, in this flow chart of the second embodiment, the steps
same as those in the flow chart of the first embodiment are added
with the same step numbers respectively, and the descriptions
thereof will be omitted.
[0115] In a step S1301, the event occurrence time point
determination unit 114 judges whether or not an event other than
"appearance" is detected, by referring to the event ID. If judged
that the event other than "appearance" is detected, the event
occurrence time point determination unit 114 advances the process
to the step S607. On the other hand, if judged that the event other
than "appearance" is not detected, the event occurrence time point
determination unit 114 advances the process to the step S613. More
specifically, it is assumed that the event ID of "appearance" is
"99". In this case, if the event ID="0" (no event) or the event
ID="99", the event occurrence time point determination unit 114
advances the process to the step S613.
[0116] If judged that the event other than "appearance" is
detected, in the step S607, the event occurrence time point
determination unit 114 judges whether or not the event is the event
of the detection-delayed type. If judged that the event is the
event of the detection-delayed type, the event occurrence time
point determination unit 114 advances the process to a step S1302.
On the other hand, if judged that the event is not the event of the
detection-delayed type, the event occurrence time point
determination unit 114 advances the process to the step S609.
[0117] In the step S1302, the event occurrence time point
determination unit 114 searches for, from among the object meta
data of the objects at the same position, the object meta data
having the "appearance" event, by referring to the circumscribed
rectangles of the past meta data from the meta data accumulation
unit 1116. Incidentally, the event occurrence time point
determination unit 114 judges, based on whether or not the central
positions of the two circumscribed rectangles are within a
predetermined value, whether or not the two objects are at the same
position. The event occurrence time point determination unit 114
obtains, by referring to the frame meta data including the object
meta data having the "appearance" event, the hour (frame number) of
the relevant frame, that is, the hour (event occurrence time) when
the object appears. Incidentally, the object here is the misplaced
object, and the hour when the misplaced object occurs is the hour
at the time point when the misplacement occurs. Thus, in a step
S1303, the event occurrence time point determination unit 114 reads
the frame image at the appearance hour from the video accumulation
unit 113. Here, the hour of the frame (frame number) is one example
of hour information. After the processes in the steps S600 to S614,
in a step S1304, the event occurrence time point determination unit
114 accumulates the meta data being the target of the process in
the meta data accumulation unit 1116.
[0118] As above, the misplacement is exemplarily described.
However, for example, in the case where carrying-away occurs, the
event occurrence time point determination unit 114 may detect as
the event the time point when the object is "carried away", and add
the detected event to the meta data. Incidentally, in the object
detection unit 103, the object is detected based on a difference
between the object and the background image. For this reason, also
in the case where the previously existed object is carried away
(disappeared), the relevant object is detected as the carried away
(disappeared) object. Therefore, even in the case of
"carrying-away", it is possible to perform the above-described
processes as well as the case of "misplacement".
[0119] As just described, the camera 100 is constituted to also
detect "appearance" as the event and thus output the detected event
as the meta data together with the position of the object. Thus, in
the case of the event of the detection-delayed type, the PC 110 can
display the video at the appearance hour (event occurrence time
point) by searching for the meta data of the objects at the same
position and having the "appearance" event from the accumulated
meta data. Therefore, a user can easily grasp an aspect at the time
point of the occurrence of the event. For example, if the event is
the "misplacement", it is possible to display the video at the time
point when a person misplaced the object, whereby the user can
easily confirm who misplaced the object.
[0120] FIG. 12 is a block diagram illustrating an example of a
system configuration of an image processing system according to the
third embodiment, and functional constitutions of apparatuses
included in the image processing system.
[0121] Here, in the third embodiment, an object tracking unit 1206
is added to the camera 100 according to the second embodiment.
Thus, the constituent elements same as those in the second
embodiment (and the first embodiment) are added with the same
numerals respectively, and the descriptions thereof will be
omitted.
[0122] In the present embodiment, since the object tracking unit
1206 is additionally provided, it is possible to confirm whether or
not the object is the same object even in a case where the relevant
object is moving. Thus, for example, as illustrated in FIG. 13, in
a case where the camera 100 detects, as a specific entrance and
exit event (entrance and exit phenomenon), a case where a person
enters a room through a specific door 902 and then exits through a
specific door 903, the PC 110 can display a momentary video that
the person entered the room. Here, it should be noted that FIG. 13
is the drawing for describing the specific entrance and exit
event.
[0123] The process to be performed at that time by the camera 100
will be described with reference to FIG. 14. That is, FIG. 14 is
the flow chart for describing the process to be performed by the
camera according to the third embodiment.
[0124] However, in this flow chart of the third embodiment, the
steps same as those in the flow chart of the first embodiment are
added with the same step numbers respectively, and the descriptions
thereof will be omitted. That is, in the third embodiment, after
the object is detected in the same manner as that in the first
embodiment, the object tracking unit 1206 tracks in a step S1401
which object in the past frame corresponds to which object in the
current frame. Further, the object tracking unit 1206 gives the
same object ID to the corresponding object (step S1401). Then, in
the step S204, the object ID is given or added as the object meta
data of the output meta data. Here, the object ID is one example of
object discrimination information.
[0125] The detail of the process to be performed by the object
tracking unit 1206 will be described.
[0126] The object tracking unit 1206 reads the object information
stored, by the object detection unit 103, in the RAM of the camera
100 by an amount corresponding to a past predetermined time, and
then obtains the central position of the object of the past frame
and the central position of the object of the current frame. Next,
the object tracking unit 1206 obtains the distances between the
central positions of the objects of the past frame and the central
positions of the objects of the current frame for all the
combinations of the objects of the past frame and the objects of
the current frame. Subsequently, the object tracking unit 1206
considers, as the same object, the objects included in the
combination of the objects of the past and current frames that the
distance between their centers is shortest, and assigns or adds the
same object ID to these objects.
[0127] Incidentally, in the above-described process, the object
tracking unit 1206 considers, as the same object, the two objects
that the distance between them is shortest. However, the object
tracking unit 1206 may consider these objects as the same object,
in another way. For example, the object tracking unit 1206 may
extract features such as luminance, color tint and the like from
the detected object area, and consider, as the same object, the
objects having the similar features.
[0128] Next, the detail of the process to be performed to detect
the specific entrance and exit event in the event detection unit
104 will be described with reference to a flow chart illustrated in
FIG. 15. That is, FIG. 15 is the flow chart indicating an example
of the process of detecting specific entrance and exit as the
event. Here, the specific entrance and exit event will be defined
anew with reference to FIG. 13, which illustrates an overall frame
image 900, the doors 902 and 903, and a person 901. Here, only in a
case where the person 901 enters the room through the door 902 and
then exits through the door 903, this phenomenon is defined as the
specific entrance and exit event. To detect this event, the user
previously designates the areas of the doors 902 and 903.
[0129] Next, the process to be performed will be described.
However, in this flow chart, the steps same as those in the flow
chart of FIG. 6 according to the first embodiment are added with
the same step numbers respectively, and the descriptions thereof
will be omitted.
[0130] In a step S1601, the event detection unit 104 judges whether
or not the object exists within the area of the door 902, by
referring to the coordinates of the circumscribed rectangle in the
object information obtained in the step S400. If judged that the
object exists within the area of the door 902, the event detection
unit 104 advances the process to a step S1602. On the other hand,
if judged that the object does not exist within the area of the
door 902, the event detection unit 104 advances the process to a
step S1604.
[0131] If the object exists within the area of the door 902, there
is a possibility that the relevant object is the object (person)
which entered the room through the door 902. Thus, in the step
S1602, the event detection unit 104 judges whether or not the
object ID has been temporarily stored in the RAM of the camera 100
as an already "in-object" (i.e., the person who entered the room).
Then, if judged that the object ID has been already stored as the
"in-object", the event detection unit 104 advances the process to
the step S1604. On the other hand, if judged that the object ID is
not stored as the "in-object", the event detection unit 104
advances the process to a step S1603.
[0132] In the step S1603, the event detection unit 104 temporarily
stores, as the "in-object", the object ID of the currently
processed object information in the RAM of the camera 100.
[0133] In the step S1604, the event detection unit 104 judges
whether or not the object exists within the area of the door 903,
by referring to the coordinates of the circumscribed rectangle in
the object information. If judged that the object exists within the
area of the door 903, the event detection unit 104 advances the
process to a step S1605. On the other hand, if judged that the
object does not exist within the area of the door 903, the event
detection unit 104 advances the process to the step S405.
[0134] If the object exists within the area of the door 903, there
is a possibility that the relevant object is the object (person)
which intends to exit through the door 903. Thus, in the step
S1605, the event detection unit 104 judges whether or not the
object ID of the currently processed object information has been
temporarily stored already as the "in-object" in the RAM of the
camera 100. This is the important judgment to discriminate the
person who intends to exit from the person who entered through the
door 903. Then, if judged that the object ID has been already
stored as the "in-object", the event detection unit 104 advances
the process to the step S405. On the other hand, if judged that the
object ID is not stored as the "in-object", the event detection
unit 104 advances the process to the step S1606.
[0135] If the object ID is the object ID which has been already
stored temporarily as the "in-object", the relevant object is the
person who entered through the door 902 and intends to exit through
the door 903. Therefore, in the step S1606, the event detection
unit 104 detects the specific entrance and exist event, and adds
the predetermined event ID to the object meta data. Incidentally,
as well as the second embodiment, the object existing time in the
meta data is not used in the third embodiment, whereby the meta
data may not include the object existing time.
[0136] In a step S1607, the event detection unit 104 deletes the
currently processed object ID stored as the "in-object", from the
RAM of the camera 100.
[0137] Next, the process to be performed by the PC 110 will be
described with reference to a flow chart illustrated in FIG. 16.
That is, FIG. 16 is the flow chart for describing the process to be
performed by the PC according to the third embodiment.
[0138] However, in this flow chart, the steps same as those in the
flow chart according to the first embodiment are added with the
same step numbers respectively, and the descriptions thereof will
be omitted.
[0139] In a step S1501, the event occurrence time point
determination unit 114 searches for first object meta data which
includes the object ID same as the object ID of the object being
the target of the current process, by referring to the object ID of
the past meta data in the meta data accumulation unit 1116. The
event occurrence time point determination unit 114 refers to the
frame meta data corresponding to the first object meta data for the
same object. Thus, the event occurrence time point determination
unit 114 can obtain the hour (frame number) of the relevant frame,
i.e., the hour when the object appeared. Here, in the present
embodiment, the object is the misplaced object, and the hour when
the misplaced object occurred is, i.e., the hour when the
misplacement occurred. Thus, the event occurrence time point
determination unit 114 reads the frame image at the appearance hour
from the video accumulation unit 113 (step S1502).
[0140] After the processes in the steps S600 to S614, in a step
S1503, the event occurrence time point determination unit 114
accumulates the meta data being the target of the process in the
meta data accumulation unit 1116.
[0141] As described above, even if the object is the moving object,
it is possible to discriminate whether the target objects are the
same object by additionally providing the object tracking unit 1206
to the camera 100. The camera 100 outputs the object ID in the form
of the meta data as the information indicating that the target
objects have been considered as the same object. Thus, in the case
of the event of the detection-delayed type, the PC 110 can display
the video at the object appearance hour (event occurrence time
point) by searching for the first meta data of the same object from
the accumulated meta data. Therefore, a user can easily grasp an
aspect at the time point of the occurrence of the event. For
example, if the event is the specific entrance and exist event, it
is possible to display the video at the time point when a person
enters, whereby the user can easily confirm who entered the
room.
[0142] In the first to third embodiments, as described with
reference to FIG. 8, if the event is the event of the
detection-delayed type such as the misplacement detection, the PC
110 displays the frame image at the time of the occurrence of the
misplacement (steps S608, S610). However, the PC 110 may display
not only one frame image at the time of the occurrence of the
misplacement but also sequentially plural frame images included in
a predetermined interval between a time immediately before the
occurrence of the misplacement and a time immediately after the
occurrence of the misplacement. By doing so, the user can further
confirm in detail the aspect at the time when the misplacement
occurred. Incidentally, the interval may be determined by either a
method of always using a fixed time (for example, one second before
and two second after the time point of occurrence) as the interval
or a method of using a fixed time determined according to the
classification of event (for example, one second before and two
second after the occurrence of the misplacement, three seconds
after the occurrence of the carrying-away, or the like).
[0143] Further, in the first to third embodiments, the warning
screen as illustrated in FIG. 9B is displayed as the popup.
However, of course, the screen to be displayed is not limited to
this. That is, the frame image at the time of the occurrence of the
event may be presented to the user. For example, the PC 110 may
dispose the frame image at the time of the occurrence of the event,
nearby the current image of the camera within the main window.
[0144] Further, although the existing time is used in the first
embodiment, the PC 110 may of course us the occurrence hour (mode
creation hour) as it is.
[0145] Furthermore, in the first to third embodiments, the object
detection unit 103 performs the object detection by using the
feature amount (DCT coefficient) generated when the encoding is
performed by the video compression encoding unit 102. However, of
course, the object detection unit 103 may directly extract the
feature amounts such as luminance, edge, color tint and the like
from the image data imaged and obtained by the image pickup unit
101. Further, the block is not limited to the 8.times.8 pixels.
That is, the block may be N.times.N pixels (N is an arbitrary
number), N.times.M pixels (N, M are arbitrary numbers, but N # M).
Furthermore, the object detection unit 103 may perform the object
detection process for each pixel.
[0146] Furthermore, in the first to third embodiments, the object
detection unit 103 has the plural modes. However, if high accuracy
is not required in such a case where the camera is used in the
stable circumstances, the object detection unit 103 may use only a
single mode. In this case, it is unnecessary to hold the active
mode and the number of times of appearance in the mode.
[0147] Furthermore, in the first to third embodiments, the object
detection unit 103 has the DCT coefficient itself as the feature
amount in the mode. However, the object detection unit 103 may
have, in the mode, the difference amounts respectively obtained in
the above-described equations (1) to (3). In this case, the object
detection unit 103 may judge whether or not the similar mode exists
(step S302), by judging whether or not the difference between the
difference amounts is within a threshold.
[0148] Furthermore, in the first to third embodiments, the object
detection unit 103 discriminates whether the area is the background
or the foreground (step S312), after merging the blocks having the
same existing time (step S311). However, the object detection unit
103 may merge only the blocks respectively having the existing
times longer than a predetermined time.
[0149] Furthermore, in the first to third embodiments, the number
of times of appearance is included in the mode. This may not be
included in the mode, if the activity ratio is not used in the
event detection unit 104.
[0150] Furthermore, in the case where the object tracking unit 1206
is additionally provided in the camera 100, the object detection
unit 103 may detect the object by obtaining a difference amount
from the background image by a difference of luminance for each
pixel, and combining the pixels having the same difference
amount.
[0151] Furthermore, in the first to third embodiments, the camera
100 and the PC 110 mutually exchange the meta data in the
communication. On the other hand, the video compression encoding
unit 102, the object detection unit 103 and the event detection
unit 104 may be provided on the side of the PC 110. In such a case,
the video transmission unit 105, the network and the video
reception unit 111 are unnecessary in the image processing system.
Incidentally, in the constitution like this, it is necessary to
further add a video interface to the PC 110 to be able to capture
the frame image from the camera 100 through a coaxial cable or the
like.
[0152] As just described, according to the embodiments of the
present invention, even if the event is detected after the
occurrence of the relevant event, the PC 110 can display the video
at the time of the occurrence of the relevant event. For this
reason, the user can easily grasp the aspect at the time point of
the occurrence of the relevant event. For example, in case of the
misplacement, since the PC 110 can display the video at the time
point when the person misplaced the object, the user can confirm
who misplaced the object.
[0153] That is, according to the above-described embodiments, it is
possible to perform the display by which the user can easily
confirm the event, even if the timing of the event detection is
different from the timing of the event occurrence.
[0154] Various exemplary embodiments, features, and aspects of the
present invention will now be herein described in detail below with
reference to the drawings. It is to be noted that the relative
arrangement of the components, the numerical expressions, and
numerical values set forth in these embodiments are not intended to
limit the scope of the present invention.
[0155] Aspects of the present invention can also be realized by a
computer of a system or an apparatus (or a device such as a CPU or
an MPU) that reads out and executes a program recorded on a memory
device to perform the functions of the above-described embodiments,
and by a method, the steps of which are performed by a computer of
a system or an apparatus by, for example, reading out and executing
a program recorded on a memory device to perform the functions of
the above-described embodiments. For this purpose, the program is
provided to the computer for example via a network or from a
recording medium of various types serving as the memory device
(e.g., a computer-readable medium).
[0156] While the present invention has been described with
reference to the exemplary embodiments, it is to be understood that
the invention is not limited to the disclosed exemplary
embodiments. The scope of the following claims is to be accorded
the broadest interpretation so as to encompass all such
modifications and equivalent structures and functions.
[0157] This application claims the benefit of Japanese Patent
Application No. 2009-240655, filed Oct. 19, 2009, which is hereby
incorporated by reference herein in its entirety.
* * * * *