U.S. patent application number 17/590190 was filed with the patent office on 2022-05-19 for commodity monitoring device, commodity monitoring system, output destination device, commodity monitoring method, display method and program.
This patent application is currently assigned to NEC Corporation. The applicant listed for this patent is NEC Corporatlion. Invention is credited to Nobuyki Ikuta, Takayuki NAKAGAWA, Satoshi Ohkubo, Kenichi Oohashi, Yuma Shibuya, Hiroshi Tezuka, Shigeaki Tokuda, Junpei Yamasaki.
Application Number | 20220156686 17/590190 |
Document ID | / |
Family ID | 1000006114423 |
Filed Date | 2022-05-19 |
United States Patent
Application |
20220156686 |
Kind Code |
A1 |
NAKAGAWA; Takayuki ; et
al. |
May 19, 2022 |
COMMODITY MONITORING DEVICE, COMMODITY MONITORING SYSTEM, OUTPUT
DESTINATION DEVICE, COMMODITY MONITORING METHOD, DISPLAY METHOD AND
PROGRAM
Abstract
A commodity monitoring device includes a commodity state
detection unit configured to, on the basis of image information of
a commodity group area set on a commodity shelf, detect removal of
a commodity in a quantity satisfying a predetermined condition from
the commodity group area; and a change detection information output
unit configured to, if the commodity state detection unit has
detected the removal, output to an output destination device change
detection information including image time-series data of the
commodity group area from a point in time at which the commodity
state detection unit detected the removal to a predetermined past
point in time.
Inventors: |
NAKAGAWA; Takayuki; (Tokyo,
JP) ; Shibuya; Yuma; (Tokyo, JP) ; Tokuda;
Shigeaki; (Tokyo, JP) ; Yamasaki; Junpei;
(Tokyo, JP) ; Ohkubo; Satoshi; (Tokyo, JP)
; Tezuka; Hiroshi; (Tokyo, JP) ; Ikuta;
Nobuyki; (Tokyo, JP) ; Oohashi; Kenichi;
(Tokyo, JP) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
NEC Corporatlion |
Tokyo |
|
JP |
|
|
Assignee: |
NEC Corporation
Tokyo
JP
|
Family ID: |
1000006114423 |
Appl. No.: |
17/590190 |
Filed: |
February 1, 2022 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
16490398 |
Aug 30, 2019 |
11270257 |
|
|
PCT/JP2017/044360 |
Dec 11, 2017 |
|
|
|
17590190 |
|
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06Q 20/208 20130101;
G06Q 10/087 20130101; G08B 13/19613 20130101; G06Q 20/203 20130101;
H04W 4/35 20180201 |
International
Class: |
G06Q 10/08 20060101
G06Q010/08; H04W 4/35 20060101 H04W004/35; G06Q 20/20 20060101
G06Q020/20; G08B 13/196 20060101 G08B013/196 |
Foreign Application Data
Date |
Code |
Application Number |
Mar 6, 2017 |
JP |
2017-042079 |
Claims
1. A commodity monitoring system comprising: at least one memory
configured to store instructions; and at least one processor
configured to execute the instructions to: detect an unauthorized
action based on image information; and cause an output destination
device, to display an image included in time-series data of the
image information from a point in time at which the unauthorized
removal has been detected to a predetermined past point in time,
and a registration screen for inputting loss information.
2. The commodity monitoring system according to claim 1, wherein
the unauthorized action includes an unauthorized removal of a
commodity.
3. The commodity monitoring system according to claim 2, wherein
the unauthorized removal of the commodity indicates a removal of
the commodity in a quantity satisfying a predetermined
condition.
4. The commodity monitoring system according to claim 1, wherein
the loss information includes status information.
5. The commodity monitoring system according to claim 1, wherein
the at least one processor is further configured to execute the
instructions to cause the output destination device, to display an
attribute information of a person.
6. An output destination device, comprising: a display for
displaying an image included in time-series data of image
information from a point in time at which an unauthorized removal
has been detected based on the image information to a predetermined
past point in time, and a registration screen for inputting loss
information.
7. A commodity monitoring method comprising: detecting an
unauthorized action based on image information; and causing an
output destination device, to display an image included in
time-series data of the image information from a point in time at
which the unauthorized removal has been detected to a predetermined
past point in time, and a registration screen for inputting loss
information.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] The present application is a continuation application of
U.S. patent application Ser. No. 16/490,398 filed on Aug. 30, 2019,
which is a National Stage Entry of international application
PCT/JP2017/044360, filed on Dec. 11, 2017, which claims the benefit
of priority from Japanese Patent Application 2017-042079 filed on
Mar. 6, 2017, the disclosures of all of which are incorporated in
their entirety by reference herein.
TECHNICAL FIELD
[0002] The present invention relates to a commodity monitoring
device, a commodity monitoring system, an output destination
device, a commodity monitoring method, a display method and a
program.
BACKGROUND ART
[0003] Several techniques have been proposed in connection with the
management of commodities placed on commodity shelves.
[0004] For example, in the stock state management method disclosed
in Patent Document 1, when an image of a display shelf for
commodities (commodity shelf) is monitored and an out-of-commodity
marker provided on the display shelf is detected, the commodity
associated with the out-of-commodity marker is identified. Then, a
process is performed to superimpose a message indicating that a
stock check is underway at a position in the image of the display
shelf where the out-of-commodity marker is detected.
CITATION LIST
Patent Literature
[0005] [Patent Document 1] Japanese Unexamined Patent Application,
First Publication No. 2012-174154
SUMMARY OF INVENTION
Problems to be Solved by the Invention
[0006] In a store that sells commodities, it is preferable to be
able to detect, at a cost as low as possible, an event in which a
commodity is expected to be brought out of the store improperly. In
this respect, Patent Document 1 does not disclose a method for
detecting an event in which a commodity is expected to be brought
out of the store improperly.
[0007] An object of the present invention is to provide a commodity
monitoring device, a commodity monitoring system, an output
destination device, a commodity monitoring method, a display method
and a program that can solve the above-mentioned problem.
Means for Solving the Problems
[0008] According to an exemplary first aspect of the present
invention, a commodity monitoring device includes with a commodity
state detection unit configured to, on the basis of image
information of a commodity group area set on a commodity shelf,
detect removal of a commodity in a quantity satisfying a
predetermined condition from the commodity group area, and a change
detection information output unit configured to, if the commodity
state detection unit has detected the removal, output to an output
destination device change detection information including image
time-series data of the commodity group area from a point in time
at which the commodity state detection unit detected the removal to
a predetermined past point in time.
[0009] According to an exemplary second aspect of the present
invention, a commodity monitoring system is provided with a
commodity monitoring device and an output destination device, the
commodity monitoring device including a commodity state detection
unit configured to, on the basis of image information of a
commodity group area set on a commodity shelf, detect removal of a
commodity in a quantity satisfying a predetermined condition from
the commodity group area; and a change detection information output
unit configured to, if the commodity state detection unit has
detected the removal, output to the output destination device
change detection information including image time-series data of
the commodity group area from a point in time at which the
commodity state detection unit detected the removal to a
predetermined past point in time.
[0010] According to an exemplary third aspect of the present
invention, an output destination device includes a change detection
information acquisition unit configured to, when removal of a
commodity in a quantity satisfying a predetermined condition from a
commodity group area set on a commodity shelf is detected, acquire
change detection information including image time-series data of
the commodity group area from a point in time at which the removal
was detected to a predetermined past point in time; and a display
unit configured to display the image of the commodity group area
included in the change detection information.
[0011] According to an exemplary fourth aspect of the present
invention, a commodity monitoring method includes the step of:
when, based on image information of a commodity group area set on a
commodity shelf, removal of a commodity in a quantity satisfying a
predetermined condition from the commodity group area is detected,
outputting to an output destination device change detection
information including image time-series data of the commodity group
area from a point in time at which the removal was detected to a
predetermined past point in time.
[0012] According to an exemplary fifth aspect of the present
invention, a display method includes the steps of: when removal of
a commodity in a quantity satisfying a predetermined condition from
a commodity group area set on a commodity shelf is detected,
acquiring change detection information including image time-series
data of the commodity group area from a point in time at which the
removal was detected to a predetermined past point in time; and
displaying the image of the commodity group area included in the
change detection information.
[0013] According to an exemplary sixth aspect of the present
invention, a program causes a computer to execute a process
including the step of: when, based on image information of a
commodity group area set on a commodity shelf, removal of a
commodity in a quantity satisfying a predetermined condition from
the commodity group area is detected, outputting to an output
destination device change detection information including image
time-series data of the commodity group area from a point in time
at which the removal was detected to a predetermined past point in
time.
[0014] According to an exemplary seventh aspect of the present
invention, a program causes a computer to execute a process
including the steps of: when removal of a commodity in a quantity
satisfying a predetermined condition from a commodity group area
set on a commodity shelf is detected, acquiring change detection
information including image time-series data of the commodity group
area from a point in time at which the removal was detected to a
predetermined past point in time; and displaying the image of the
commodity group area included in the change detection
information.
Advantageous Effects of Invention
[0015] According to the present invention, it is possible to detect
at a comparatively low cost an event in which a commodity is
expected to be brought out of a store improperly.
BRIEF DESCRIPTION OF DRAWINGS
[0016] FIG. 1 is a schematic block diagram showing a device
configuration example of the commodity monitoring system according
to one embodiment.
[0017] FIG. 2 is a schematic block diagram showing an example of
the function configuration of the commodity monitoring device
according to the same embodiment.
[0018] FIG. 3 is a drawing showing an example of setting of
commodity group areas according to the same embodiment.
[0019] FIG. 4 is a schematic block diagram showing an example of a
functional configuration of the terminal device according to the
same embodiment.
[0020] FIG. 5 is a drawing showing an example of a notification
screen displayed by the second display unit according to the same
embodiment.
[0021] FIG. 6 is a drawing showing an example of a detection
information screen displayed by the second display unit according
to the same embodiment.
[0022] FIG. 7 is a drawing showing a first example of the details
registration screen displayed by the second display unit according
to the same embodiment.
[0023] FIG. 8 is a drawing showing a second example of the details
registration screen displayed by the second display unit according
to the same embodiment.
[0024] FIG. 9 is a drawing showing an example of the loss
information list screen displayed by the second display unit
according to the same embodiment.
[0025] FIG. 10 is a drawing showing an example of the sales history
screen displayed by the second display unit according to the same
embodiment.
[0026] FIG. 11 is a drawing showing an example of a flow line
screen displayed by the second display unit according to the same
embodiment.
[0027] FIG. 12 is a diagram showing an example of operation of the
commodity monitoring system in the case of detecting removal in the
same embodiment.
[0028] FIG. 13 is a flowchart showing an example of the processing
procedure in which the commodity monitoring device determines the
presence or absence of removal according to the same
embodiment.
[0029] FIG. 14 is a drawing showing an example of the minimum
configuration of the commodity monitoring device according to the
present embodiment.
[0030] FIG. 15 is a drawing showing an example of the minimum
configuration of the commodity monitoring system according to the
present embodiment.
[0031] FIG. 16 is a drawing showing an example of the minimum
configuration of the output destination device according to the
present embodiment.
DESCRIPTION OF EMBODIMENTS
[0032] The best mode for carrying out the present invention will
hereinbelow be described in detail. The following embodiments are
not intended to limit the scope of the claims of the present
invention. At the same time, not all combinations of features
described in the embodiments are essential to the solving means of
the present invention.
[0033] FIG. 1 is a schematic block diagram showing a device
configuration example of the commodity monitoring system according
to the embodiment. In the example of FIG. 1, the commodity
monitoring system 10 is provided with a shelf monitoring camera 11,
a hub 12, a commodity monitoring device 13, a wireless router
device 14, a terminal device 15, an in-store monitoring camera 16,
and a POS (Point Of Sales) system 17.
[0034] Further, FIG. 1 shows a commodity shelf 910 on which the
commodity to be monitored by the commodity monitoring system 10 is
displayed. The number of commodity shelves 910 targeted by the
commodity monitoring system 10 may be one or more.
[0035] Further, in the example of FIG. 1, the shelf monitoring
camera 11, the terminal device 15, the in-store monitoring camera
16, the POS system 17, and the commodity shelf 910 are disposed in
the selling area. The hub 12, the commodity monitoring device 13,
and the wireless router device 14 are disposed in a backroom. The
backroom is basically a room in which customers are not allowed to
enter.
[0036] The commodity monitoring system 10 monitors a commodity
displayed on the commodity shelf 910. In particular, the commodity
monitoring system 10 detects that a commodity in a quantity
satisfying a predetermined condition has been collectively removed
from the commodity shelf 910.
[0037] In the case of a large number of commodities being removed
from the commodity shelf at one time, the loss if stolen will be
greater than if one commodity is stolen. In addition, cases have
been observed in which when many commodities have been stolen at
one store in a single instance, a similar theft will occur at the
same store or another store of the same chain.
[0038] Therefore, when the commodity monitoring system 10 detects
that a commodity has been removed, a clerk confirms the situation
in the vicinity of the commodity shelf 910, calls out to a
customer, work near the customer, and informs a security guard as
needed. Thereby it is possible to prevent a theft from occurring.
Further, when it is later found out that commodities whose removal
was detected by the commodity monitoring system 10 have been
stolen, the theft is brought to the attention of the employees or
chain store. This may prevent a recurrence of theft.
[0039] The shelf monitoring camera 11 images the commodity shelf
910 and transmits an image of the commodity shelf 910 to the
commodity monitoring device 13.
[0040] The in-store monitoring camera 16 is provided, for example,
on the ceiling of the store and images the inside of the store. In
particular, the in-store monitoring camera 16 images an aisle in
the store. The in-store monitoring camera 16 transmits the in-store
image that has been captured to the commodity monitoring device 13.
The image captured by the in-store monitoring camera 16 is used by
the commodity monitoring device 13 to detect the customer traffic
line.
[0041] The POS system 17 is used for accepting payments when a
customer purchases a commodity or service, and also stores sales
information. The POS system 17 transmits sales information to the
commodity monitoring device 13 in response to a request from the
commodity monitoring device 13. The POS system 17 corresponds to an
example of a sales information management device.
[0042] The commodity monitoring device 13 monitors commodities on
the commodity shelf 910 on the basis of the image captured by the
shelf monitoring camera 11. In particular, the commodity monitoring
device 13 detects that commodities in quantities satisfying a
predetermined condition have been collectively removed from the
commodity shelf 910.
[0043] The commodity monitoring device 13 functions as a server
device for the terminal device 15. In particular, the commodity
monitoring device 13 transmits change detection information to the
terminal device 15 when it is detected that a commodities in
quantities satisfying the predetermined condition have been
collectively removed from the commodity shelf 910. The change
detection information mentioned here is information for notifying
that commodities in quantities satisfying a predetermined condition
have been collectively removed from the commodity shelf 910. The
change detection information includes image data of an image of the
commodity shelf 910 at a point in time heading backward into the
past from the point in time at which the commodities in quantities
satisfying the predetermined condition were collectively removed
from the commodity shelf 910.
[0044] The commodity monitoring device 13 is configured using, for
example, a computer such as a notebook personal computer (PC) or a
workstation.
[0045] FIG. 2 is a schematic block diagram showing an example of
the function configuration of the commodity monitoring device 13.
In the example of FIG. 2, the commodity monitoring device 13 is
provided with a first communication unit 110, a first operation
input unit 120, a first display unit 130, a first storage unit 180,
and a first control unit 190. The first control unit 190 is
provided with a commodity state detection unit 191, a change
detection information output unit 192, and a moving image
processing unit 193.
[0046] The first communication unit 110 communicates with other
devices. In particular, the first communication unit 110 receives
image data captured by the shelf monitoring camera 11 and the
in-store monitoring camera 16. In addition, the first communication
unit 110 transmits a request for sales information to the POS
system 17 and receives the sales information transmitted by the POS
system 17. The first communication unit 110 also transmits change
detection information to the terminal device 15.
[0047] The first operation input unit 120 is provided with input
devices such as a keyboard and a mouse, for example, and receives
user operations.
[0048] The first display unit 130 is provided with, for example, a
front screen such as a liquid crystal panel, and displays various
images according to the control of the first control unit 190.
[0049] The first storage unit 180 stores various forms of data. The
first storage unit 180 is configured using a storage device
provided in the commodity monitoring device 13.
[0050] The first control unit 190 controls each portion of the
commodity monitoring device 13 to perform various processes. The
first control unit 190 is constituted by a CPU (central processing
unit) included in the commodity monitoring device 13 reading and
executing a program from the first storage unit 180.
[0051] The commodity state detection unit 191 detects a removal of
a commodity in a quantity satisfying a predetermined condition from
the commodity group area on the basis of image information of the
commodity group area set on the commodity shelf 910. Specifically,
the commodity state detection unit 191 compares an image of the
commodity shelf 910 captured by the shelf monitoring camera 11 with
a past image for each commodity group area. When, in the image area
of a commodity group area, a proportion of a partial area in which
a difference with the past image has been detected is equal to or
greater than a predetermined threshold value, the commodity state
detection unit 191 determines that commodities in quantities
satisfying the predetermined condition have been removed from the
commodity group area.
[0052] In order to reduce the possibility of false detection due to
a temporary image change when a customer passes in front of the
shelf, a past image that extends back a somewhat long time, such as
an image of 3 seconds ago, may be used as the past image used for
comparison.
[0053] A description will be given using as an example the case
where a quantity satisfying a predetermined condition used in the
determination criterion of the commodity state detection unit 191
indicates a proportion of the area. The predetermined condition is
one in which, for example, the image area of removed commodities in
the image captured by the shelf monitoring camera 11 accounts for a
predetermined proportion or greater of the image area of the
commodity group area including the image of the commodities.
[0054] However, the quantity satisfying the predetermined condition
used as the determination criterion by the commodity state
detection unit 191 is not limited to this example. For example, the
quantity satisfying the predetermined condition may indicate
commodities of a predetermined number or more. In this case, for
example, the commodity state detection unit 191 may calculate the
number of commodities that have been removed on the basis of the
image information in the commodity group area, and determine
whether the number of commodities that were removed is equal to or
greater than the predetermined number.
[0055] The predetermined threshold value here is set to, for
example, 50%. However, the predetermined threshold value is not
limited to 50%. Also, different threshold values may be set for
each commodity group area. For example, the user may set a
relatively small threshold value for an area of expensive
commodities. As a result, for expensive commodities, the commodity
state detection unit 191 detects the case of a relatively small
number of commodities being removed.
[0056] FIG. 3 is a drawing showing an example of setting commodity
group areas. The solid line rectangles shown in FIG. 3 indicate
commodity group areas. As shown in FIG. 3, in the image of the
commodity shelf 910, one or more commodity group areas are preset
according to the arrangement of commodities. As described above,
the commodity state detection unit 191 determines, for each
commodity group area shown in FIG. 3, whether or not a commodity in
a quantity satisfying a predetermined condition has been
removed.
[0057] The setting criteria of a commodity group area is not
limited to a specific criteria. For example, a commodity group area
may be set for each arrangement area of the same commodity, or a
commodity group area may be set for each arrangement area of
commodities of the same maker. Alternatively, the commodity group
area may be set for each arrangement area of commodities of the
same type such as magazines. Alternatively, the commodity group
area may be set for each row of the commodity shelf 910. Further,
the commodity group area may be set by a combination of a plurality
of criteria.
[0058] From the viewpoint of enhancing the detection accuracy of
the commodity state detection unit 191, it is preferable that the
sizes of the commodities arranged in the same commodity group area
be approximately the same size.
[0059] In the following, removal of a commodity in a quantity
satisfying a predetermined condition from the commodity group area
is also referred to simply as removal.
[0060] When the commodity state detection unit 191 has detected
removal, the change detection information output unit 192 outputs
to the terminal device 15 change detection information that
includes image time-series data of the commodity group area from
the time when the commodity state detection unit 191 detects
removal until a predetermined point in time in the past.
[0061] Specifically, the first storage unit 180 stores images of an
immediate predetermined time duration among the images of the
commodity shelf 910 captured by the shelf monitoring camera 11.
When the commodity state detection unit 191 has detected removal,
for example, image data of each predetermined period such as every
one second, among image data of a predetermined period of time such
as 10 seconds preceding for example the time of removal detection,
are read out from the first storage unit 180. Then, the change
detection information output unit 192 generates change detection
information including the image data that has been read out. The
change detection information output unit 192 transmits the
generated change detection information to the terminal device 15
via the first communication unit 110.
[0062] Further, the change detection information output unit 192
acquires, from the POS system 17 via the first communication unit
110, sales information indicating the sales history of the
commodity after the point in time at which the commodity state
detection unit 191 has detected the removal. Then, the change
detection information output unit 192 outputs the change detection
information including the acquired sales information to the
terminal device 15 via the first communication unit 110. By
referring to the sales information, the store clerk who refers to
the terminal device 15 can determine whether or not the removal is
a legitimate removal for the purpose of purchase. That is, the
clerk determines, on the basis of the sales information, whether or
not purchase of the commodities has been performed after the
removal of the commodities has been detected.
[0063] The change detection information output unit 192 may output
the acquired sales information to the terminal device 15 after
outputting the change detection information to the terminal device
15. Specifically, the change detection information output unit 192
acquires, from the POS system 17, sales information showing the
sales history of the removed commodities after the point in time at
which the commodity state detection unit 191 has detected the
removal, and outputs the acquired sales information to the terminal
device 15. In this way, the change detection information output
unit 192 outputs change detection information to the terminal
device 15 without waiting for the acquisition of the sales
information. Thereby, the store clerk possessing the terminal
device 15 (the user of the terminal device 15) can quickly
recognize and respond to removal of commodities from the commodity
shelf 910.
[0064] The change detection information output unit 192 may also
acquire the sales information registered in the POS system 17
within a predetermined time from when the commodity state detection
unit 191 has detected removal. The predetermined time in this case
is also referred to as a first time.
[0065] As described above, by limiting the time range in which the
change detection information output unit 192 acquires the sales
information, it is possible to limit the sales information to which
the clerk possessing the terminal device 15 refers. In this
respect, the burden on the store clerk can be reduced.
[0066] When it is not possible to acquire sales information of the
removed commodities from the POS system 17 within a predetermined
time from the point in time when the commodity state detection unit
191 has detected removal, the change detection information output
unit 192 may output change detection information that does not
include sales information to the terminal device 15. The
predetermined time in this case is also referred to as a second
time.
[0067] If the sales information is not included in the change
detection information, the store clerk who possesses the terminal
device 15 can assume there is a high possibility that the
commodities have been removed of the store without being processed
by the POS system 17. That is, the clerk can surmise that there is
a high possibility that the commodities have been stolen.
[0068] The first time and the second time may be the same time or
may be different times. For example, the second time may be set to
a time longer than the first time. When the first time becomes
long, it is conceivable that the number of items of sales
information referred to by the clerk possessing the terminal device
15 increases and so the burden on the clerk also increases. On the
other hand, even if the second time lengthens, the burden on the
store clerk does not increase. Therefore, the second time may be
set to a relatively long time. As a result, the determination as to
whether or not commodities have been removed from the store without
being processed by the POS system 17 can be made more
accurately.
[0069] The change detection information output unit 192 may acquire
information relating to the person who has performed the removal on
the basis of a past image of the commodity group area referenced to
the point in time when the commodity state detection unit 191
detected the removal. Then, the change detection information output
unit 192 may output change detection information including
information relating to the person who has performed the removal to
the terminal device 15.
[0070] It is highly likely that a person appearing in a past image
close to the point in time when the commodity state detection unit
191 detects removal is the person who has performed the removal.
Therefore, the change detection information output unit 192 may
detect a person appearing in the image of the shelf monitoring
camera 11 heading backward into the past from the point in time
when the commodity state detection unit 191 has detected removal.
For example, the change detection information output unit 192
performs person detection until succeeding in person detection,
heading backward into the past in sequence from an image at the
time the commodity state detection unit 191 detected removal.
Alternatively, for images within a predetermined time in the past
(for example, 3 seconds) from the point in time at which removal
was detected by the commodity state detection unit 191, the change
detection information output unit 192 may detect all persons
appearing in the images.
[0071] When the change detection information output unit 192 has
succeeded in detecting a person, the change detection information
output unit 192 detects information relating to that person as
information relating to the person who has performed the removal
(the person who may have performed the removal).
[0072] The change detection information output unit 192 determines,
for example, the color of clothing of the person, includes
information on the determined color of clothing in the change
detection information, and outputs the change detection information
to the terminal device 15. Information relating to the person who
performed removal is not limited to information on the color of the
person's clothing, and can be various types of information. For
example, as information on the person who performed removal, the
change detection information output unit 192 may include separate
information relating to this person in the change detection
information in addition to information on the color of the person's
clothing, or instead of information on the color of the clothing of
the person. Other information related to the person may be, for
example, a facial image of the person, flow line information
indicating the movement of the person, information indicating
whether the person processed commodities by the POS system 17,
attribute information of the person or a combination of the above.
Attribute information of a person is either one or both of gender
and height.
[0073] In this way, when the clerk in possession of the terminal
device 15 is specifying the person who has performed the removal,
the clerk refers to information on the person who has performed the
removal. This can increase the accuracy of person
identification.
[0074] Alternatively, the change detection information output unit
192, after outputting the change detection information to the
terminal device 15, may output the information regarding the person
who has performed removal to the terminal device 15.
[0075] As described above, the change detection information output
unit 192 outputs the change detection information to the terminal
device 15 without waiting for acquisition of information on the
person who has performed the removal. Thereby, the clerk in
possession of the terminal device 15 can quickly recognize and
respond to removal of commodities from the commodity shelf 910.
[0076] The change detection information output unit 192, based on
information on a person who has performed removal, may determine
whether the person has passed the commodities through a
predetermined checkout area. Specifically, the change detection
information output unit 192 may determine whether this person has
settled payment at the POS system 17 or not. Upon determining that
the person has not passed commodities through the checkout area,
the change detection information output unit 192, after outputting
the change detection information to the terminal device 15, may
output information on the person who has performed the removal to
the terminal device 15.
[0077] As described above, the change detection information output
unit 192 outputs information relating to the person who has
performed the removal to the terminal device 15 only when the
person who has performed the removal is determined to have not
passed the commodities through the predetermined checkout area. As
a result, the change detection information output unit 192 can
reduce the number of times the store clerk possessing the terminal
device 15 refers to information relating to the person. If the
person who performed the removal does not pass the commodities
through the predetermined checkout area, there is a high
possibility that the commodities were stolen. Accordingly, the
clerk need only refer to the information relating to the person who
removed the commodities when the commodities are likely to have
been stolen. Therefore, the burden on the store clerk can be
reduced.
[0078] When the commodity state detection unit 191 detects removal,
the moving image processing unit 193 displays in the first display
unit 130 a moving image of the commodity shelf 910 of a past
predetermined time from a point in time at which the commodity
state detection unit 191 detected removal. In this case, the
commodity monitoring device 13 provided with the first display unit
130 corresponds to an example of a display device.
[0079] For example, the shelf monitoring camera 11 captures a
moving image of the commodity shelf 910 and transmits moving image
data to the commodity monitoring device 13. In the commodity
monitoring device 13, the moving image processing unit 193 causes
the first storage unit 180 to temporarily store an immediate
predetermined time duration from the past moving image data
captured by the shelf monitoring camera 11. When the commodity
state detection unit 191 detects removal, the moving image
processing unit 193 saves in the first storage unit 180 moving
image data of a past predetermined time duration from the point in
time at which the commodity state detection unit 191 has detected
removal, from the moving image data temporarily stored in the first
storage unit 180. The first storage unit 180 stores, without
deleting, the moving image data that the moving image processing
unit 193 has instructed to be saved. The moving image processing
unit 193 plays back moving image data according to the user
operations and causes the first display unit 130 to display the
data.
[0080] The commodity monitoring device 13 displays a moving image
of the commodity shelf 910, and the terminal device 15 displays a
frame-by-frame advancing image (a still image for every
predetermined time) of the commodity shelf 910. Thus, images can be
used appropriately depending on the communication band and the
processing capability of the device.
[0081] However, the terminal device 15 may display a moving image
of the commodity shelf 910.
[0082] The terminal device 15, upon receiving the change detection
information from the commodity monitoring device 13, outputs a
removal alarm (alarm). Here, the removal alarm is an alarm that
indicates that a removal has occurred. The terminal device 15
corresponds to an example of an output destination device.
[0083] The terminal device 15 also generates and updates loss
information according to user operation. The loss information
referred to here is information indicating whether or not the
removal is an unauthorized removal, and the status of the
removal.
[0084] The terminal device 15 is configured to use, for example, a
portable terminal device such as a tablet terminal device or a
smartphone. Alternatively, the terminal device 15 may be configured
as a fixed terminal device. For example, the terminal device 15 is
installed in a fixed manner near the POS system 17.
[0085] For example, a clerk in the sales department holds the
terminal device 15. When the terminal device 15 outputs a removal
alarm, the clerk confirms the condition of the commodity shelf 910.
As described above, the clerk confirms the situation in the
vicinity of the commodity shelf 910, calls out to the customer,
work near the customer, and informs a security guard as needed.
Thereby it is possible to prevent a theft from occurring.
[0086] The clerk inputs loss information to the terminal device 15
on the basis of the confirmation result of the state of the
commodity shelf 910.
[0087] FIG. 4 is a schematic block diagram showing an example of a
functional configuration of the terminal device 15. In the example
of FIG. 4, the terminal device 15 is provided with a second
communication unit 210, a second operation input unit 220, a second
display unit 230, a second storage unit 280, and a second control
unit 290. The second control unit 290 is provided with a change
detection information acquisition unit 291 and a loss information
processing unit 292.
[0088] The second communication unit 210 communicates with other
devices. In particular, the second communication unit 210 receives
change detection information from the commodity monitoring device
13. Further, the second communication unit 210 transmits the loss
information to the commodity monitoring device 13.
[0089] The second display unit 230 is provided with, for example, a
display screen such as a liquid crystal panel and displays various
images according to the control of the second control unit 290. In
particular, the second display unit 230 displays any image of the
image time-series data included in the change detection information
on the basis of the user operation according to the control of the
second control unit 290. Specifically, the change detection
information includes an image for each predetermined time among the
images of the commodity shelf 910 captured by the shelf monitoring
camera 11. The second display unit 230 displays the image
designated by the user operation among the images of the commodity
shelf 910 included in the change detection information according to
the control of the second control unit 290.
[0090] The second operation input unit 220 is provided with, for
example, a touch sensor provided on the display screen of the
second display unit 230 to configure a touch panel, and receives
user operations.
[0091] The second storage unit 280 stores various forms of data.
The second storage unit 280 is configured to use a storage device
provided in the terminal device 15.
[0092] The second control unit 290 controls each portion of the
terminal device 15 to perform various processes. The second control
unit 290 is constituted by a CPU included in the terminal device 15
reading and executing a program from the second storage unit
280.
[0093] The change detection information acquisition unit 291
acquires change detection information from the commodity monitoring
device 13. Specifically, when the commodity monitoring device 13
detects removal and transmits change detection information, the
second communication unit 210 receives communication data from the
commodity monitoring device 13. The change detection information
acquisition unit 291 extracts change detection information from the
communication data received by the second communication unit
210.
[0094] The change detection information acquisition unit 291 causes
the second display unit 230 to display the removal alarm on the
basis of the change detection information. The terminal device 15
may output the removal alarm with either one or both of audio and
vibration in addition to or instead of displaying the removal alarm
on the second display unit 230.
[0095] The loss information processing unit 292 generates and
updates the loss information in accordance with a user operation.
Specifically, the loss information processing unit 292 causes the
second display unit 230 to display a details registration screen
for editing the loss information, on the occasion (trigger) of
receiving the change detection information or a user operation.
Then, when the store clerk, who is the user of the terminal device
15, performs a user operation of inputting the loss information
according to the details registration screen, the loss information
processing unit 292 generates or updates the loss information
according to the user operation.
[0096] Returning to FIG. 1, the hub 12 mediates communication
between devices connected to the hub 12. In particular, the hub 12
establishes a communication path between the shelf monitoring
camera 11 and the commodity monitoring device 13, between the
in-store monitoring camera 16 and the commodity monitoring device
13, between the POS system 17 and the commodity monitoring device
13, and between the wireless router device 14 and the commodity
monitoring device 13.
[0097] The wireless router device 14 wirelessly communicates with
the terminal device 15 to mediate communication between the
terminal device 15 and each device. In particular, the combination
of the hub 12 and the wireless router device 14 establishes a
communication path between the commodity monitoring device 13 and
the terminal device 15.
[0098] However, the configuration of the commodity monitoring
system 10 is not limited to the configuration shown in FIG. 1.
[0099] For example, the terminal device 15 may also be provided
with the function of the commodity monitoring device 13. In this
case, the commodity monitoring system 10 need not be provided with
the commodity monitoring device 13.
[0100] Further, the in-store monitoring camera 16 and the POS
system 17 are not essential to the commodity monitoring system 10.
The commodity monitoring system 10 need not be provided with one or
both of the in-store monitoring camera 16 and the POS system
17.
[0101] Further, the commodity monitoring device 13 may directly
communicate with another device. In this case, the commodity
monitoring system 10 need not be provided with one or both of the
hub 12 and the wireless router device 14.
[0102] Next, screen display by the second display unit 230 (FIG. 4)
of the terminal device 15 will be described with reference to FIGS.
5 to 11.
[0103] FIG. 5 is a drawing showing an example of a notification
screen displayed by the second display unit 230. The display of the
notification screen by the second display unit 230 corresponds to
an example of the output of the removal alarm.
[0104] In the example of FIG. 5, the second display unit 230
displays the commodity group area in which removal is detected, and
also displays a message indicating that a removal has been
detected. Furthermore, the second display unit 230 displays a
button icon in each of areas A111 and A112.
[0105] When a touch operation on the area A111 is performed, the
second display unit 230 ends the display of the removal alarm
according to the control of the second control unit 290, and
displays the standby screen (desktop screen).
[0106] When a touch operation on the area A112 is performed, the
second display unit 230 displays the detection information screen
according to the control of the second control unit 290. The
detection information screen will be described with reference to
FIG. 6.
[0107] FIG. 6 is a drawing showing an example of a detection
information screen displayed by the second display unit 230.
[0108] In the example of FIG. 6, the second display unit 230
displays an image of the commodity shelf 910 captured by the shelf
monitoring camera 11 in area A211. An area A12 in the image of the
commodity shelf 910 is an area in which the commodity state
detection unit 191 has detected a change in the image. The second
display unit 230 shows an area in the image of the commodity shelf
910 in which a change in the image has been detected.
[0109] The image displayed in the area A211 by the second display
unit 230 is one of a plurality of time-series images from a point
in time at which the commodity state detection unit 191 has
detected the removal to a predetermined past point in time. The
second display unit 230 switches the image according to the user
operation. For example, when a flick operation to the right
direction is performed in the area A211, the second display unit
230 displays an image at a later point in time according to the
control of the second control unit 290. When a flick operation to
the left direction is performed in the area A211, the second
display unit 230 displays an image at an earlier time point
according to the control of the second control unit 290.
[0110] The second display unit 230 shows in an area A221 capture
time information of the image displayed in the area A211. When a
touch operation to an area A222 is performed, the second display
unit 230, in accordance with the second control unit 290, switches
the image displayed in the area A211 to the image at the time when
the commodity state detection unit 191 detected removal.
[0111] The second display unit 230 displays an icon of a push
button in each of the display areas A231 and A232.
[0112] When a touch operation on the area A231 is performed, the
second display unit 230 displays a details registration screen in
accordance with the control of the second control unit 290. The
details registration screen will be described later with reference
to FIG. 7.
[0113] On the other hand, when a touch operation on the area A232
is performed, the second display unit 230 displays a standby
screen. This means that the store clerk who is the user of the
terminal device 15 has determined that the removal is a valid
removal. Therefore, the terminal device 15 ends the screen display
regarding the removal without requesting the user to input detailed
information on the removal.
[0114] The second display unit 230 displays the time at which the
commodity state detection unit 191 detected the removal in an area
A241, and displays the name of the commodity group area in which
the commodity state detection unit 191 detects the removal in an
area A242. The name of the commodity group area is, for example,
set in advance by the user.
[0115] FIG. 7 is a drawing showing a first example of the details
registration screen displayed by the second display unit 230.
[0116] In the example of FIG. 7, areas A311, A321, and A322 are the
same as the areas A211, A221, and A222 of FIG. 6, respectively.
Areas A331 and A332 are the same as the areas A241 and A242 of FIG.
6, respectively.
[0117] In the details registration screen of FIG. 7, the second
display unit 230 displays status information in the area A333. This
status information is information indicating a determination result
in which a store clerk who is the user of the terminal device 15
determines the presence or absence of a problem with regard to a
removal. The value of the status information is set to "Problem" by
a touch operation on the area A231 in FIG. 6. Alternatively, the
value of the status information is set to "No problem" by a touch
operation on the area A232 in FIG. 6. In the area A333, the user
can change the value of the status information. Specifically, when
a touch operation is performed on the button with a black
down-pointing triangle in the area A333, the second display unit
230 displays a drop-down menu indicating "Problem" and "No
problem". When a touch operation to either "Problem" or "No
problem" is performed in the drop-down menu, the loss information
processing unit 292 updates the value of the status information to
the value selected by the touch operation.
[0118] FIG. 8 is a drawing showing a second example of the details
registration screen displayed by the second display unit 230.
[0119] FIG. 8 shows an example of the details registration screen
continued from the lower side of the details registration screen
shown in FIG. 7. The second display unit 230 displays the screen of
FIG. 8 according to the control of the loss information processing
unit 292 by scrolling down the details registration screen of FIG.
7.
[0120] An area A334 is an input field in which information relating
to lost commodities can be input. The second operation input unit
220 receives, by free word input, inputs of the commodity name and
quantity of a stolen commodity (the commodity removed from the
commodity shelf 910).
[0121] An area A335 is an input column for the total loss amount.
The second operation input unit 220 receives, by free word input,
an input of the total loss amount due to the removal in question
(for example, the total amount of theft loss).
[0122] An area A336 is a comment input field. The second operation
input unit 220 receives user comments by free word input.
[0123] An area A337 is a display field of the registrant. The
second operation input unit 220 may receive an input operation of a
registrant's name. Alternatively, the second operation input unit
220 may cause the loss information processing unit 292 to
automatically display in the area A337 the name of the user who has
logged in to the terminal device 15 during input of the loss
information.
[0124] When a touch operation on an area A341 is performed, the
loss information processing unit 292 transmits the loss information
that has been input to the commodity monitoring device 13 via the
second communication unit 210. In the commodity monitoring device
13, the first storage unit 180 stores the loss information in
association with the change detection information. Further, the
second display unit 230 ends the display of the details
registration screen by a touch operation on the area A341, and
displays, for example, a loss information list screen. The loss
information list screen will be described referring to FIG. 9.
[0125] FIG. 9 is a drawing showing an example of the loss
information list screen displayed by the second display unit
230.
[0126] In the example of FIG. 9, the second display unit 230
displays a list of loss information items in an area A411.
[0127] An area A421 is a touch area for receiving an update of the
loss information. When a touch operation is performed on the area
A421 in a state where any one of the loss information items is
selected by a touch operation to the list in the area A411, the
loss information processing unit 292 acquires the selected loss
information from the commodity monitoring device 13. The loss
information processing unit 292 causes the second display unit 230
to display a details registration screen having the acquired loss
information. The loss information processing unit 292 updates the
loss information in accordance with the information updated
according to the user operation.
[0128] An area A422 is a touch area that receives deletion of loss
information. When a touch operation is performed on the area A422
in a state where any one of the loss information items is selected
by a touch operation to the list in the area A411, the loss
information processing unit 292 transmits to the commodity
monitoring device 13 via the first communication unit 110 an
instruction to delete the selected loss information item. The
commodity monitoring device 13 deletes the loss information in
accordance with the deletion instruction.
[0129] In order to prevent an unauthorized operation with respect
to the loss information, only a store clerk who is authorized among
the plurality of store clerks may update or delete loss
information.
[0130] An area A431 is a touch area that receives an instruction to
end a display regarding removal. When a touch operation on the area
A431 is performed, the second display unit 230, according to the
control of the second control unit 290, ends the display related to
removal and displays the standby screen.
[0131] FIG. 10 is a drawing showing an example of the sales history
screen displayed by the second display unit 230.
[0132] If there is sales information associated with a removal
detected by the commodity state detection unit 191, the commodity
monitoring device 13 acquires the sales information from the POS
system 17, associates the change detection information with the
sales information, and stores the sales information in the first
storage unit 180. Here, various correspondence relationships can be
adopted as the correspondence relationship between the removal and
the sales information. For example, the commodity monitoring device
13 may determine sales information within a predetermined time
after a removal occurs to be the sales information corresponding to
the removal. Alternatively, the commodity monitoring device 13 may
determine the sales information indicating a purchase of the
commodity removed from the commodity shelf 910 among the sales
information within a predetermined time after a removal occurs, to
be the sales information corresponding to the removal.
[0133] In the event of a user operation that instructs display of
the sales history screen (FIG. 10) being performed while the second
display unit 230 is displaying the detection information screen
(FIG. 6), the loss information processing unit 292 causes the
second display unit 230 to display the sales information associated
with the detection information screen. Specifically, the loss
information processing unit 292 acquires, from the commodity
monitoring device 13, sales information associated with the removal
that is the subject of the detection information screen being
displayed, and causes the second display unit 230 to display the
sales information.
[0134] In the event of a user operation that instructs display of
the sales history screen (FIG. 10) being performed while the second
display unit 230 is displaying the details registration screen
(FIG. 7, FIG. 8), the loss information processing unit 292 causes
the second display unit 230 to display the sales information
associated with details registration screen. Specifically, the loss
information processing unit 292 acquires, from the commodity
monitoring device 13, sales information associated with the removal
that is the subject of the details registration screen being
displayed, and causes the second display unit 230 to display the
sales information.
[0135] When a user operation instructing the end of the display of
the sales history screen is performed, the second display unit 230
ends the display of the sales history screen, and returns to
display of the detection information screen (FIG. 6) or display of
the details registration screen (FIG. 7, FIG. 8).
[0136] FIG. 11 is a drawing showing an example of a flow line
screen displayed by the second display unit 230.
[0137] In the example of FIG. 11, the second display unit 230
displays, on a floor plan of the store, the flow line information
of a customer who performed removal.
[0138] In the example of FIG. 11, the customer has stopped at the
counter. The counter corresponds to an example of a predetermined
checkout area. The second display unit 230 displays, in addition to
the flow line information of the customer, the time at which the
customer stopped at the counter.
[0139] The store clerk who is the user of the terminal device 15
causes the terminal device 15 to display sales information at the
timing when the customer stops by the counter. Thereby it is
possible to determine whether or not the removal is a valid
removal. If the sales information at the timing when the customer
stopped by the counter indicates that the commodity subject to
removal has been purchased, it is possible to determine that the
removal is a valid removal.
[0140] Alternatively, in place of or in addition to the flow line
information, the second display unit 230 may display, with text or
symbols, information indicating whether or not the customer who has
performed removal settled payment at the counter. Text indicating
whether or not payment has been made is, for example, "paid"/"not
paid". Symbols indicating whether or not the payment is made are,
for example, "O"/"x".
[0141] Next, the operation of the commodity monitoring system 10
will be described with reference to FIGS. 12 and 13.
[0142] FIG. 12 is a diagram showing an example of operation of the
commodity monitoring system 10 in the case of detecting
removal.
[0143] The shelf monitoring camera 11 transmits image data
generated by capturing an image of the commodity shelf 910 to the
commodity monitoring device 13 (sequence S101). The shelf
monitoring camera 11 may continuously transmit moving image data of
the commodity shelf 910 to the commodity monitoring device 13.
Alternatively, the shelf monitoring camera 11 may collectively
transmit moving image data of the commodity shelf 910 to the
commodity monitoring device 13 in blocks of fixed time durations.
Alternatively, the shelf monitoring camera 11 may transmit still
image data corresponding to a frame of moving image data to the
commodity monitoring device 13 instead of transmitting moving image
data.
[0144] The commodity monitoring device 13 upon acquiring the image
data of the commodity shelf 910 determines whether or not a removal
has occurred on the basis of the obtained image data (sequence
S102). Specifically, the commodity monitoring device 13 compares
the newly acquired image of the commodity shelf 910 with a past
image of the commodity shelf 910 and detects the difference between
the images. The commodity monitoring device 13 determines, for any
commodity group area, whether the proportion of a partial area
corresponding to the detected difference to the area of the entire
commodity group area is equal to or greater than a threshold value.
If it is determined that the proportion is equal to or greater than
the threshold value, the commodity monitoring device 13 determines
that a removal has occurred. In the example of FIG. 12, the
commodity monitoring device 13 determines that there has been a
removal.
[0145] The commodity monitoring device 13 that has determined that
a removal has occurred generates change detection information
related to the detected removal and transmits the change detection
information to the terminal device 15 (sequence S103).
[0146] The terminal device 15 outputs a removal alarm on the basis
of the change detection information received from the commodity
monitoring device 13 (sequence S104).
[0147] Further, the commodity monitoring device 13 saves (stores)
the generated change detection information (sequence S105).
[0148] The commodity monitoring device 13 may generate change
detection information including one or both of sales information
and flow line information. Alternatively, after the commodity
monitoring device 13 transmits the change detection information to
the terminal device 15, the commodity monitoring device 13 may
transmit one or both of the sales information and the flow line
information to the terminal device 15.
[0149] FIG. 13 is a flowchart showing an example of the processing
procedure in which the commodity monitoring device 13 determines
the presence or absence of removal.
[0150] In the process of FIG. 13, the commodity state detection
unit 191 acquires image data of the commodity shelf 910 from the
shelf monitoring camera 11 via the first communication unit 110
(Step S201). Then, the commodity state detection unit 191 compares
the image data of the commodity shelf 910 obtained in Step S201
with past image data (for example, an image of 10 seconds before)
of the commodity shelf 910 (Step S202). Then, for each of the
commodity group areas, the commodity state detection unit 191
determines whether the proportion of the area of the difference
portion (difference) of the image of the commodity group area to
the area of the entire commodity group area is equal to or greater
than the threshold value (Step S203).
[0151] If it is determined that the proportion is less than the
threshold value for any of the commodity group areas (Step S203:
NO), the process returns to Step S201. On the other hand, if it is
determined that the proportion is greater than or equal to the
threshold value for any of the commodity group areas (Step S203:
YES), the commodity monitoring device 13 performs processing upon
detection of removal (Step S204). For example, as described with
reference to FIG. 12, the commodity monitoring device 13 generates
change detection information and transmits the change detection
information to the terminal device 15, or stores the generated
change detection information.
[0152] After Step S204, the process returns to Step S201.
[0153] As described above, based on the image information of the
commodity group area set on the commodity shelf 910, the commodity
state detection unit 191 detects the removal of a commodity in a
quantity satisfying the predetermined condition from the commodity
group area.
[0154] When the commodity state detection unit 191 has detected
removal, the change detection information output unit 192 outputs
to the terminal device 15 change detection information including
image time-series data of the commodity group area from a point in
time at which the removal was detected to a predetermined past
point in time.
[0155] A store clerk who is a user of the terminal device 15 can
recognize the possibility of unauthorized removal by recognizing a
removal based on the change detection information. For example, the
clerk can recognize the removal by the terminal device 15
outputting a removal alarm on the basis of the change detection
information.
[0156] In particular, the commodity monitoring system 10 can detect
the possibility of unauthorized removal with a relatively simple
configuration in which the shelf monitoring camera 11 captures
images of the commodity shelf 910. In this regard, according to the
commodity monitoring system 10, an event in which a commodity is
expected to be brought out of a store improperly can be detected at
relatively low cost.
[0157] The change detection information output unit 192 may acquire
sales information after the point in time at which the commodity
state detection unit 191 detects removal, and output the change
detection information including the sales information to the
terminal device 15.
[0158] Thereby, the store clerk who is the user of the terminal
device 15 can determine whether or not the removal is a valid
removal with reference to the change detection information.
[0159] After the change detection information output unit 192
outputs the change detection information to the terminal device 15,
the change detection information output unit 192 may acquire sales
information after the point in time at which the commodity state
detection unit 191 detected the removal and output the sales
information to the terminal device 15.
[0160] The user of the terminal device 15 can determine whether or
not the removal is a valid removal by referring to the sales
information. The change detection information output unit 192
outputs the change detection information to the terminal device 15
without waiting for the acquisition of the sales information. For
this reason, the user of the terminal device 15 can quickly
recognize and respond to removal of the commodity from the
commodity shelf 910.
[0161] The change detection information output unit 192 may acquire
sales information registered in the POS system 17 within a
predetermined time from the point in time at which the commodity
state detection unit 191 detected the removal. In this way, the
change detection information output unit 192 limits the time range
for acquiring the sales information. As a result, it is possible to
limit the sales information to which a store clerk possessing the
terminal device 15 refers. In this respect, the burden on the store
clerk can be reduced.
[0162] If the change detection information output unit 192 cannot
acquire the sales information related to the removed commodities
from the POS system 17 within a predetermined time from the point
in time at which the commodity state detection unit 191 detected
the removal, the change detection information output unit 192 may
output change detection information that does not include the sales
information to the terminal device 15.
[0163] If sales information is not included in the change detection
information, the store clerk who possesses the terminal device 15
can estimate that there is a high possibility that the
corresponding commodities have been removed from the store without
going through the POS system 17. That is, the clerk can surmise
that there is a high possibility that the commodities have been
stolen.
[0164] The change detection information output unit 192 may acquire
information relating to the person who has performed the removal on
the basis of a past image of the product group area referenced to a
point in time at which the commodity state detection unit 191
detected removal, and output the information to the terminal device
15.
[0165] The user of the terminal device 15, when specifying the
person who performed the removal, can improve the accuracy of the
person specification by referring to the information relating to
the person who performed the removal.
[0166] The change detection information output unit 192 detects
information relating to the person who has performed the removal
based on a past image of the commodity group area referenced to the
point in time at which the product state detection unit 191
detected the removal. At this time, the change detection
information output unit 192 may output information relating to the
person who performed the removal to the terminal device 15 after
outputting the change detection information to the terminal device
15.
[0167] The user of the terminal device 15, when specifying the
person who performed the removal, can improve the specification
accuracy by referring to the information relating to the person who
performed the removal. Further, the change detection information
output unit 192 outputs the change detection information to the
terminal device 15 without waiting for acquisition of the
information relating to the person who has performed the removal.
Thereby, the user of the terminal device 15 can quickly recognize
and respond to removal of commodities from the commodity shelf
910.
[0168] The change detection information output unit 192 may
determine, on the basis of the information relating to the person
who has performed a removal, whether the person has passed the
commodities (the removed commodities) through a predetermined
checkout area. If it is determined that the checkout area has not
been passed, the change detection information output unit 192,
after outputting the change detection information to the terminal
device 15, may output information relating to the person who has
performed the removal to the terminal device 15.
[0169] In this way, the change detection information output unit
192 determines whether the person who has performed removal has
passed the removed commodities through the checkout area. Thereby,
the user of the terminal device 15 can use the information as
reference information when determining whether a removal is a valid
removal.
[0170] The second display unit 230 of the terminal device 15 may
display any image of the image time-series data included in the
change detection information on the basis of a user operation.
[0171] The user of the terminal device 15 can ascertain the
situation when a removal is performed by referring to the image
displayed by the second display unit 230.
[0172] The first display unit 130 of the commodity monitoring
device 13 may display a moving image of the commodity group area
from the point in time at which the commodity state detection unit
191 detected a removal to a predetermined past point in time.
[0173] The user of the commodity monitoring device 13 can ascertain
the situation when a removal is performed by referring to this
moving image.
[0174] Next, the minimum configuration of the present embodiment
will be described with reference to FIGS. 14 to 16.
[0175] FIG. 14 is a drawing showing an example of the minimum
configuration of the commodity monitoring device according to the
present embodiment. A commodity monitoring device 20 shown in FIG.
14 is provided with a commodity state detection unit 21 and a
change detection information output unit 22.
[0176] With such a configuration, the commodity state detection
unit 21 detects the removal of a commodity in a quantity satisfying
a predetermined condition from the commodity group area on the
basis of image information of the commodity group area set on the
commodity shelf. When the commodity state detection unit 21 has
detected a removal, the change detection information output unit 22
outputs to an output destination device change detection
information that includes image time-series data of the commodity
group area from the point in time at which the commodity state
detection unit 21 detected the removal to a predetermined past
point in time.
[0177] The user can recognize the possibility of unauthorized
removal by recognizing removals on the basis of the change
detection information. In particular, the commodity monitoring
device 20 can detect the possibility of unauthorized removal with a
relatively simple configuration of capturing the image of the
commodity shelf by a camera. In this regard, according to the
commodity monitoring device 20, an event in which a commodity is
expected to be brought out of a store improperly can be detected at
relatively low cost.
[0178] FIG. 15 is a drawing showing an example of the minimum
configuration of a commodity monitoring system according to the
present embodiment. A commodity monitoring system 30 shown in FIG.
15 is provided with a commodity monitoring device 31 and an output
destination device 34. The commodity monitoring device 31 is
provided with a commodity state detection unit 32 and a change
detection information output unit 33.
[0179] With this configuration, the commodity state detection unit
32 detects the removal of a commodity in a quantity satisfying a
predetermined condition from the commodity group area on the basis
of image information of the commodity group area set on the
commodity shelf. When the commodity state detection unit 32 has
detected a removal, the change detection information output unit 33
outputs to the output destination device 34 change detection
information that includes image time-series data of the commodity
group area from the point in time at which the commodity state
detection unit 32 detected the removal to a predetermined past
point in time.
[0180] The user can recognize the possibility of unauthorized
removal by recognizing removals on the basis of the change
detection information. In particular, the commodity monitoring
device 30 can detect the possibility of unauthorized removal with a
relatively simple configuration of capturing the image of the
commodity shelf by a camera. In this regard, according to the
commodity monitoring device 30, an event in which a commodity is
expected to be brought out of a store improperly can be detected at
relatively low cost.
[0181] FIG. 16 is a drawing showing an example of the minimum
configuration of an output destination device according to the
present embodiment. The output destination device 40 illustrated in
FIG. 16 is provided with a change detection information acquisition
unit 41 and a display unit 42.
[0182] In this configuration, when the removal of a commodity in a
quantity satisfying a predetermined condition from the commodity
group area set on the commodity shelf has been detected, the change
detection information acquisition unit 41 acquires change detection
information that includes the image time-series data of the
commodity group area from the point in time at which the removal
was detected to a predetermined past point in time. The display
unit 42 displays images of the commodity group area included in the
change detection information.
[0183] The user can ascertain the situation in which the removal
was performed by referring to images of the commodity group
area.
[0184] A program for realizing all or a part of the functions of
the first control unit 190 and the second control unit 290 may be
recorded in a computer-readable recording medium, and the program
recorded in the recording medium read and executed by a computer
system to perform the processing of each unit. The "computer
system" referred to herein includes an OS and hardware such as
peripheral devices.
[0185] The "computer readable recording medium" is a portable
medium such as a flexible disk, a magneto-optical disk, a ROM, and
a CD-ROM, and a memory device such as a hard disk incorporated in
the computer system. The abovementioned program may be a program
for achieving a portion of the above-described functions. Moreover,
the above-described program may be a program which can realize the
above-described functions by being combined with a program which
has already been recorded in the computer system.
[0186] Hereinabove, exemplary embodiments of the present invention
have been described in detail with reference to the drawings.
However, the specific configuration is not limited to the exemplary
embodiments, and also includes design changes that do not depart
from the scope of the present invention.
[0187] All or some of the above exemplary embodiments can be
described as in the following supplementary notes, but the present
invention is not limited to the following supplementary notes.
(Supplementary Note 1)
[0188] A commodity monitoring device provided with: a commodity
state detection unit configured to, on the basis of image
information of a commodity group area set on a commodity shelf,
detect removal of a commodity in a quantity satisfying a
predetermined condition from the commodity group area; and a change
detection information output unit configured to, if the commodity
state detection unit has detected the removal, output to an output
destination device change detection information including image
time-series data of the commodity group area from a point in time
at which the commodity state detection unit detected the removal to
a predetermined past point in time.
(Supplementary Note 2)
[0189] The commodity monitoring device according to supplementary
note 1, wherein the change detection information output unit
acquires sales information indicating sales history of the
commodity at a point in time after the commodity state detection
unit detected the removal, and outputs the change detection
information including the sales information to the output
destination device.
(Supplementary Note 3)
[0190] The commodity monitoring device according to supplementary
note 1, wherein the change detection information output unit, after
outputting the change detection information to the output
destination device, acquires sales information indicating sales
history of the commodity at a point in time after the commodity
state detection unit detected the removal, and outputs the sales
information to the output destination device.
(Supplementary Note 4)
[0191] The commodity monitoring device according to supplementary
note 2 or supplementary note 3, wherein the change detection
information output unit acquires sales information registered in a
sales information management device within a predetermined time
from a point in time at which the commodity state detection unit
detected the removal.
(Supplementary Note 5)
[0192] The commodity monitoring device according to supplementary
note 4, wherein when the change detection information output unit
cannot acquire sales information relating to the commodity from the
sales information management device within a predetermined time
from a point in time at which the commodity state detection unit
detected the removal, the change detection information output unit
outputs the change detection information not including the sales
information to the output destination.
(Supplementary Note 6)
[0193] The commodity monitoring device according to any one of
supplementary notes 1 to 5, wherein the change detection
information output unit, on the basis of a past image of the
commodity group area referenced to a point in time at which the
commodity state detection unit detected the removal, acquires
information relating to a person who removed the commodity and
outputs the information relating to the person who removed the
commodity to the output destination device.
(Supplementary Note 7)
[0194] The commodity monitoring device according to any one of
supplementary notes 1 to 5, wherein the change detection
information output unit, on the basis of a past image of the
commodity group area referenced to a point in time at which the
commodity state detection unit detected the removal, detects
information relating to a person who removed the commodity and,
after outputting the change detection information to the output
destination device, outputs the information relating to the person
to the output destination device.
(Supplementary Note 8)
[0195] The commodity monitoring device according to supplementary
note 7, wherein the change detection information output unit, on
the basis of the information relating to the person who removed the
commodity, determines whether the person passed the commodity
through a predetermined checkout area and, in the case of having
determined that the commodity was not passed through the checkout
area, outputs the information relating to the person to the output
destination device.
(Supplementary Note 9)
[0196] A commodity monitoring system comprising a commodity
monitoring device and an output destination device, the commodity
monitoring device being provided with: a commodity state detection
unit configured to, on the basis of image information of a
commodity group area set on a commodity shelf, detect removal of a
commodity in a quantity satisfying a predetermined condition from
the commodity group area; and a change detection information output
unit configured to, if the commodity state detection unit has
detected the removal, output to the output destination device
change detection information including image time-series data of
the commodity group area from a point in time at which the
commodity state detection unit detected the removal to a
predetermined past point in time.
(Supplementary Note 10)
[0197] The commodity monitoring system according to supplementary
note 9, wherein the output destination device is provided with a
display unit configured to display any image of the image
time-series data included in the change detection information on
the basis of a user operation.
(Supplementary Note 11) The commodity monitoring system according
to supplementary note 9 or supplementary note 10, further provided
with a display device configured to display a moving image of the
commodity group area from the point in time at which the commodity
state detection unit detected the removal to a predetermined past
point in time.
(Supplementary Note 12)
[0198] An output destination device provided with: a change
detection information acquisition unit configured to, when removal
of a commodity in a quantity satisfying a predetermined condition
from a commodity group area set on a commodity shelf is detected,
acquire change detection information including image time-series
data of the commodity group area from a point in time at which the
removal was detected to a predetermined past point in time; and a
display unit configured to display the image of the commodity group
area included in the change detection information.
(Supplementary Note 13)
[0199] A commodity monitoring method including the step of: when,
based on image information of a commodity group area set on a
commodity shelf, removal of a commodity in a quantity satisfying a
predetermined condition from the commodity group area is detected,
outputting to an output destination device change detection
information including image time-series data of the commodity group
area from a point in time at which the removal was detected to a
predetermined past point in time.
(Supplementary Note 14)
[0200] A display method including the steps of: when removal of a
commodity in a quantity satisfying a predetermined condition from a
commodity group area set on a commodity shelf is detected,
acquiring change detection information including image time-series
data of the commodity group area from a point in time at which the
removal was detected to a predetermined past point in time; and
displaying the image of the commodity group area included in the
change detection information.
(Supplementary Note 15)
[0201] A program for causing a computer to execute a processing of:
when, based on image information of a commodity group area set on a
commodity shelf, removal of a commodity in a quantity satisfying a
predetermined condition from the commodity group area is detected,
outputting to an output destination device change detection
information including image time-series data of the commodity group
area from a point in time at which the removal was detected to a
predetermined past point in time.
(Supplementary Note 16)
[0202] A program for causing a computer to execute a processing of:
when removal of a commodity in a quantity satisfying a
predetermined condition from a commodity group area set on a
commodity shelf is detected, acquiring change detection information
including image time-series data of the commodity group area from a
point in time at which the removal was detected to a predetermined
past point in time; and displaying the image of the commodity group
area included in the change detection information.
[0203] Priority is claimed on Japanese Patent Application No.
2017-042079, filed Mar. 6, 2017, the content of which is
incorporated herein by reference.
INDUSTRIAL APPLICABILITY
[0204] According to the present invention, it is possible to detect
at a comparatively low cost an event in which a commodity is
expected to be brought out of a store improperly.
REFERENCE SIGNS LIST
[0205] 10: Commodity monitoring system [0206] 11: Shelf monitoring
camera [0207] 12: Hub [0208] 13: Commodity monitoring device [0209]
14: Wireless router device [0210] 15: Terminal device [0211] 16:
In-store monitoring camera [0212] 17: POS system [0213] 110: First
communication unit [0214] 120: First operation input unit [0215]
130: First display unit [0216] 180: First storage unit [0217] 190:
First control unit [0218] 191: Commodity state detection unit
[0219] 192: Change detection information output unit [0220] 210:
Second communication unit [0221] 220: Second operation input unit
[0222] 230: Second display unit [0223] 280: Second storage unit
[0224] 290: Second control unit [0225] 291: Change detection
information acquisition unit [0226] 292: Loss information
processing unit
* * * * *