U.S. patent application number 15/691319 was filed with the patent office on 2018-03-08 for article reading device.
The applicant listed for this patent is TOSHIBA TEC KABUSHIKI KAISHA. Invention is credited to Yuichiro Hatanaka, Hitoshi Iizaka, Hidehiko Miyakoshi, Hidehiro NAITOU, Yuta Sasaki.
Application Number | 20180068292 15/691319 |
Document ID | / |
Family ID | 59886995 |
Filed Date | 2018-03-08 |
United States Patent
Application |
20180068292 |
Kind Code |
A1 |
NAITOU; Hidehiro ; et
al. |
March 8, 2018 |
ARTICLE READING DEVICE
Abstract
An article reading device according to an embodiment includes a
display device and an image capturing device that generates an
image of an article. A processor extracts, from the image, first
feature data for recognizing the article and second feature data
for determining whether to recognize the article based on the first
feature data. The processor determines whether to recognize the
article. If it is determined to recognize the article, the
processor recognizes the article based on the extracted first
feature data, and controls the display device to display a
recognition result. If it is determined to not recognize the
article, extract a barcode from the image, the processor identifies
the article based on the extracted barcode, and control the display
device to display an identification result. The processor performs
a transaction settlement with respect to the recognition result, if
any, and the identification result, if any.
Inventors: |
NAITOU; Hidehiro; (Mishima
Shizuoka, JP) ; Iizaka; Hitoshi; (Fuji Shizuoka,
JP) ; Miyakoshi; Hidehiko; (Mishima Shizuoka, JP)
; Hatanaka; Yuichiro; (Mishima Shizuoka, JP) ;
Sasaki; Yuta; (Izunokuni Shizuoka, JP) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
TOSHIBA TEC KABUSHIKI KAISHA |
Tokyo |
|
JP |
|
|
Family ID: |
59886995 |
Appl. No.: |
15/691319 |
Filed: |
August 30, 2017 |
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G07G 1/0063 20130101;
G06K 9/18 20130101; G06K 2209/17 20130101; G06K 9/68 20130101; G07G
1/0045 20130101; G06K 7/10722 20130101; G06K 9/46 20130101; G06Q
20/208 20130101; G06K 7/1413 20130101 |
International
Class: |
G06Q 20/20 20060101
G06Q020/20; G06K 7/14 20060101 G06K007/14; G06K 7/10 20060101
G06K007/10 |
Foreign Application Data
Date |
Code |
Application Number |
Sep 2, 2016 |
JP |
2016-171795 |
Claims
1. An article reading device comprising: a display device; an image
capturing device configured to generate an image of an article; and
a processor configured to: receive the generated image of the
article, extract, from the image, first feature data for
recognizing the article and second feature data for determining
whether to recognize the article based on the first feature data,
determine, based on the extracted second feature data, whether to
recognize the article, if it is determined to recognize the
article, recognize the article based on the extracted first feature
data, and control the display device to display a recognition
result corresponding to the recognized article, if it is determined
to not recognize the article, extract a barcode from the image,
identify the article based on the extracted barcode, and control
the display device to display an identification result
corresponding to the identified article, and perform a transaction
settlement with respect to the recognition result, if any, and the
identification result, if any.
2. The device according to claim 1, wherein the processor
determines whether to recognize the article by transmitting the
extracted second feature data to a POS terminal, and receiving,
from the POS terminal, information indicating whether the article
is a candidate for object recognition.
3. The device according to claim 1, wherein the processor
recognizes the article by transmitting the extracted first feature
data to a POS terminal, and receiving, from the POS terminal, at
least one candidate article.
4. The device according to claim 3, wherein the recognition result
includes an image of each of a plurality of candidate articles.
5. The device according to claim 4, wherein the processor is
configured to receive a selection of one of the plurality of
candidate articles, and the transaction settlement is performed
with respect to the selection.
6. The device according to claim 1, wherein the processor is
further configured to determine whether an article is positioned
within an image capturing area of the image capturing device, and
the processor receives the generated image when the article is
determined to be positioned within the image capturing area.
7. The device according to claim 1, wherein the processor performs
the transaction settlement by transmitting the recognition result,
if any, and the identification result, if any, to a POS
terminal.
8. An article reading device comprising: a display device; an image
capturing device configured to generate an image of an article; and
a processor configured to: receive the generated image of the
article, determine whether the image includes a barcode, if it is
determined that the image includes the barcode, identify the
article based on the barcode, and control the display device to
display an identification result corresponding to the identified
article, and if it is determined that the image does not include
the barcode, extract feature data of the article, recognize the
article based on the extracted feature data, and control the
display device to display a recognition result corresponding to the
recognized article, and perform a transaction settlement with
respect to the recognition result, if any, and the identification
result, if any.
9. The device according to claim 8, wherein the processor
recognizes the article by transmitting the extracted first feature
data to a POS terminal, and receiving, from the POS terminal, at
least one candidate article.
10. The device according to claim 9, wherein the recognition result
includes an image of each of a plurality of candidate articles.
11. The device according to claim 10, wherein the processor is
configured to receive a selection of one of the plurality of
candidate articles, and the transaction settlement is performed
with respect to the selection.
12. The device according to claim 8, wherein the processor is
further configured to determine whether an article is positioned
within an image capturing area of the image capturing device, and
the processor receives the generated image when the article is
determined to be positioned within the image capturing area.
13. The device according to claim 8, wherein the processor performs
the transaction settlement by transmitting the recognition result,
if any, and the identification result, if any, to a POS
terminal.
14. A non-transitory computer readable medium containing a program
that causes article reading device including a display device, an
image capturing device, and a processor to perform an article
reading method comprising the steps of: generating, with the image
capturing device, an image of an article; extracting, from the
image, first feature data for recognizing the article and second
feature data for determining whether to recognize the article based
on the first feature data; determining, based on the extracted
second feature data, whether to recognize the article; if it is
determined to recognize the article, recognizing the article based
on the extracted first feature data, and controlling the display
device to display a recognition result corresponding to the
recognized article; if it is determined to not recognize the
article, extract a barcode from the image, identifying the article
based on the extracted barcode, and controlling the display device
to display an identification result corresponding to the identified
article; and performing a transaction settlement with respect to
the recognition result, if any, and the identification result, if
any.
15. The non-transitory computer readable medium according to claim
14, wherein whether to recognize the article is determined by
transmitting the extracted second feature data to a POS terminal,
and receiving, from the POS terminal, information indicating
whether the article is a candidate for object recognition.
16. The non-transitory computer readable medium according to claim
14, wherein the article is recognized by transmitting the extracted
first feature data to a POS terminal, and receiving, from the POS
terminal, at least one candidate article.
17. The non-transitory computer readable medium according to claim
16, wherein the recognition result includes an image of each of a
plurality of candidate articles.
18. The non-transitory computer readable medium according to claim
17, the method further comprising: receiving a selection of one of
the plurality of candidate articles, wherein the transaction
settlement is performed with respect to the selection.
19. The non-transitory computer readable medium according to claim
14, the method further comprising: determining whether an article
is positioned within an image capturing area of the image capturing
device, and the image is generated when the article is determined
to be positioned within the image capturing area.
20. The non-transitory computer readable medium according to claim
14, wherein the transaction settlement is performed by transmitting
the recognition result, if any, and the identification result, if
any, to a POS terminal.
Description
CROSS-REFERENCE TO RELATED APPLICATION
[0001] This application is based upon and claims the benefit of
priority from Japanese Patent Application No. 2016-171795, filed
Sep. 2, 2016, the entire contents of which are incorporated herein
by reference.
FIELD
[0002] Embodiments described herein relate generally to an article
reading device.
BACKGROUND
[0003] In the related art, when merchandise is to be sold at a
store, a salesperson registers the merchandise by reading a barcode
attached to the merchandise. In addition, if the barcode is not
attached to the merchandise, the salesperson compares the
merchandise with a plurality of selection buttons corresponding to
respective merchandise items displayed on a screen, operates a
selection button corresponding to the merchandise, and registers
the merchandise. On the other hand, there is also a technique in
which merchandise registration is performed using generic object
recognition for recognizing types or the like of merchandise by
extracting feature data of the merchandise from captured images
obtained by imaging the merchandise and by comparing the extracted
feature data with verification data prepared in advance.
Accordingly, a device which can execute both of merchandise
registration based on a barcode and merchandise registration based
on an object recognition result without performing an operation
such as switching has been developed.
[0004] As described above, in an article reading device which can
execute the barcode reading and object recognition without
performing the switching operation, it is desirable to
simultaneously display a selection button instructing merchandise
registration and an image of merchandise actually captured on a
screen, in order to increase efficiency of merchandise
registration. However, since two different pieces of information
are displayed in one screen, respective display ranges are reduced
and it is difficult to view display contents. Accordingly, there is
a problem that efficiency of a registration operation is
reduced.
DESCRIPTION OF THE DRAWINGS
[0005] FIG. 1 is an exterior perspective view illustrating an
example of a check-out system including an article reading device
according to an embodiment.
[0006] FIG. 2 is a block diagram illustrating an example of a
hardware configuration of the check-out system.
[0007] FIG. 3 illustrates an example configuration of a PLU
file.
[0008] FIG. 4 is a block diagram illustrating an example of a
functional configuration of the check-out system.
[0009] FIG. 5A illustrates an example of a pre-set screen.
[0010] FIG. 5B illustrates an example of a selection screen on
which a selection button of a merchandise candidate is displayed
together with a captured image of an article.
[0011] FIG. 5C illustrates an example of a reading result screen
displayed when contents of a barcode attached to merchandise are
read by a barcode reading process.
[0012] FIG. 5D illustrates an example of a re-recognition selection
screen displayed in a case where the merchandise candidate is not
uniquely determined.
[0013] FIG. 6 is a state transition diagram illustrating an example
of screen transitions.
[0014] FIG. 7 is a sequence diagram illustrating a process example
for performing one transaction.
[0015] FIG. 8 is a flow chart illustrating an example sequence of
operations for a series of merchandise registration processes.
[0016] FIG. 9 is a flow chart illustrating a modified example
sequence of operations for the series of merchandise registration
processes.
DETAILED DESCRIPTION
[0017] Embodiments provide an article reading device capable of
simultaneously executing barcode reading and object recognition so
that a captured image of merchandise can be displayed for easy
viewing without performing a special operation as necessary.
[0018] An article reading device according to an embodiment
includes a display device and an image capturing device that
generates an image of an article. A processor receives the
generated image of the article and extracts from the image, first
feature data for recognizing the article and second feature data
for determining whether to recognize the article based on the first
feature data. The processor determines, based on the extracted
second feature data, whether to recognize the article. If it is
determined to recognize the article, the processor recognizes the
article based on the extracted first feature data, and controls the
display device to display a recognition result corresponding to the
recognized article. If it is determined to not recognize the
article, extract a barcode from the image, the processor identifies
the article based on the extracted barcode, and control the display
device to display an identification result corresponding to the
identified article. The processor performs a transaction settlement
with respect to the recognition result, if any, and the
identification result, if any.
Descriptions of Overall Configuration of Check-Out System
[0019] Hereinafter, embodiments of an article reading device and a
program will be described with reference to drawings. In the
present embodiment, as an example of the article reading device and
the program, a check-out system 1 including an article reading
device 101 provided in a store such as a supermarket is
described.
[0020] FIG. 1 is an exterior perspective view illustrating an
example of the check-out system 1. As illustrated in FIG. 1, the
check-out system 1 includes the article reading device 101 which
captures an image of an article (or merchandise) which is a
registration target when performing sales registration and reads
feature data included in the captured image of the article. The
article reading device also reads identification information of a
barcode or the like attached to the article. In addition, the
check-out system 1 includes a merchandise settlement device (POS
(Point-of-Sale) terminal) 11 which performs sales registration for
merchandise read by the article reading device 101 and performs a
settlement process. The POS terminal 11 chooses a merchandise
candidate based on the feature data of the article output by the
article reading device 101. In addition, the POS terminal 11
provides the merchandise candidate to a salesperson (e.g., cashier)
and allows the salesperson to select the correct merchandise. In
the check-out system 1 of the embodiment, the article reading
device 101 specifies merchandise with respect to natural objects
such as vegetables, fruits, meat, and the like by object
recognition. Other merchandise (for example, manufactured objects
and the like) is specified by reading a barcode attached to the
merchandise.
[0021] In an example of FIG. 1, the POS terminal 11 is positioned
on a drawer 21, on a cash register counter 41. The POS terminal 11
has a keyboard 22 on which keys for operation are arranged on an
upper surface of the POS terminal 11. In addition, the POS terminal
11 has a first display device 23 used for displaying registration
information and receiving an input of information required for
settlement by the cashier. A second display device 24 is positioned
above the keyboard 22 and is used for displaying the registration
information to a customer. The first display device 23 has a
display panel 23a such as a liquid crystal panel and is a touch
input type in which a touch panel 26 is overlaid on display panel
23a. The second display device 24 has a display panel 24a such as a
liquid crystal panel. The second display device 24 is positioned at
an upper portion of the POS terminal 11 so as to be rotatable in a
direction easy to see from the customer. The drawer 21 accommodates
bills, coins, and the like therein and is opened by an instruction
from the POS terminal 11.
[0022] Next to the cash register counter 41, a table 151 arranged
is provided, forming an L-shape together with the cash register
counter 41. On an upper surface of the table 151, a load receiving
surface 152 for placing a shopping basket 153 and the like is
formed. FIG. 1 illustrates an example in which the shopping basket
153 includes a first shopping basket 153a and a second shopping
basket 153b. The first shopping basket 153a is a shopping basket
which the customer brought, that is, a shopping basket containing
articles (for example, an article G) before reading. The second
shopping basket 153b is a shopping basket for accommodating the
article G after reading. After the cashier picks up, for example,
the article G from the first shopping basket 153a and causes the
article reading device 101 to read the article G, the cashier
transfers the article G to the second shopping basket 153b. The
shopping basket 153 is not limited to a basket shape but may be a
tray shape, a box shape, a bag shape, or the like.
[0023] The article reading device 101 is positioned at a central
portion of the load receiving surface 152 of the table 151 and is
connected with the POS terminal 11 so as to be able to transmit and
receive data by wire or wireless. The article reading device 101
includes a reading unit 110 in which a reading window 103 is
disposed on a front surface (cashier side) in a housing 102. The
reading window 103 is made of, for example, a transparent glass
plate or the like with transparency. In addition, the article
reading device 101 includes a display/operation unit 104 at an
upper portion of the housing 102. The display/operation unit 104
has a third display device 106 such as a liquid crystal panel. The
cashier uses the display/operation unit 104 for checking
information of an image and the like read by the reading unit 110
and for an input of information when selecting merchandise among
merchandise candidates recognized by the article reading device
101. A touch panel 105 is overlaid on a screen of the third display
device 106 so that a touch input can be performed. In addition, a
keyboard 107 is provided on a right side of the third display
device 106, so that a keyboard input can be performed by the
cashier. Further, the article reading device 101 includes a fourth
display device 109 on a rear side (e.g., customer side) of the
display/operation unit 104 for the customer to check a reading
result and the like of the article G by the customer.
[0024] The reading unit 110 includes an imaging unit 164 for
imaging the article G in a back of the reading window 103. When the
article G is imaged by the imaging unit 164, the reading unit 110
extracts feature data of the article G among images of the imaged
article G and outputs the extracted feature data to the POS
terminal 11. In addition, when recognizing that a code symbol such
as a barcode for identifying the article G exists in the captured
image, the reading unit 110 reads contents of the barcode and
specifies the article G.
[0025] The display/operation unit 104 displays various screens such
as a pre-set screen, a selection screen of a merchandise candidate,
and the like output from the POS terminal 11 to the third display
device 106. The pre-set screen includes a plurality of selection
buttons for selecting merchandise. The selection screen is a screen
on which merchandise candidate information read by the reading unit
110 is displayed. Details of a screen configuration will be
described below.
[0026] In addition, the display/operation unit 104 displays an
image captured by the imaging unit 164 on the third display device
106 which is an example of a display unit. By viewing the captured
image displayed on the third display device 106, the cashier checks
whether or not the article G is properly captured. The cashier
checks the captured image displayed on the third display device
106. When an inappropriate image is captured for reading such as
occurrence of frame-out of the article G, occurrence of a blur, or
the like, the cashier instructs the reading unit 110 to read the
article G again.
Descriptions of Hardware Configuration of Check-Out System
[0027] Next, a hardware configuration of the POS terminal 11 and
the article reading device 101 will be described. FIG. 2 is a block
diagram illustrating the hardware configuration of the POS terminal
11 and the article reading device 101. The POS terminal 11 includes
a Central Processing Unit (CPU) 61, a Read Only Memory (ROM) 62, a
Random Access Memory (RAM) 63, a Hard Disk Drive (HDD) 64, a
connection interface 65, a printer 66, the drawer 21, the keyboard
22, the first display device 23, the second display device 24, a
communication interface 25, the touch panel 26, and the like. The
CPU 61, the ROM 62, and the RAM 63 are connected with each other by
an internal bus 27. In addition, all of the drawer 21, the keyboard
22, the first display device 23, the second display device 24, the
communication interface 25, the touch panel 26, the HDD 64, the
connection interface 65, and the printer 66 are connected to the
internal bus 27 described above via various input/output circuits
(not illustrated)
[0028] The CPU 61 is a central processing unit which controls the
overall POS terminal 11. The ROM 62 is a nonvolatile memory which
stores fixed programs and the like. The RAM 63 is a volatile memory
used by the CPU 61 as a work area or the like.
[0029] The HDD 64 is a memory unit which stores various programs or
various files. The various programs include a program PR for
merchandise sales data processing including a process of
determining a merchandise candidate and displaying information
indicating the merchandise candidate on a selection screen, and the
like. The various files include a PLU file F1 and the like that is,
for example, transmitted from a store computer SC and stored. In
addition, the HDD 64 stores a registration table, a sales table, or
the like for registering merchandise.
[0030] The communication interface 25 is configured with a network
card or the like for performing data communication with the store
computer SC. The store computer SC is installed in a back office or
the like of a store. The store computer SC may store the PLU file
F1 and the like for transmitting to the POS terminal 11 in an HDD
(not illustrated).
[0031] The connection interface 65 is an interface for
communicating with a connection interface 175 or a connection
interface 176 of the article reading device 101. The communication
is performed by wire or wireless. The printer 66 is a printing
device which prints transaction contents on a receipt or the like
and discharges the receipt.
[0032] The article reading device 101 includes the reading unit 110
and the display/operation unit 104. The reading unit 110 has a CPU
161, a ROM 162, a RAM 163, the imaging unit 164, an audio output
unit 165, and the connection interface 175. In addition, the
display/operation unit 104 has the connection interface 176, the
touch panel 105, the third display device 106, the keyboard 107,
and the fourth display device 109.
[0033] The CPU 161, the ROM 162, and the RAM 163 are connected with
each other by an internal bus 166. In addition, all of the imaging
unit 164, the audio output unit 165, and the connection interface
175 are connected to the internal bus 166 described above via
various input/output circuits (not illustrated). Then, all of the
touch panel 105, the third display device 106, the keyboard 107,
and the fourth display device 109 are connected to the connection
interface 176.
[0034] The CPU 161 is a central processing unit which controls the
overall article reading device 101. The ROM 162 is a nonvolatile
memory which stores control programs and the like. The control
program includes a program or the like for a process of extracting
feature data from a captured image and outputting the extracted
feature data to the POS terminal 11. The RAM 163 is a volatile
memory used by the CPU 161 as a work area or the like.
[0035] The imaging unit 164 is a color image sensor having an
imaging element such as a Charge Coupled Device (CCD), a
Complementary Metal Oxide Semiconductor (CMOS), or the like. The
imaging unit 164 starts imaging by receiving an imaging start
signal from the CPU 161, converts light reflected from the article
G held by the reading window 103 to an electric signal at a
predetermined frame rate (for example, 30 fps), and sequentially
outputs the captured image.
[0036] The audio output unit 165 has a reproduction circuit, such
as a speaker for playing a pre-set reading sound (e.g., "beep"),
warning sound, voice, and the like. The audio output unit 165 plays
the reading sound, the warning sound, the voice, and the like based
on a control signal from the CPU 161 and notifies a reading state
or the like of the article G by the sound.
[0037] Further, the connection interface 175 is connected to the
CPU 161. The connection interface 175 communicates with the
connection interface 65 of the POS terminal 11 and thereby performs
data transmission and reception between the article reading device
101 and the POS terminal 11.
[0038] The connection interface 176 communicates with the
connection interface 65 of the POS terminal 11 and thereby performs
data transmission and reception between the article reading device
101 and the POS terminal 11.
[0039] Here, the PLU file F1 will be described. The PLU file F1 is
a data file in which merchandise information and feature data of
the merchandise are associated with each other for each of the
articles G. The merchandise information is merchandise
identification information (merchandise ID or the like) for
uniquely identifying merchandise, merchandise classification, an
illustration image showing merchandise, information indicating a
unit price, and the like. The feature data includes, for example, a
surface state (e.g., texture) such as color tone or unevenness of
the surface, which indicates a feature of each of the articles G
previously extracted from a sample image of each of the articles G
by a predetermined algorithm.
[0040] FIG. 3 illustrates an example of a data configuration of the
PLU file F1. The PLU file F1 stores data in which the merchandise
information and the feature data of the merchandise are associated
with each other for each of the articles G. In an example
illustrated in FIG. 3, data such as a merchandise ID field f1, a
merchandise classification field f2, a merchandise name field f3,
an unit price field f4, an illustration image field f5, and the
like is stored as merchandise information. Data in feature data
field f6 of the merchandise is associated with the merchandise
information. The data stored in the feature data field f6 is used
as data for verification when determining a similarity to be
described below. Data "XXXXXXXX" illustrated in the merchandise ID
field f1 is a value uniquely corresponding to each piece of
merchandise. Data "YYYYYYYY" illustrated in the feature data field
f6 is a value calculated for each piece of merchandise.
Descriptions of Functional Configuration of Check-Out System
[0041] Next, a functional configuration of the POS terminal 11 and
the article reading device 101 of the check-out system 1 will be
described. FIG. 4 is a block diagram illustrating an example
configuration of a functional block of each of the POS terminal 11
and the article reading device 101. In the article reading device
101, the CPU 161 illustrated in FIG. 2 appropriately executes a
program stored in the ROM 162. Accordingly, as illustrated in FIG.
4, an image capturing unit 51, a merchandise detection unit 52, a
barcode recognition unit 53, a feature data extraction unit 54, and
an object recognition application determination unit 55 are
realized as functional units by the CPU 161. In one embodiment, the
CPU 161 is a processor that is programmed to carryout the functions
of the image capturing unit 51, the merchandise detection unit 52,
the barcode recognition unit 53, the feature data extraction unit
54, and the object recognition application determination unit 55.
In another embodiment, the CPU 61 is a hardware controller, e.g.,
an application specific integrated circuit (ASIC) and field
programmable gate array (FPGA), that is configured to carry out the
functions of the image capturing unit 51, the merchandise detection
unit 52, the barcode recognition unit 53, the feature data
extraction unit 54, and the object recognition application
determination unit 55.
[0042] In addition, in the POS terminal 11, the CPU 61 illustrated
in FIG. 2 appropriately executes a program stored in the ROM 62 and
the program PR stored in the HDD 64. Accordingly, as illustrated in
FIG. 4, a merchandise candidate extraction unit 71, a display
control unit 72, an input receiving unit 73, and a sales
registration unit 74 are realized as functional units by the CPU
61. In one embodiment, the CPU 61 is a processor that is programmed
to carryout the functions of the merchandise candidate extraction
unit 71, the display control unit 72, the input receiving unit 73,
and the sales registration unit 74. In another embodiment, the CPU
61 is a hardware controller, e.g., an application specific
integrated circuit (ASIC) and field programmable gate array (FPGA),
that is configured to carry out the functions of the merchandise
candidate extraction unit 71, the display control unit 72, the
input receiving unit 73, and the sales registration unit 74.
[0043] The image capturing unit 51 outputs an imaging start signal
to the imaging unit 164 (FIG. 2) and starts an imaging operation of
the imaging unit 164. After that, the image capturing unit 51
obtains the image captured and output by the imaging unit 164 and
sequentially stores the image in the RAM 163 (FIG. 2).
[0044] The merchandise detection unit 52 detects that the article G
is held near the reading window 103 (FIG. 1) based on a plurality
of successive images captured and output by the imaging unit 164.
Specifically, the merchandise detection unit 52 reads the captured
image output by the imaging unit 164 (FIG. 2) from the RAM 163
(FIG. 2) and compares luminance of the captured image with
luminance of a captured image read most recently. The merchandise
detection unit 52 determines that the captured image includes
reflection of hands, the article G, and the like with a condition
that luminance gradation is changed according to comparison of
luminance. Further, the merchandise detection unit 52 binarizes the
captured image having the reflection and the successive captured
image and extracts a contour line of the article G. When a position
of a contour line of the successive captured image is changed, the
merchandise detection unit 52 specifies that an area inside the
contour line is an area indicating the article G held by the
reading window 103. In this way, the merchandise detection unit 52
detects that the article G is held by the reading window 103.
[0045] A method of specifying that the article G is held is not
limited to the method described above and other methods may be
used. For example, it may be specified not by the contour line of
the article G in the captured image but by presence or absence of a
flesh color area. That is, in a case where there is the flesh color
area inside the captured image, it is expected that the cashier's
hands are reflected. Therefore, for example, a contour line may be
extracted from the captured image and an area inside the contour
line of a position at which the article is gripped may be specified
as an area indicating the article G among contour lines showing a
shape of the hands.
[0046] The barcode recognition unit 53 is an example of a barcode
reading unit and recognizes that a code symbol such as a barcode is
imaged in the captured image. In addition, the barcode recognition
unit 53 performs a reading process of reading a recognized barcode
and provides a reading result to the merchandise candidate
extraction unit 71. In addition, in a case where the code symbol
such as a barcode or the like is not imaged in the captured image,
the barcode recognition unit 53 provides notification that the
barcode is not read to the merchandise candidate extraction unit
71. Since a method of determining whether or not the barcode is
imaged in the captured image or a method of reading code
information of the barcode imaged in the captured image is already
used as a known technique, details of the description will be
omitted.
[0047] The feature data extraction unit 54 is an example of an
extraction unit and extracts feature data 80a included in a
recognition target article for object recognition from an area of
the article G imaged in the captured image. Then, the feature data
extraction unit 54 outputs the extracted feature data 80a to the
merchandise candidate extraction unit 71 of the POS terminal 11.
Specifically, the feature data extraction unit 54 reads a captured
image from the RAM 163. Then, the feature data extraction unit 54
specifies an area of the article G from the captured image. Next,
the feature data extraction unit 54 extracts the feature data 80a
corresponding to the feature data field f6 registered in the PLU
file F1 in advance as the feature data 80a included in the
recognition target article for object recognition with respect to
an inside of the specified area. The feature data extraction unit
54 outputs the extracted feature data 80a to the connection
interface 175 (FIG. 2). The connection interface 175 outputs the
received feature data 80a to the connection interface 65 (FIG. 2)
and provides the feature data 80a via the connection interface 65
to the merchandise candidate extraction unit 71.
[0048] In addition, the feature data extraction unit 54 extracts
feature data 80b for determining whether or not the article G is a
non-target article for object recognition from an area of the
article G imaged in the captured image. For example, in the present
example, merchandise is identified by object recognition for
natural objects such as vegetables, fruits, meat, and the like and
the merchandise is identified by a barcode attached to merchandise
for manufactured objects. Then, the feature data extraction unit 54
extracts the feature data 80b for distinguishing whether an area of
the article G imaged in the captured image is manufactured objects
or natural objects. Specifically, the feature data 80b which is not
present in natural objects such as vegetables, fruits, meat, and
the like among feature data common to artificial objects such as
plastic, paper, cloth, or the like is selected and is used.
[0049] The object recognition application determination unit 55 is
an example of a determination unit and determines whether or not
the article G imaged in the captured image is a non-target article
for object recognition based on whether the feature data 80b is
extracted by the feature data extraction unit 54. Further, the
object recognition application determination unit 55 determines
that it is necessary to read a barcode so as to identify the
article G on the condition that the article G imaged in the
captured image is determined to be a manufactured object, that is,
a non-target article for object recognition. On the other hand, the
object recognition application determination unit 55 determines
that it is necessary to perform object recognition so as to
identify the article G on condition that the article G imaged in
the captured image is determined to be a natural object, that is, a
recognition target article for object recognition. A determination
result of the object recognition application determination unit 55
is provided to the merchandise candidate extraction unit 71 and the
display control unit 72.
[0050] The merchandise candidate extraction unit 71 is an example
of an object recognition unit and performs an object recognition
process for recognizing the article G inside the captured image.
Specifically, the merchandise candidate extraction unit 71
calculates a similarity by comparing the feature data 80a of the
article G provided by the feature data extraction unit 54 with the
feature data field f6 of each piece of merchandise registered in
the PLU file F1 (FIG. 3). Further, the merchandise candidate
extraction unit 71 chooses merchandise having similarity is equal
to or larger than a predetermined threshold among the PLU file F1
as a candidate of the article G. The merchandise candidate
extraction unit 71 outputs the chosen merchandise candidate
information to the display control unit 72. In addition, the
merchandise candidate extraction unit 71 identifies the merchandise
by verifying the merchandise ID field f1 read by the barcode
recognition unit 53 with the PLU file F1 with respect to the
article G to which a barcode is attached. The merchandise candidate
extraction unit 71 outputs the identifies merchandise information
to the display control unit 72.
[0051] The object recognition for recognizing the article G
included in an image is also called generic object recognition.
Since various recognition techniques used for generic object
recognition are described in detail in the related art, the
description will be omitted. Example generic object recognition
techniques are describe in the following: Yanai Keiji, "Current and
Future of Generic Object Recognition", Journal of Information
Processing Society, Vol. 48, No. SIG16 [Search on Aug. 5, 2016],
Internet <URL:
http://mm.cs.uec.ac.jp/IPSJ-TCVIM-Yanai.pdf>
[0052] Jamie Shotton, "Semantic Texton Forests for Image
Categorization and Segmentation", [Search on Aug. 5, 2016],
Internet <URL:
http://mi.eng.cam.ac.uk/-cipolla/publications/inproceeding
s/2008-CVPR-semantic-texton-forests.pdf>
[0053] Here, in a case where, for example, the feature data field
f6 of each of articles inside the PLU file F1 is set to
100%="similarity: 1.0", the similarity calculated when the
merchandise candidate extraction unit 71 performs object
recognition and specifies the article G indicates how similar the
feature data 80a of the article G imaged in the captured image is.
A similarity may be calculated by absolute evaluation or relative
evaluation.
[0054] For example, a similarity is calculated by absolute
evaluation. In this case, the feature data 80a of the article G
imaged in the captured image and the feature data field f6 of each
of articles registered in the PLU file F1 are compared one by one
and a similarity calculated as a result of this comparison is
adopted as-is. Here, it is assumed that a similarity of 95% is set
as a threshold. In this case, the merchandise candidate extraction
unit 71 sequentially assigns merchandise having a similarity of 95%
equal to or more in order of similarities and outputs the
merchandise to the display control unit 72.
[0055] In addition, similarities may be calculated by relative
evaluation. In this case, similarities are calculated so that a
total of the similarities to the merchandise of the PLU file F1 is
1.0 (100%). For example, it is assumed that the feature data of
five pieces of merchandise in the PLU file F1 are registered in the
feature data field f6. In this case, for example, a similarity for
the first merchandise is "0.6", a similarity for the second
merchandise is "0.1", a similarity for the third merchandise is
"0.1", a similarity for the fourth merchandise is "0.1", and a
similarity for the fifth merchandise is "0.1", so that a total of
the similarities is always 1.0. Here, it is assumed that a
similarity 0.5 is a threshold. In this case, the merchandise
candidate extraction unit 71 sequentially assigns merchandise
information of merchandise having a similarity 0.5 equal to or more
in order of similarities and outputs the merchandise to the display
control unit 72.
[0056] The display control unit 72 is an example of a display
control unit and controls respective display states of the first
display device 23, the second display device 24, the third display
device 106, and the fourth display device 109 (all are illustrated
in FIG. 2). The display control unit 72 controls the display states
of the first display device 23 and the second display device 24 via
the internal bus 27. In addition, the display control unit 72
controls the display states of the third display device 106 and the
fourth display device 109 via the connection interface 65 and the
connection interface 176. Hereinafter, a configuration will be
described with respect to exemplary embodiments of the present
disclosure, in which the display control unit 72 controls the
display state of the third display device 106.
[0057] The display control unit 72 displays a pre-set screen G1
(see FIG. 5A) on which a plurality of selection buttons for
selecting merchandise are arranged at predetermined positions on
the third display device 106. Further, when receiving merchandise
candidate information from the merchandise candidate extraction
unit 71, the display control unit 72 displays a selection screen G2
(see FIG. 5B) having merchandise candidate information with respect
to the third display device 106.
[0058] Specifically, the display control unit 72 generates update
information of the pre-set screen G1 as a setting indicating a
merchandise candidate by setting a display of selection buttons of,
for example, top four merchandise candidates chosen as merchandise
candidate information. The display control unit 72 outputs update
information of the pre-set screen G1 to the connection interface
65. The output update information is output from the connection
interface 65 to the third display device 106 via the connection
interface 176. Accordingly, a display screen of the third display
device 106 transitions from the pre-set screen G1 to the
merchandise candidate selection screen G2.
[0059] In addition, on condition that the object recognition
application determination unit 55 determines that it is necessary
to perform an object recognition process for specifying the article
G imaged in the captured image, the display control unit 72
controls a display form when a captured image captured by the
imaging unit 164 is displayed on the third display device 106.
Specific control contents of the display form will be described
below.
[0060] The input receiving unit 73 receives various input
operations from an input/output device such as the keyboards 22 and
107 and the touch panels 26 and 105. For example, while a
merchandise candidate selection screen is displayed on the third
display device 106, the cashier presses a selection button of the
keyboard 107 or touches a selection button of the touch panel 105.
In this case, operation information of the operated selection
button is output from the input/output device to the connection
interface 176. The operation information is output from the
connection interface 176 to the connection interface 65 and is
provided to the input receiving unit 73 via the connection
interface 65.
[0061] Based on the operation information of the selection button
received from the keyboard 107 or the touch panel 105 by the input
receiving unit 73, the sales registration unit 74 sales-registers
merchandise information of a piece of merchandise corresponding to
the operation information. That is, the sales registration unit
adds the merchandise information to a registration table or the
like. In addition, based on the closing operation information
received from the touch panel 26 by the input receiving unit 73,
the sales registration unit 74 performs a settlement process of one
transaction of all merchandise that has been sales-registered and
registers the sales information to a sales table or the like.
Descriptions of Contents of Display Screen
[0062] Next, contents of a screen displayed on the third display
device 106 by the check-out system 1 when registering the article G
will be described. FIGS. 5A to 5D illustrate examples of screens
displayed on the third display device 106. FIG. 5A is an example of
the pre-set screen G1 displayed when the check-out system 1 starts
to operate. FIG. 5B is an example of the selection screen G2
displayed when the merchandise candidate extraction unit 71
extracts merchandise candidates from images of the article G
captured by the imaging unit 164. FIG. 5C is an example of a
reading result screen G3 displayed when the barcode recognition
unit 53 reads a barcode attached to the article G. FIG. 5D is an
example of a re-recognition selection screen G4 displayed when the
merchandise candidate extraction unit 71 cannot extract the
merchandise candidates from images of the article G captured by the
imaging unit 164.
[0063] As illustrated in FIG. 5A, the pre-set screen G1 has tabs
T1, T2, . . . . , a selection button display area E1, and the like.
The tabs T1, T2, . . . are buttons for switching selection buttons
by categories and displaying the selection buttons. The selection
button display area E1 is a display area in which selection buttons
K1, K2, K3, K4, K5, . . . of a category designated by the tabs T1,
T2, . . . are displayed at respective positions determined.
[0064] In a case of performing sales registration of merchandise
without performing the object recognition process or the barcode
reading process described above, the cashier designates a category
among the tabs T1, T2, . . . by a touch input of the screen and
further selects the corresponding selection buttons K1, K2, K3, K4,
K5, . . . from the selection button display area E1 corresponding
to the selected category. Operation information of the selection
buttons K1, K2, K3, K4, K5, . . . selected by this operation is
received by the input receiving unit 73 of the POS terminal 11.
Merchandise information corresponding to the operated selection
buttons K1, K2, K3, K4, K5, . . . is retrieved from the PLU file F1
(FIG. 2) and is sales-registered by the sales registration unit 74
(FIG. 4).
[0065] In a case where the barcode recognition unit 53 (FIG. 4)
performs a barcode reading process of reading a barcode of the
article G and recognizes contents of the barcode, the display
control unit 72 displays the reading result screen G3 illustrated
in FIG. 5C. That is, the display control unit 72 displays only a
selection button (in this case, the selection button K5)
corresponding to the contents of the recognized barcode in the
selection button display area E1 of the reading result screen G3 in
an operable state. The display control unit 72 displays selection
buttons other than the selection button K5 in an inoperable state,
such as greyed-out on the reading result screen G3. After the
cashier checks that the selection button K5 displayed in the
operable state corresponds to the article G, the cashier presses
the selection button K5. Then, operation information of the
selection button K5 is received by the input receiving unit 73 of
the POS terminal 11. Merchandise information corresponding to the
operated selection button K5 is called from the PLU file F1 (FIG.
2) and is sales-registered by the sales registration unit 74 (FIG.
4). Here, a configuration in which a result of the barcode reading
process is displayed as the reading result screen G3 and the
cashier performs sales registration after checking the displayed
contents is described, the configuration is not limited thereto.
That is, on condition that the article G is specified based on the
result of the barcode reading process, sales registration may be
performed without the cashier re-checking a result of specifying
the article G.
[0066] Further, as illustrated in FIG. 5B, in a case where it is
necessary to perform object recognition for specifying the article
G imaged in the captured image, the display control unit 72
displays the selection screen G2 having merchandise candidates
obtained by the object recognition process. At this time, the
display control unit 72 simultaneously displays an image of the
article G captured by the imaging unit 164 in a captured image
display area E2. At this time, the display control unit 72
superimposes and displays an object detection frame W1 indicating a
position of the article G object-recognized by the merchandise
candidate extraction unit 71 on the image of the article G. The
cashier sees the displayed image and checks that the article G is
properly imaged in the captured image display area E2 and that the
object detection frame W1 is correctly superimposed on the article
G.
[0067] The display control unit 72 displays, for example, the top
four pieces of merchandise with the highest similarity among
merchandise candidates extracted by the merchandise candidate
extraction unit 71 (FIG. 4) in a merchandise candidate display area
E3 on the selection screen G2. That is, the display control unit 72
displays selection buttons (K10, K11, K12, and K13) in the
merchandise candidate display area E3.
[0068] In addition, on the condition that the article G imaged in
the captured image is a target article for object recognition and
the merchandise candidate extraction unit 71 cannot extract a
merchandise candidate from the captured image, the display control
unit 72 displays the re-recognition selection screen G4 suggesting
execution of re-recognition illustrated in FIG. 5D. That is, the
display control unit 72 displays a message suggesting execution of
re-recognition and a re-recognition start button K20 in the
captured image display area E2 of the re-recognition selection
screen G4. Further, the display control unit 72 displays the
merchandise candidate display area E3 of the re-recognition
selection screen G4 in an inoperable state such as greyed-out.
Descriptions of Screen Transition in Third Display Device
[0069] Next, screen transition displayed on the third display
device 106 by the check-out system 1 will be described. FIG. 6 is a
state transition diagram illustrating an example of screen
transitions on the third display device 106.
[0070] In a state (node N1) where the pre-set screen G1 (FIG. 5A)
is displayed on the third display device 106, when the object
recognition application determination unit 55 determines that the
article G is a target article for object recognition, the
merchandise candidate extraction unit 71 performs the object
recognition process. Then, when a merchandise candidate is
extracted (arc A1), the display control unit 72 displays the
selection screen G2 having the captured image and the merchandise
candidate on the third display device 106 (node N2).
[0071] On the other hand, in a state of the node N1, when the
merchandise candidate extraction unit 71 performs the object
recognition process and the merchandise candidate is not extracted
(arc A2), the display control unit 72 displays the re-recognition
selection screen G4 on the third display device 106 (node N4).
[0072] Further, in a state of the node N1, when the object
recognition application determination unit 55 determines that the
article G is a non-target article for object recognition, the
barcode recognition unit 53 performs a barcode reading process (arc
A3). Then, the display control unit 72 displays the reading result
screen G3 on the third display device 106 (node N3).
[0073] In a state (node N2) where the selection screen G2 is
displayed on the third display device 106, when the cashier presses
the selection buttons (K10, K11, K12, and K13) (FIG. 5B)
corresponding to the article G in the selection screen G2 and
performs a registration operation (arc A4), the sales registration
unit 74 (FIG. 4) registers the article G as purchased merchandise.
Then, the display control unit 72 displays the pre-set screen G1 on
the third display device 106 (node N1).
[0074] In a state (node N4) where the re-recognition selection
screen G4 is displayed on the third display device 106, when the
cashier presses the re-recognition start button K20 (FIG. 5D), the
article reading device 101 performs the object recognition process
again. As a result, when a merchandise candidate is extracted (arc
A5), the display control unit 72 displays the selection screen G2
having the captured image and the merchandise candidate on the
third display device 106 (node N2).
[0075] On the other hand, in a state of the node N4, when the
merchandise candidate extraction unit 71 performs the object
recognition process again and the merchandise candidate is not
extracted (arc A6), the display control unit 72 displays the
re-recognition selection screen G4 on the third display device 106
again (node N4).
[0076] In a state (node N3) where the reading result screen G3 is
displayed on the third display device 106, when the cashier presses
the selection button K5 (FIG. 5C) selected by the barcode reading
process and performs the registration operation on the reading
result screen G3 (arc A7), the sales registration unit 74 registers
the article G as purchased merchandise. Then, the display control
unit 72 displays the pre-set screen G1 on the third display device
106 (node N1).
Descriptions of Operation of Check-Out System
[0077] Next, an operation example of the check-out system 1 will be
described. FIG. 7 is a sequence diagram illustrating a process
example for performing one transaction of merchandise registration
and a settlement process of the check-out system 1. In the
following description, in the POS terminal 11, it is assumed that
the program stored in the ROM 62 and the program PR stored in the
HDD 64 are executed by the CPU 61 and functional units such as the
merchandise candidate extraction unit 71, the display control unit
72, the input receiving unit 73, the sales registration unit 74,
and the like are realized. In addition, in the article reading
device 101, it is assumed that the program stored in the ROM 162 is
executed by the CPU 161 and functional units such as the image
capturing unit 51, the merchandise detection unit 52, the barcode
recognition unit 53, the feature data extraction unit 54, the
object recognition application determination unit 55, and the like
are realized. Further, in the article reading device 101, the
pre-set screen G1 (FIG. 5A) is displayed on the third display
device 106 and the imaging unit 164 is also always activated.
[0078] First, the article reading device 101 detects that the
article G is held by the reading window 103 and performs a reading
process of the article G (step S1). Specifically, when the cashier
picks up the article G before reading from the first shopping
basket 153a illustrated in FIG. 1 and moves the article G to the
second shopping basket 153b, the article G is held by the reading
window 103 and an appearance of the merchandise is read by the
imaging unit 164 (FIG. 2). The article reading device 101 obtains
an image captured by the imaging unit 164 from the image capturing
unit 51 (FIG. 4) and stores the image in the RAM 163. Then, the
merchandise detection unit 52 detects the article G from the image.
At this time, the feature data extraction unit 54 extracts the
feature data 80b of the detected article G and the object
recognition application determination unit 55 determines whether or
not the article G is a non-target article for object recognition
based on the feature data 80b.
[0079] Next, the CPU 161 of the article reading device 101 provides
the feature data 80a extracted by the feature data extraction unit
54 to the merchandise candidate extraction unit 71 of the POS
terminal 11 (step S2). Further, based on a determination result of
the object recognition application determination unit 55, on
condition that the article G is not a non-target article for object
recognition, the CPU 161 provides instructions to perform the
object recognition process to the merchandise candidate extraction
unit 71.
[0080] In the POS terminal 11, the merchandise candidate extraction
unit 71 performs the object recognition process and extracts
merchandise information from the PLU file F1 in descending order of
similarities based on the feature data 80a notified from the
article reading device 101. The merchandise candidate extraction
unit 71 outputs the extracted merchandise candidate information to
the display control unit 72 (step S3). The number of pieces of
output merchandise candidate information may be appropriately set.
In the present embodiment, the top four pieces of merchandise
candidate information are extracted.
[0081] Further, in the POS terminal 11, the display control unit 72
outputs the selection buttons (K10, K11, K12, and K13) (FIG. 5B)
corresponding to top four merchandise candidates extracted as
merchandise candidate information to the third display device 106
of the article reading device 101 as update information of the
pre-set screen G1 (step S4).
[0082] Based on the update information output by the display
control unit 72, the article reading device 101 displays the
selection screen G2 (FIG. 5B) of the merchandise candidates to the
third display device 106 (step S5).
[0083] The cashier selects a selection button (any one of K10, K11,
K12, and K13 illustrated in FIG. 5B) corresponding to the
merchandise candidates of the selection screen G2 via the touch
panel 105 (or the keyboard 107) which is the input receiving unit
73. Then, in the article reading device 101, an input/output
circuit (not illustrated) of the touch panel 105 (or the keyboard
107) notifies operation information of the article reading device
101 to the input receiving unit 73 of the POS terminal 11 (step
S6).
[0084] When the input receiving unit 73 of the POS terminal 11
receives merchandise information corresponding to the article G
selected by the cashier, the sales registration unit 74 registers
the merchandise information to a registration table or the like
(step S7).
[0085] The processes from step S1 to step S7 described above are
performed for all of the articles G in the first shopping basket
153a illustrated in FIG. 1. After that, the cashier presses a sum
key of the keyboard 22 or the touch panel 26 in the POS terminal
11. The input receiving unit 73 of the POS terminal 11 receives a
notification that a pressing operation is performed, and the sales
registration unit 74 performs a merchandise settlement process and
registers sales information to a sales table or the like (step
S8).
Descriptions of Flow of Process Performed by Check-Out System
[0086] Next, a flow of processes performed by the check-out system
1 will be described. FIG. 8 is a flow chart illustrating an example
sequence of operations for a series of merchandise registration
processes performed by the check-out system 1.
[0087] The display control unit 72 displays the pre-set screen G1
(FIG. 5A) on which predetermined selection buttons (K1, K2, . . . )
are displayed on the third display device 106 (step S10).
[0088] The image capturing unit 51 performs an imaging process of
capturing an image of the article G by controlling the imaging unit
164 (FIG. 2) (step S12).
[0089] The merchandise detection unit 52 determines whether or not
merchandise is detected (step S14). When the merchandise detection
unit 52 detects the merchandise (step S14: Yes), the process
proceeds to step S16. Otherwise (step S14: No), the process returns
to step S12.
[0090] The object recognition application determination unit 55
determines whether or not the article G is a non-target article for
object recognition based on the feature data 80b of the article G
imaged in the captured image (step S16). When the article G is a
non-target article for object recognition (step S16: Yes), the
process proceeds to step S18. Otherwise (step S16: No), the process
proceeds to step S30.
[0091] The barcode recognition unit 53 performs a barcode reading
process of reading contents of a barcode with respect to the
captured image (step S18).
[0092] The barcode recognition unit 53 determines whether or not
the contents of the barcode are read (step S20). When the contents
of the barcode are read (step S20: Yes), the process proceeds to
step S22. Otherwise (step S20: No), the process proceeds to step
S26.
[0093] The display control unit 72 displays the reading result
screen G3 (FIG. 5C) on the third display device 106 (step S22).
[0094] The sales registration unit 74 receives a notification that
the cashier performs an operation of selecting merchandise read by
the barcode reading process with respect to the input receiving
unit 73, and performs a merchandise registration process (step
S24). After that, the process in FIG. 8 is ended. Although not
described in the flowchart of FIG. 8, in a case where unregistered
merchandise remains, the process returns to step S10 and the same
process is repeated. In addition, as described above, when the
barcode recognition unit 53 reads the contents of the barcode, a
merchandise registration process may be performed without
displaying the reading result screen G3.
[0095] In step S20, when it is determined that the barcode
recognition unit 53 does not read the contents of the barcode (step
S20: No), the barcode recognition unit 53 determines whether or not
a predetermined time elapses (step S26). When the predetermined
time elapses (step S26: Yes), the process proceeds to step S28.
Otherwise (step S26: No), the process returns to step S18.
[0096] When the barcode recognition unit 53 does not read the
barcode even after the predetermined time elapses (step S26: Yes),
the display control unit 72 displays guidance such as "Please
change the direction to hold the merchandise" or the like on the
third display device 106 (step S28). After that, the process
returns to step S18.
[0097] In step S16, when the object recognition application
determination unit 55 determines that the article G imaged in the
captured image is not a non-target article for object recognition
(step S16: No), the merchandise candidate extraction unit 71
performs the object recognition process (step S30).
[0098] The display control unit 72 displays the captured image on
the third display device 106 (step S32).
[0099] As a result of performing the object recognition process,
the merchandise candidate extraction unit 71 determines whether or
not a merchandise candidate is obtained (step S34). When the
merchandise candidate is obtained (step S34: Yes), the process
proceeds to step S36. When the merchandise candidate is not
obtained (step S34: No), the process proceeds to step S40.
[0100] The display control unit 72 displays the selection screen G2
(FIG. 5B) having top four merchandise candidates in descending
order of similarities on the third display device 106 (step
S36).
[0101] The input receiving unit 73 determines whether or not
merchandise is selected from the selection screen G2 (step S38).
When the merchandise is selected (step S38: Yes), the process
proceeds to step S24. Otherwise (step S38: No), the process returns
to step S36.
[0102] In step S34, when the merchandise candidate is not obtained
(step S34: No), the display control unit 72 displays the
re-recognition selection screen G4 (FIG. 5D) suggesting execution
of re-recognition on the third display device 106 (step S40).
[0103] The input receiving unit 73 determines whether or not
re-recognition is instructed (step S42). When the re-recognition is
instructed (step S42: Yes), the process proceeds to step S30.
Otherwise (step S42: No), the process returns to step S40.
[0104] In the present embodiment, on the third display device 106,
the display control unit 72 displays captured images which have not
yet been displayed at a timing during which object recognition is
executed and removes an image of the article G at a timing during
which the merchandise candidate is selected. Display control
performed by the display control unit 72 is not limited thereto.
That is, the display control unit 72 may always display the
captured image captured by the imaging unit 164 in a display screen
of the third display device 106 with a predetermined size and may
perform display control to enlarge a display size of the captured
image at a timing during which object recognition is executed. At
this time, the display control unit 72 returns the captured image
enlarged to a predetermined size at a timing during which the
cashier selects a merchandise candidate.
Descriptions of Flow of Modification Example of Process Performed
by Check-Out System
[0105] Next, a modification example of a series of processes
performed by the check-out system 1 will be described with
reference to FIG. 9. Since a configuration of the modification
example is the same as that of the check-out system 1 described
above, each of components will be described using the same
reference numerals.
[0106] In this modification example, an operation of the object
recognition application determination unit 55 is different from the
configuration described above. That is, in the configuration (FIG.
8) described above, the object recognition application
determination unit 55 determines whether or not the article G is a
non-target article for object recognition based on the feature data
80b of the article G imaged in the captured image. However, in the
modification example (FIG. 9) described below, the object
recognition application determination unit 55 determines whether or
not the article G is a non-target article for object recognition
based on a barcode reading result from the article G.
[0107] FIG. 9 is a flow chart illustrating the modification example
of a flow of a series of processes performed by the check-out
system 1. The display control unit 72 displays the pre-set screen
G1 (FIG. 5A) on which predetermined selection buttons (K1, K2, . .
. ) are disposed on the third display device 106 (step S50).
[0108] The image capturing unit 51 performs an imaging process of
capturing an image of the article G by controlling the imaging unit
164 (FIG. 2) (step S52).
[0109] The merchandise detection unit 52 determines whether or not
merchandise is detected (step S54). When the merchandise detection
unit 52 detects the merchandise (step S54: Yes), the process
proceeds to step S56. Otherwise (step S54: No), the process returns
to step S52.
[0110] The barcode recognition unit 53 performs a barcode reading
process of reading contents of a barcode with respect to the
captured image (step S56).
[0111] The barcode recognition unit 53 determines whether or not
the contents of the barcode are read (step S58). When the contents
of the barcode are read (step S58: Yes), the process proceeds to
step S60. Otherwise (step S58: No), the process proceeds to step
S64.
[0112] The display control unit 72 displays the reading result
screen G3 (FIG. 5C) on the third display device 106 (step S60).
[0113] The sales registration unit 74 receives notification that
the cashier performs an operation of selecting merchandise read by
the barcode reading process with respect to the input receiving
unit 73, and performs a merchandise registration process (step
S62). After that, the process in FIG. 9 is ended. Although not
described in a flowchart of FIG. 9, in a case where unregistered
merchandise remains, the process returns to step S50 and the same
process is repeated. In addition, when the barcode recognition unit
53 reads the contents of the barcode, a merchandise registration
process may be performed without displaying the reading result
screen G3.
[0114] In step S58, when it is determined that the barcode
recognition unit 53 does not read the contents of the barcode (step
S58: No), the barcode recognition unit 53 determines whether or not
a predetermined time elapses (step S64). When the predetermined
time elapses (step S64: Yes), the process proceeds to step S66.
When the predetermined time does not elapse (step S64: No), the
process returns to step S56.
[0115] In step S64, when it is determined that the barcode
recognition unit 53 does not read the barcode even after the
predetermined time elapses (step S64: Yes), the object recognition
application determination unit 55 determines that the article G
imaged in the captured image is a target article for object
recognition and the merchandise candidate extraction unit 71
performs the object recognition process (step S66).
[0116] The display control unit 72 displays the captured image on
the third display device 106 (step S68).
[0117] As a result of performing the object recognition process,
the merchandise candidate extraction unit 71 determines whether or
not a merchandise candidate is obtained (step S70). When the
merchandise candidate is obtained (step S70: Yes), the process
proceeds to step S72. When the merchandise candidate is not
obtained (step S70: No), the process proceeds to step S76.
[0118] The display control unit 72 displays the selection screen G2
(FIG. 5B) having top four merchandise candidates in descending
order of similarities on the third display device 106 (step
S72).
[0119] The input receiving unit 73 determines whether or not
merchandise is selected from the selection screen G2 (step S74).
When the merchandise is selected (step S74: Yes), the process
proceeds to step S62. Otherwise (step S74: No), the process returns
to step S72.
[0120] In step S70, when the merchandise candidate is not obtained
(step S70: No), the display control unit 72 displays the
re-recognition selection screen G4 (FIG. 5D) suggesting execution
of re-recognition on the third display device 106 (step S76).
[0121] The input receiving unit 73 determines whether or not
re-recognition is instructed (step S78). When the re-recognition is
instructed (step S78: Yes), the process returns to step S66.
Otherwise (step S78: No), the process returns to step S76.
[0122] As described above, in the check-out system 1 according to
the embodiment, based on the feature data 80b of the article G
extracted by the feature data extraction unit 54, the object
recognition application determination unit 55 determines whether
the article G is a target article requiring a recognition result of
the merchandise candidate extraction unit 71 or a target article
requiring a reading result of the barcode recognition unit 53 among
captured images obtained by imaging an appearance of the article G.
Based on a determination result of the object recognition
application determination unit 55, the display control unit 72
controls a display state of the captured image for the third
display device 106. Therefore, in a case where the article G is a
target article requiring a recognition result of the merchandise
candidate extraction unit 71, it is possible to display the
captured image of the article G for easy viewing. Thereby, when an
image of the article G is captured, it is possible to prevent the
article G from protruding from an imaging range of merchandise and
to capture the image for easy recognizing.
[0123] In addition, in the check-out system 1 according to the
embodiment, based on the feature data 80b, extracted by the feature
data extraction unit 54, representing a similarity with a
non-target article for object recognition, the object recognition
application determination unit 55 distinguishes whether or not the
article G is a non-target article for object recognition.
Therefore, it is possible to reliably determine whether or not
object recognition is necessary when registering the article G.
[0124] In the check-out system 1 according to the embodiment, on
condition that a code symbol cannot be read within a predetermined
time after the barcode recognition unit 53 starts to read the code
symbol of the article G, the object recognition application
determination unit 55 determines that the article G is a target
article for object recognition. Therefore, it is possible to
reliably determine whether or not object recognition is necessary
when registering the article G.
[0125] Further, in the check-out system 1 according to the
embodiment, on condition that the object recognition application
determination unit 55 determines that the article G is a target
article for object recognition, the display control unit 72
displays the captured image of the article G on the third display
device 106. Then, on condition that it is determined that the
article G is a non-target article for object recognition, the
object recognition application determination unit 55 does not
display the captured image of the article G on the third display
device 106. Therefore, when the object recognition process is
necessary, it is possible to display the captured image of the
article G for easy viewing without performing a special
operation.
[0126] In addition, in the check-out system 1 according to the
embodiment, on condition that the object recognition application
determination unit 55 determines that the article G is a non-target
article for object recognition, the display control unit 72
displays the captured image on the third display device 106 at a
predetermined display size. Then, on condition that it is
determined that the article G is a target article for object
recognition, the object recognition application determination unit
55 enlarges the captured image more than the predetermined display
size and displays the captured image on the third display device
106. Therefore, when the object recognition process is necessary,
it is possible to display the captured image of the merchandise for
easy viewing without performing a special operation.
[0127] While certain embodiments have been described, these
embodiments have been presented by way of example only, and are not
intended to limit the scope of the inventions. Indeed, the novel
embodiments described herein may be embodied in a variety of other
forms; furthermore, various omissions, substitutions and changes in
the form of the embodiments described herein may be made without
departing from the spirit of the inventions. The accompanying
claims and their equivalents are intended to cover such forms or
modifications as would fall within the scope and spirit of the
inventions.
* * * * *
References