U.S. patent application number 15/848011 was filed with the patent office on 2018-06-28 for image processing apparatus and image processing method.
The applicant listed for this patent is Toshiba Tec Kabushiki Kaisha. Invention is credited to Norimasa ARIGA, Kazuki TAIRA, Masaaki YASUNAGA.
Application Number | 20180181793 15/848011 |
Document ID | / |
Family ID | 62630683 |
Filed Date | 2018-06-28 |
United States Patent
Application |
20180181793 |
Kind Code |
A1 |
ARIGA; Norimasa ; et
al. |
June 28, 2018 |
IMAGE PROCESSING APPARATUS AND IMAGE PROCESSING METHOD
Abstract
An autonomous inventory tracking apparatus includes an image
acquisition unit configured to acquire an image, and a processor
configured to detect a calibration plate in the image acquired from
the image acquisition unit, calculate a color correction value for
the image according to a color block of the calibration plate, the
color block matching a reference value, correct color in the image
using the calculated color correction value to provide a
color-corrected image, and perform commodity recognition processing
on the color-corrected image so as to identify the commodity in the
image acquired from the image acquisition unit.
Inventors: |
ARIGA; Norimasa; (Izunokuni
Shizuoka, JP) ; TAIRA; Kazuki; (Ota Tokyo, JP)
; YASUNAGA; Masaaki; (Sunto Shizuoka, JP) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Toshiba Tec Kabushiki Kaisha |
Tokyo |
|
JP |
|
|
Family ID: |
62630683 |
Appl. No.: |
15/848011 |
Filed: |
December 20, 2017 |
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
H04N 1/6033 20130101;
G06K 9/03 20130101; G06T 7/90 20170101; G06T 2207/10024 20130101;
G06T 7/74 20170101; G06K 9/00208 20130101; G06K 9/4652 20130101;
G06F 16/583 20190101; G06T 1/0007 20130101; G06K 9/3216
20130101 |
International
Class: |
G06K 9/00 20060101
G06K009/00; G06T 7/73 20060101 G06T007/73; G06F 17/30 20060101
G06F017/30; G06T 7/90 20060101 G06T007/90; G06T 1/00 20060101
G06T001/00 |
Foreign Application Data
Date |
Code |
Application Number |
Dec 22, 2016 |
JP |
2016-249568 |
Claims
1. An autonomous inventory tracking apparatus, comprising: an image
acquisition unit configured to acquire an image; and a processor
configured to: detect a calibration plate in the image acquired
from the image acquisition unit; calculate a color correction value
for the image according to a color block of the calibration plate,
the color block matching a reference value; correct color in the
image using the calculated color correction value to provide a
color-corrected image; and perform commodity recognition processing
on the color-corrected image so as to identify a commodity in the
image acquired from the image acquisition unit.
2. The autonomous inventory tracking apparatus according to claim
1, wherein the calibration plate includes a plurality of blocks
having colors that match different reference color values, and the
processor is configured to generate a correction matrix for
correcting R, G, and B values of each pixel of the image.
3. The autonomous inventory tracking apparatus according to claim
1, wherein the calibration plate includes a plurality of blocks
having colors that match different reference color values.
4. The autonomous inventory tracking apparatus according to claim
1, further comprising: a robot configured to move along a route
including a plurality of predetermined stopping positions; and a
camera on the robot, wherein at each predetermined stopping
position a calibration plate is disposed at a position viewable by
the camera, and the image acquisition unit is configured to acquire
an image captured using the camera at each predetermined stopping
position at the respective predetermined stopping positions.
5. The autonomous inventory tracking apparatus according to claim
4, further comprising: a positional adjustment unit configured to
adjust a position of the robot at each predetermined stopping
position by reference to a detected position of the calibration
plate in the image captured using the camera at the predetermined
stopping position, wherein the image acquisition unit is configured
to acquire another image if the position of the robot has been
adjusted.
6. The autonomous inventory tracking apparatus according to claim
4, wherein the image acquisition unit is mounted on the robot.
7. The image processing apparatus according to claim 1, wherein the
process to identify the commodity includes comparison of the
color-corrected image to a dictionary image of the commodity
retrieved from an external non-volatile memory via an
interface.
8. An inventorying apparatus, comprising a robot configured to move
along a route including a plurality of predetermined stopping
positions; a camera on the robot; and an image acquisition unit
configured to acquire an image captured using the camera at each
predetermined stopping position; and a processor configured to:
detect a calibration plate in the image acquired from the image
acquisition unit; calculate a color correction value for the image
according to a color block of the calibration plate, the color
block matching a reference value; correct color in the image using
the calculated color correction value to provide a color-corrected
image; and perform commodity recognition processing on the
color-corrected image so as to identify a commodity in the image
acquired from the image acquisition unit.
9. The inventorying apparatus according to claim 8, wherein the
calibration plate includes a plurality of blocks having colors that
match different reference color values, and the processor is
configured to generate a correction matrix for correcting R, G, and
B values of each pixel of the image.
10. The inventorying apparatus according to claim 8, wherein the
calibration plate includes a plurality of blocks having colors that
match different reference color values.
11. The inventorying apparatus according to claim 8, further
comprising: a positional adjustment unit configured to adjust a
position of the robot at each predetermined stopping position by
reference to a detected position of the calibration plate in the
image captured using the camera at the predetermined stopping
position, wherein the image acquisition unit is configured to
acquire another image if the position of the robot has been
adjusted.
12. The inventorying apparatus according to claim 8, wherein the
image acquisition unit is mounted on the robot.
13. The inventorying apparatus according to claim 8, wherein the
process to identify the commodity includes comparison of the
color-corrected image to a dictionary image of the commodity
retrieved from an external non-volatile memory via an
interface.
14. An inventory tracking method, comprising: moving a robot along
a route including a plurality of predetermined stopping positions;
acquiring an image captured using a camera on the robot at each
predetermined stopping position, the image depicting a commodity
and a calibration plate positioned in proximity to the commodity,
wherein the calibration plate includes a color block having a color
that matches a reference color value; and calculating a color
correction value for the color of the color block according to the
reference value color value; correcting color in the image using
the calculated color correction value to provide a color-corrected
image; and processing the color-corrected image so as to identify
the commodity.
15. The inventory tracking method according to claim 14, further
comprising: generating a correction matrix for correcting R, G, and
B values of each pixel of the image, wherein the calibration plate
includes a plurality of blocks having colors that match different
reference color values.
16. The inventory tracking method according to claim 14, wherein
the calibration plate includes a plurality of blocks having
different colors matching different reference color values.
17. The inventory tracking method according to claim 14, further
comprising: adjusting a position of the robot at each predetermined
stopping position by reference to position of the calibration in
the image captured using the camera at the predetermined stopping
position; and acquiring another image including the calibration
plate if the position of the robot has been adjusted.
18. The inventory tracking method according to claim 14, wherein
the image acquisition unit is mounted on the robot.
19. The inventory tracking method according to claim 14, wherein
the process to identify the commodity includes comparison of the
color-corrected image to a dictionary image of the commodity
retrieved from an external non-volatile memory via an interface.
Description
CROSS-REFERENCE TO RELATED APPLICATION
[0001] This application is based upon and claims the benefit of
priority from Japanese Patent Application No. 2016-249568, filed
Dec. 22, 2016, the entire contents of which are incorporated herein
by reference.
FIELD
[0002] Embodiments described herein relate generally to an image
processing apparatus and an image processing method.
BACKGROUND
[0003] In the related art, an image processing apparatus acquires
an image of a shelf or the like on which a commodity is disposed,
and recognizes the commodity in the image. A color of the commodity
in the image may be different from the color pre-registered for the
commodity due to various factors such as differences in a position
and characteristics of a camera used to capture the image of the
commodity on the shelf, and lighting around the shelf when the
image was acquired.
[0004] In some related art examples, a mobile robot must internally
maintain an environment map so as to match the external environment
of the mobile robot to prevent a self-positioning function of the
mobile robot from deteriorating.
[0005] However, even though such an environment map is created so
as to match the surrounding external environment, an error in the
stopping position of the robot may still occur. Accordingly,
repeated images from the exact same position will not necessarily
be obtained by such a robot, and the deterioration of the commodity
recognition may occur due to the difference in colors of the
commodity, if such color differences are not compensated.
[0006] If the color of the commodity in the image is different from
the pre-registered or expected color of the commodity, accuracy of
recognition of the commodity from the captured image decreases in
the image processing apparatus.
[0007] When an image processing apparatus mounted on an autonomous
driving robot images the commodity shelf in which the commodity is
disposed, the mobile robot can drive a predetermined route based on
estimated self-positioning of the robot. However, if the
environment map stored by the mobile robot is different from the
actual environment, the self-positioning function of the mobile
robot deteriorates and an error in the robot positioning occurs. As
such, there can also be a problem in that a particular commodity
may not be captured as the image correctly due to an error in the
stop positioning of the robot.
DESCRIPTION OF THE DRAWINGS
[0008] FIG. 1 is a block diagram of an image processing system.
[0009] FIG. 2 is a diagram of a dictionary image storage
apparatus.
[0010] FIG. 3 is a diagram of a calibration plate.
[0011] FIG. 4 is a diagram of a commodity table.
[0012] FIG. 5 is a diagram of map information.
[0013] FIG. 6 is a diagram of a route table.
[0014] FIG. 7 is a diagram of a robot apparatus.
[0015] FIG. 8 is a diagram of a robot apparatus.
[0016] FIG. 9 is a diagram of a dictionary image storage
apparatus.
[0017] FIG. 10 is a flowchart of an operation of a dictionary image
storage apparatus.
[0018] FIG. 11 is a flowchart of an operation of a dictionary image
storage apparatus.
[0019] FIG. 12 is a flowchart of an operation of an image
processing apparatus.
[0020] FIG. 13 is a flowchart of an operation of an image
processing apparatus.
DETAILED DESCRIPTION
[0021] In general, according to embodiments, an autonomous
inventory tracking apparatus includes an image acquisition unit
configured to acquire an image, and a processor configured to
detect a calibration plate in the image acquired from the image
acquisition unit, calculate a color correction value for the image
according to a color block of the calibration plate, the color
block matching a reference value, color correct the image using the
calculated color correction value to provide a color-corrected
image, and perform commodity recognition processing on the
color-corrected image so as to identify the commodity in the image
acquired from the image acquisition unit.
[0022] Hereinafter, embodiments will be described with reference to
drawings.
[0023] An image processing system according to embodiments detects
a commodity which is disposed in a commodity shelf based on an
image of the commodity shelf or the like in which the commodity is
disposed, for example, in a store.
[0024] FIG. 1 is a block diagram of an image processing system
1.
[0025] As illustrated in FIG. 1, the image processing system 1
includes an image processing apparatus 10, a dictionary image
storage apparatus 20, a map information apparatus 30, and a missing
commodity recognition apparatus 40.
[0026] The image processing apparatus 10 detects a commodity based
on a captured image of a commodity shelf or the like in which the
commodity is placed.
[0027] The dictionary image storage apparatus 20 stores an image of
a commodity (also referred to as a dictionary image) which is
necessary for the image processing apparatus 10 to detect the
commodity.
[0028] The map information apparatus 30 provides map information
indicating a map inside a store having the image processing system
1 installed to the image processing apparatus 10.
[0029] The missing commodity recognition apparatus 40 detects a
missing commodity based on the commodity that has been detected by
the image processing apparatus 10. The missing commodity
recognition apparatus 40 detects the missing commodity based on
information indicating a commodity that has been pre-registered for
a certain position in the commodity shelf and the commodity that
has been detected at that position in the commodity shelf.
[0030] As illustrated in FIG. 1, the image processing apparatus 10
includes a commodity recognition unit 101, a self-driving robot
102, a camera 103, an operation unit 104, and the like. The
commodity recognition unit 101 includes a central processing unit
(CPU) 11, a random access memory (RAM) 12, a non-volatile memory
(NVM) 13, and interfaces (I/F) 14 to 19. The elements of the
commodity recognition unit 101 are connected to each other through
a data bus or the like. The commodity recognition unit 101 may have
more elements in addition to the elements depicted in FIG. 1, or
some of the elements depicted in FIG. 1 may be omitted in some
embodiments.
[0031] The CPU 11 has a function of controlling the overall
operation of the commodity recognition unit 101. The CPU 11 may
include an internal cache, various interfaces, and the like. The
CPU 11 executes a program stored in an internal memory or NVM 13 in
advance to provide various processing operations. The CPU 11 is,
for example, a processor.
[0032] Some of the processing operations by the CPU 11 executing a
program may be realized via a hardware circuit. In this case, the
CPU 11 controls an operation of the hardware circuit.
[0033] The RAM 12 is a volatile memory. The RAM 12 temporarily
stores data of CPU 11 in processing or the like. The RAM 12 stores
various application programs based on an instruction from the CPU
11. The RAM 12 may store data which is necessary for executing the
application program, an execution result of the application
program, and the like.
[0034] The NVM 13 is a non-volatile memory capable of writing and
rewriting of data. The NVM 13 may be, for example, a (solid-state
drive) SSD, an EEPROM.RTM., a flash memory, or the like. The NVM 13
stores a control program, an application program, and various data
according to an operational purpose of the commodity recognition
unit 101.
[0035] The NVM 13 includes a storage region 13a that stores a
reference color table, a storage region 13b that stores a route
table, and the like.
[0036] The interface 14 is an interface for communicating data with
the dictionary image storage apparatus 20. The CPU 11 acquires the
dictionary image or the like from the dictionary image storage
apparatus 20 through the interface 14.
[0037] The interface 15 is an interface for communicating data with
the self-driving robot 102. For example, the CPU 11 sets a stop
position or the like on the self-driving robot 102 through the
interface 15.
[0038] The interface 16 is an interface for communicating data with
the map information apparatus 30. For example, the CPU 11 acquires
map information from the map information apparatus 30 through the
interface 16.
[0039] The interface 17 is an interface for communicating data with
the camera 103. For example, the CPU 11 acquires a captured image
from the camera 103 through the interface 17.
[0040] The interface 18 is an interface for communicating data with
the operation unit 104. For example, the CPU 11 acquires an
operation input into the operation unit 104 through the interface
18.
[0041] The interface 19 is an interface for communicating data with
the missing commodity recognition apparatus 40. For example, the
CPU 11 transmits information indicating a recognized commodity to
the missing commodity recognition apparatus 40 through the
interface 19.
[0042] The interfaces 14 to 19 support wireless connection or wired
connection protocols. For example, the interfaces 14 to 19 may
support a local area network (LAN) protocol or a Universal Serial
Bus (USB) protocol.
[0043] The self-driving robot 102 is a mobile object that carries
the camera 103. The self-driving robot 102 includes a main body to
mount the camera, a motor, a tire, and the like. The self-driving
robot 102 moves by rotating the tire using driving force of the
motor. The self-driving robot 102 can move with the camera 103
mounted to a predetermined position. The self-driving robot 102 may
include a mechanism for changing a position and an angle of the
camera 103. The self-driving robot 102 may include a sensor for
detecting its own position.
[0044] The self-driving robot 102 moves based on a signal from the
commodity recognition unit 101. For example, the self-driving robot
102 receives information including a stop position from the
commodity recognition unit 101. The self-driving robot 102 moves to
and then stops at the stop position.
[0045] The self-driving robot 102 may determine a stop position
based on a calibration plate. For example, the self-driving robot
102 detects the calibration plate based on an image of the
calibration plate captured by the camera 103. The self-driving
robot 102 may adjust the stop position based on a position of the
detected calibration plate or the like.
[0046] The camera 103 captures images of a commodity. The camera
103 captures an image of a commodity shelf in which the commodity
is disposed. The camera 103 is mounted on the self-driving robot
102 at a predetermined height. For example, the camera 103 is
installed at a height at which a predetermined commodity shelf can
be imaged.
[0047] The camera 103 may capture an image according to a signal
from the commodity recognition unit 101. The camera 103 may capture
an image when the self-driving robot 102 stops at a stop position.
The camera 103 transmits the captured image to the commodity
recognition unit 101.
[0048] The camera 103 is, for example, a charge coupled device
(CCD) camera.
[0049] The operation unit 104 receives various operation
instructions by an operator. The operation unit 104 transmits, to
the commodity recognition unit 101, a signal indicating the
operation instruction input by the operator. The operation unit 104
is, for example, a keyboard, a numeric keypad, and a touch
panel.
[0050] As illustrated in FIG. 1, the dictionary image storage
apparatus 20 includes a CPU 21, a RAM 22, an NVM 23, interfaces
(I/F) 24 and 25, and the like. The elements in the dictionary image
storage apparatus 20 are connected to each other through a data bus
or the like. The dictionary image storage apparatus 20 may have
more elements in addition to the elements depicted in FIG. 1, or
some of the elements depicted in FIG. 1 may be omitted. The
dictionary image storage apparatus 20 is connected to a camera
26.
[0051] The CPU 21 has a function of controlling the overall
operation of the dictionary image storage apparatus 20. The CPU 21
may include an internal cache, various interfaces, and the like.
The CPU 21 executes a program stored in an internal memory or NVM
23 in advance to provide various processing. The CPU 21 is, for
example, a processor.
[0052] Some of the processing operations by the CPU 21 executing a
program may be realized via a hardware circuit. In this case, the
CPU 21 controls an operation of the hardware circuit.
[0053] The RAM 22 is a volatile memory. The RAM 22 temporarily
stores data of CPU 21 in processing or the like. The RAM 22 stores
various application programs based on an instruction from the CPU
21. The RAM 22 may store data which is necessary for executing the
application program, an execution result of the application
program, and the like.
[0054] The NVM 23 is a non-volatile memory capable of writing and
rewriting of data. The NVM 23 may be a hard disk, an SSD, an
EEPROM.RTM., a flash memory, or the like. The NVM 23 stores a
control program, an application program, and various data according
to an operational purpose of the dictionary image storage apparatus
20.
[0055] The NVM 23 includes a storage region 23a that stores a
reference color table, a storage region 23b that stores a commodity
table and the like.
[0056] The interface 25 is an interface for communicating data with
the commodity recognition unit 101. For example, the CPU 21
transmits the dictionary image or the like to the commodity
recognition unit 101 through the interface 25.
[0057] The interface 24 is an interface for communicating data with
the camera 26. For example, the CPU 21 acquires a captured image or
the like from the camera 26 through the interface 24.
[0058] The interfaces 24 and 25 support wireless connection or
wired connection protocols. For example, the interfaces 24 and 25
may support a LAN protocol or a USB protocol.
[0059] The camera 26 captures an image of a commodity. The camera
26 may capture an image according to an operator's instruction. The
camera 26 transmits the captured image to the dictionary image
storage apparatus 20. The camera 26 may transmit the captured image
to the dictionary image storage apparatus 20, at the time of
capturing the image. The camera 26 may transmit the captured images
to the dictionary image storage apparatus 20, at a predetermined
interval or after the camera has captured the predetermined number
of images. The camera 26 may store the image in a detachable
memory, and the dictionary image storage apparatus 20 may acquire
the image from the detachable memory.
[0060] The camera 26 is, for example, a CCD camera or the like.
[0061] The CPU 21 has a function of acquiring an image of a
commodity and a calibration plate captured by the camera 26.
[0062] The CPU 21 acquires a captured image from the camera 26. The
camera 26 may store a captured image to a detachable memory, and
the CPU 21 may acquire the captured image from the detachable
memory.
[0063] FIG. 2 illustrates an operation example in which the CPU 21
acquires a captured image.
[0064] As illustrated in FIG. 2, the camera 26 captures an image of
a commodity 50 and a calibration plate 60. The commodity 50 and the
calibration plate 60 are disposed adjacent each other so as to be
contained in one captured image. The positional relationship
between the commodity 50 and the calibration plate 60 is not
limited to a specific configuration.
[0065] The calibration plate 60 is a plate having a plurality of
blocks in different colors.
[0066] FIG. 3 illustrates an example of the calibration plate
60.
[0067] As illustrated in FIG. 3, the calibration plate 60 includes
blocks 601 to 618. Each of the blocks 601 to 618 has different
colors. Each of the blocks 601 to 618 may be individually membered.
The blocks 601 to 618 may be regions in which inks of predetermined
colors are painted on the calibration plate 60.
[0068] In the example illustrated in FIG. 3, the calibration plate
60 includes 18 blocks. The calibration plate 60 may include three
or more, or nine or more blocks. The number of blocks on the
calibration plate 60 is not limited to a specific number.
[0069] The CPU 21 has a function of generating a correction matrix,
including correction values, based on the captured image of the
calibration plate 60.
[0070] The correction matrix is a matrix that maps a color of a
block of the calibration plate 60 in the captured image to an
actual color of the block. The correction matrix can eliminate
color deviation due to camera characteristics, external light, or
the like.
[0071] The CPU 21 extracts a region of the captured image
corresponding to the calibration plate 60, using a pattern
recognition algorithm or the like.
[0072] Upon extracting the region of the image corresponding to the
calibration plate 60, the CPU 21 extracts color information of each
block of the calibration plate 60 from the extracted image. The CPU
21 detects a position of each block from the extracted image. The
CPU 21 extracts the color information of the position of each
block. Here, the color information indicates colors in RGB
format.
[0073] Upon extracting the color information, the CPU 21 acquires
reference color information of each block of the calibration plate
60.
[0074] The CPU 21 acquires the reference color information from the
reference color table stored in the storage region 23a.
[0075] The reference color table stores the reference color
information, indicating reference colors of blocks of the
calibration plate 60 individually. The reference color indicates a
block color registered in advance. For example, the reference color
may be an actual color specified by an ink used to print the block,
or a block color imaged under a predetermined condition.
[0076] The reference color table stores information indicating a
location of the a block, for example, a position or a label of a
block, and reference color information indicating a reference color
of the block in a correlated manner.
[0077] The reference color information indicates colors, for
example, in RGB format. The reference color information may include
Rs, Gs, and Bs.
[0078] The storage region 23a stores the reference color table in
advance.
[0079] The reference color table stored in the storage region 13a
is the same as the reference color table stored in the storage
region 23a.
[0080] Upon acquiring the reference color information of each
block, the CPU 21 generates a correction matrix based on the color
information and the reference color information of each block.
[0081] For example, the CPU 21 calculates a correction matrix A
satisfying the following Equation (1):
( R s G s B s ) = A ( R G B ) Equation ( 1 ) ##EQU00001##
[0082] Here, Rs, Gs, and Bs are reference color information for one
block of the calibration plate 60. R, G, and B are color
information of the block extracted from the extracted image. A is a
correction matrix.
[0083] For example, A is represented by the following Equation
(2):
A = ( a 11 a 13 a 31 a 33 ) Equation ( 2 ) ##EQU00002##
[0084] Here, a.sub.11 to a.sub.33 are correction values as elements
of the correction matrix A.
[0085] The CPU 21 generates the correction matrix A using the
reference color information and the color information of each
block. The CPU 21 generates the correction matrix A by which the
reference color information and the color information of each block
satisfy Equation (1). The CPU 21 calculates the elements of the
correction matrix using the least square method, for example.
[0086] The CPU 21 has a function of storing an image of the
commodity that has been corrected (by using the correction matrix
A) as a dictionary image.
[0087] The CPU 21 specifies a region of the captured image
corresponding to the commodity 50. The CPU 21 may detect an edge of
the commodity 50 from the captured image and thus specify the
region in the captured image corresponding to the commodity 50.
Upon specifying the region of the captured image corresponding to
the commodity 50, the CPU 21 corrects colors in the region of the
captured image corresponding to the commodity region (also referred
to as a commodity image) using the correction matrix.
[0088] For example, the CPU 21 corrects R, G, and B values for each
pixel in the commodity image using the correction matrix values on
each pixel in the commodity image.
[0089] Specifically, the CPU 21 may correct color of the commodity
image according to the following Equation (3):
( R n G n B n ) = A ( R G B ) Equation ( 3 ) ##EQU00003##
[0090] where Rn, Gn, and Bn represent color information of a pixel
after the correction and R, G, and B represent color information of
the pixel before the color correction.
[0091] The CPU 21 assigns a file name to the color-corrected
commodity image and stores this corrected commodity image in the
NVM 23 as a dictionary image.
[0092] The CPU 21 has a function of storing the commodity table
that includes information about a commodity.
[0093] FIG. 4 illustrates a configuration example of the commodity
table.
[0094] As illustrated in FIG. 4, the commodity table stores
"commodity name", "commodity code", and "dictionary image file
name" in a correlated manner.
[0095] The "commodity name" is a text name of a commodity.
[0096] The "commodity code" is ID for specifying the commodity
according to a numerical value, a character string, a symbol, a
combination thereof, or the like.
[0097] The "dictionary image file name" is a file name under which
the dictionary image of the commodity (generated by correcting the
commodity image with the correction matrix) is stored.
[0098] In the example illustrated in FIG. 4, the commodity table
stores "tea", "0000", and "img_0000" as "commodity name",
"commodity code", and "dictionary image file name" in a first
row.
[0099] The CPU 21 acquires a commodity name and a commodity code of
the commodity 50. The CPU 21 may receive a commodity name and a
commodity code from an operator through an operation unit 104 or
the like.
[0100] Upon acquiring a commodity name and a commodity code of the
commodity 50, the CPU 21 adds the commodity name, the commodity
code, and a file name of a dictionary image to the commodity table
in a correlated manner.
The CPU 11 has a function of directing movement of the self-driving
robot 102 to a predetermined stop position and stopping the
self-driving robot 102 at the stop position.
[0101] The CPU 11 acquires the map information from the map
information apparatus 30. The map information indicates an imaging
position corresponding to a commodity shelf in a store. The imaging
position is a position at which the camera 103 mounted on the
self-driving robot 102 can capture an image of the commodity
shelf.
[0102] FIG. 5 illustrates a configuration example of map
information.
[0103] As illustrated in FIG. 5, the map information indicates
imaging positions on passage A (A0 to A7) and imaging positions on
passage B (B0 to B7).
[0104] The passage A is between a commodity shelf column 701 and a
commodity shelf column 702. The passage B is between the commodity
shelf column 702 and a commodity shelf column 703. Each of the
commodity shelf columns 701 to 703 is configured to have a
plurality of commodity shelves.
[0105] The CPU 11 determines a stop position of the self-driving
robot 102 based on a route table.
[0106] The route table indicates a stop position of the
self-driving robot 102. The route table may indicate a plurality of
imaging positions as stop position of the self-driving robot 102.
The route table may indicate that the self-driving robot 102 stops
sequentially at the plurality of imaging positions.
[0107] FIG. 6 illustrates a configuration example of a route
table.
[0108] As illustrated in FIG. 6, the route table stores "route
name" and "stop position" in a correlated manner.
[0109] The "route name" is a name for identifying a particular
route.
[0110] The "stop position" is an imaging position at which the
self-driving robot 102 can stop and then from which an image of the
commodity 50 can be captured. There may be a plurality of "stop
positions". The self-driving robot 102 may be stopped sequentially
at one of several different imaging positions along a route.
[0111] In the example illustrated in FIG. 6, the route table
indicates that the stop positions in the "route 1" are A0 to A7.
The route table indicates that in "route 1" the self-driving robot
102 stops sequentially at A0 to A7.
[0112] The CPU 11 selects a route from the route table. The CPU 11
may select a route based on an operator's instruction received via
the operation unit 104. The CPU 11 may set a route according to a
predetermined condition.
[0113] The CPU 11 causes the self-driving robot 102 to stop at a
stop position according to the selected route. The CPU 11 causes
the self-driving robot 102 to move to and stop at a first stop
position indicated by the route. The CPU 11 causes the self-driving
robot 102 to move to and stop at stop positions in an order
indicated by the route.
[0114] FIGS. 7 and 8 illustrate operation examples in which the CPU
11 causes the self-driving robot 102 to stop at a predetermined
position, for example, A4. FIG. 7 is a diagram of a store having
commodity shelves 701, 702, and 703 installed, viewed from the top.
FIG. 8 is a diagram of the commodity shelves 701 and 702 in the
store, viewed horizontally.
[0115] Here, the CPU 11 selects "route 1" from the route table
along the commodity shelf 701. When the self-driving robot 102 is
at A3, the CPU 11 transmits a signal, to the self-driving robot
102, for instructing to move to and stop at A4.
[0116] The self-driving robot 102 moves to and stops at A4
according to the signal.
[0117] The CPU 11 has a function of adjusting the imaging position
based on an image of a commodity and the calibration plate 60
captured by the camera 103
[0118] After the self-driving robot 102 has stopped at a
predetermined position, the CPU 11 transmits a signal, to the
camera 103, for capturing an image of the commodity shelf 701. The
camera 103 captures an image according to the signal. The camera
103 transmits the captured image to the CPU 11. The CPU 11 acquires
the captured image from the camera 103.
[0119] FIG. 9 illustrates an operation example in which the CPU 11
acquires an image of a commodity and the calibration plate 60.
[0120] As illustrated in FIG. 9, a plurality of commodities is
disposed in a commodity shelf 70. The type or the number of the
commodities disposed in the commodity shelf 70 is not limited to a
specific type or number.
[0121] The calibration plate 60 is installed at a predetermined
position of the commodity shelf 70. In the example illustrated in
FIG. 9, the calibration plate 60 is installed at the top of the
commodity shelf 70. The calibration plate 60 may be installed near
the commodity shelf 70. The location of the calibration plate 60 is
not limited to a specific location, but the calibration plate 60
may be disposed at any predetermined position near a commodity
shelf.
[0122] After the self-driving robot 102 has stopped at a
predetermined position, for example, A4, the CPU 11 acquires an
image captured by the camera 103. Upon detecting that the
self-driving robot has stopped at A4, the CPU 11 transmits a
signal, to the camera 103, for capturing an image of the commodity
shelf. The camera 103 captures an image according to the signal.
Here, the camera 103 captures an image of the commodity shelf 70
and the calibration plate 60. The camera 103 transmits the captured
image to the CPU 11. The CPU 11 acquires the captured image from
the camera 103.
[0123] The CPU 11 extracts an image of the calibration plate 60
from the captured image. For example, the CPU 11 extracts the image
of the calibration plate 60 from the captured image using pattern
detection or the like.
[0124] Upon extracting the image of the calibration plate 60, the
CPU 11 detects a pixel position in the captured image of the
calibration plate 60 from the extracted image. The CPU 11 adjusts
the stop position of the self-driving robot 102 such that the
calibration plate 60, which has been installed at a predetermined
position on a commodity shelf, is at a predetermined pixel position
in the image. For example, the CPU 11 controls the self-driving
robot 102, by instructing the self-driving robot 102 to move
forward, backward, and rotate, so the calibration plate 60 will be
displayed at the predetermined pixel position in the image.
[0125] The CPU 11 has a function of acquiring the image including
the commodity and the calibration plate 60 using the camera 103 or
an image acquisition unit.
[0126] Upon completing the position adjustment of the self-driving
robot 102, the CPU 11 acquires another image using the camera
103.
[0127] The CPU 11 generates a correction matrix based on the
calibration plate 60 in the image captured after the position
adjustment.
[0128] For example, the CPU 11 extracts an image of the calibration
plate 60 from the image captured after repositioning of the
self-driving robot 102 using a pattern recognition algorithm or the
like.
[0129] Upon extracting the image of the calibration plate 60, the
CPU 11 extracts color information for each block on the calibration
plate 60. The CPU 11 detects a position of each block from the
extracted image. The CPU 11 extracts color information of the
position of each block. Here, the color information corresponds to
a color value in RGB format.
[0130] Upon extracting the color information, the CPU 11 generates
the correction matrix based on the reference color information,
included in the reference color table stored in the storage region
13a, and the color information extracted from the captured image of
the calibration plate 60.
[0131] Since the method of generating the correction matrix by the
CPU 11 is the same as the method of generating a correction matrix
by the CPU 21, a detailed description thereof will be omitted.
[0132] The CPU 11 has a function of correcting the captured image
using the correction matrix.
[0133] The CPU 11 may correct color information (R, G, and B
values) of each pixel in the captured image.
[0134] Specifically, the CPU 11 corrects the color of the captured
image according to Equation (3), described above.
[0135] The CPU 11 has a function of recognizing a commodity in the
captured image using the color-corrected captured image of the
commodity shelf.
[0136] The CPU 11 acquires a dictionary image from the dictionary
image storage apparatus 20. The CPU 11 recognizes the commodity
from the corrected captured image by using the dictionary image.
The CPU 11 calculates image feature data from the dictionary image,
and then searches for matching image feature data in the corrected
captured image.
[0137] The CPU 11 searches for an image region with a coincidence
ratio (matching score) exceeds a predetermined threshold value. For
example, the CPU 11 searches for a matching image region by raster
scan or the like. Upon finding a matching image region, the CPU 11
determines the commodity matching the dictionary image is at the
position of the region.
[0138] The CPU 11 then masks or excludes this matching image region
from the corrected captured image and further attempts to find
additional matching regions using image feature data matching. If
no other region of the corrected captured image can be found, the
CPU 11 terminates the detection process for the commodity
corresponding to the dictionary image.
[0139] If there is another dictionary image to evaluate, the CPU 11
performs the same operation with the other dictionary image. The
CPU 11 repeats the operation but this time using the other
dictionary image stored in the dictionary image storage apparatus
20 to recognize the corresponding commodity.
[0140] The CPU 11 may acquire the dictionary images from the
dictionary image storage apparatus 20 one by one or acquire several
dictionary images at once.
[0141] The method of detecting a commodity by the CPU 11 is not
limited to the example described above.
[0142] The CPU 11 may acquire a commodity name and a commodity code
of a detected commodity by referencing the commodity table stored
in the dictionary image storage apparatus 20. The CPU 11 may
transmit the acquired commodity name and commodity code to the
missing commodity recognition apparatus 40.
[0143] Next, an operation example of the CPU 21 of the dictionary
image storage apparatus 20 will be described.
[0144] First, an operation example in which the CPU 21 stores a
dictionary image will be described.
[0145] FIG. 10 is a flowchart of an operation example in which the
CPU 21 stores a dictionary image.
[0146] First, the CPU 21 captures or otherwise acquires an image of
a commodity and the calibration plate 60 using the camera 26 (ACT
11). Upon acquiring the captured image, the CPU 21 generates a
correction matrix based on the captured image (ACT 12).
[0147] After generating the correction matrix, the CPU 21 extracts
an image of the commodity from the captured image (ACT 13). Upon
extracting the image, the CPU 21 corrects the image with the
correction matrix (ACT 14). Upon correcting the image using the
correction matrix, the CPU 21 stores the corrected image of the
commodity as a dictionary image in the NVM 13 (ACT 15).
[0148] Upon storing the dictionary image in the NVM 13, the CPU 21
adds to a commodity list a commodity name, a commodity code, and a
dictionary image file name of the dictionary image in a correlated
manner (ACT 16).
[0149] Upon adding the names and codes to the commodity list, the
CPU 21 terminates the operation.
[0150] If a plurality of captured images is acquired, the CPU 21
repeats ACTs 11 to 16 for each captured image.
[0151] In some embodiments, the CPU 21 may execute ACT 12 after ACT
13.
[0152] Next, an operation example in which the CPU 21 generates the
correction matrix (ACT 12) will be described.
[0153] FIG. 11 is a flowchart of an operation example in which the
CPU 21 generates the correction matrix (ACT 12).
[0154] The CPU 21 extracts a region of the captured image
corresponding to the calibration plate 60 (ACT 22). Upon extracting
the region of the captured image corresponding to the calibration
plate 60, the CPU 21 acquires color information of each block of
the calibration plate 60 from the extracted image (ACT 23).
[0155] Upon acquiring the color information of each block, the CPU
21 acquires reference color information of each block from the
reference color table (ACT 23). Upon acquiring the reference color
information of each block, the CPU 21 generates the correction
matrix based on the color information and reference color
information (ACT 24).
[0156] Upon generates the correction matrix, the CPU 21 stores the
generated correction matrix in the RAM 12 or the NVM 13 (ACT 25).
Upon storing the correction matrix, the CPU 21 terminates the
operation.
[0157] The CPU 21 may execute ACT 22 after ACT 23.
[0158] Next, an example of an operation of the CPU 11 will be
described.
[0159] First, an example in which the CPU 11 recognizes a commodity
will be described.
[0160] FIG. 12 is a flowchart of an example in which the CPU 11
detects the commodity.
[0161] The CPU 11 initializes the image processing apparatus 10
(ACT 31). Upon initializing the image processing apparatus 10, the
CPU 11 selects a route from the route table (ACT 32).
[0162] After selecting the route, the CPU 11 determines whether
commodity imaging has been completed (ACT 33). The CPU 11
determines whether an image has been captured at each stop position
in the selected route.
[0163] Upon determining that imaging has not been completed at a
stop position (ACT 33, NO), the CPU 11 instructs the self-driving
robot 102 to move to and stop at the stop position (ACT 34). Upon
instructing the self-driving robot 102 to move to and stop at the
stop position, the CPU 11 determines whether the self-driving robot
102 has stopped at the stop position (ACT 35).
[0164] Upon determining that the self-driving robot 102 has not
stopped at a stop position (ACT 35, NO), the CPU 11 returns to ACT
35.
[0165] Upon determining that the self-driving robot 102 has stopped
at the stop position (ACT 35, YES), the CPU 11 acquires an image
using the camera 103 and, as necessary, adjusts the position of the
self-driving robot 102 (ACT 36). After adjusting the position of
the self-driving robot 102, the CPU 11 acquires an image using the
camera 103 (ACT 37). The CPU 11 then generates a correction matrix
based on the captured image (ACT 38).
[0166] The CPU 11 then corrects the captured image using the
correction matrix (ACT 39). CPU 11 then detects the commodity in
the corrected captured image (ACT 40).
[0167] After detecting the commodity, the CPU 11 returns to ACT
33.
[0168] Once the imaging has been completed (ACT 33, YES), the CPU
11 causes the self-driving robot 102 to move to an initial position
(ACT 41). Upon causing the self-driving robot 102 to move to the
initial position, the CPU 11 terminates the operation.
[0169] Since an operation example in which the CPU 11 generates a
correction matrix (ACT 38) is the same as in FIG. 11, repeated
description will be omitted.
[0170] FIG. 13 is a flowchart of an operation example in which the
CPU 11 recognizes or detects a commodity.
[0171] The CPU 11 selects a dictionary image (ACT 51). The CPU 11
acquires this dictionary image from the dictionary image storage
apparatus 20. Upon selecting the dictionary image, the CPU 11
calculates image feature data for the dictionary image (ACT 52).
The CPU 11 then searches for a region of the corrected captured
image having image features that coincides with image feature of a
region of the dictionary image at a ratio (matching score)
exceeding a predetermined threshold value (ACT 53).
[0172] Upon finding a region where a coincidence ratio of the
feature data exceeds a predetermined threshold value (ACT 54, YES),
the CPU 11 determines the commodity registered in the dictionary
image is at a position corresponding to the region of the corrected
captured image (ACT 55). Upon detecting the commodity depicted in
the dictionary image at the position corresponding to the region of
the corrected captured image, the CPU 11 masks the region of the
corrected captured image that has been found in ACT 55 (ACT 56).
Upon masking the region of the corrected captured image, the CPU 11
returns to ACT 53.
[0173] Upon finding no region of the corrected captured image
having image features that coincide with image features of the
dictionary image at a coincidence ratio exceeding the predetermined
threshold value (ACT 54, NO), the CPU 11 next determines whether
there is another dictionary image to evaluate for matching in the
corrected captured image (ACT 57). Upon determining that there is
another dictionary image (ACT 57, YES), the CPU 11 selects another
dictionary image (ACT 58). Upon selecting the other dictionary
image, the CPU 11 returns to ACT 52.
[0174] Upon determining that there is no other dictionary image to
evaluate for matching (ACT 57, NO), the CPU 11 terminates the
operation.
[0175] In some embodiments, the self-driving robot 102 may
incorporate the commodity recognition unit 101.
[0176] In some embodiments, the image processing apparatus 10 need
not include the self-driving robot 102. For example, the CPU 11 may
detect a commodity from an image captured by the camera 103
according to an operator's instruction or manipulation.
[0177] The image processing apparatus 10 and the dictionary image
storage apparatus 20 may be integrated.
[0178] In the above-described embodiments, the image processing
system generates a correction matrix that alleviates imaging
variations that might be caused by the influence of camera
characteristics, external lighting, or the like using an image of a
calibration plate in the captured image. The image processing
system then corrects a captured image using the correction matrix.
The image processing system detects a commodity in the corrected
image. Accordingly, the image processing system detects the
commodity more reliably from the captured image. As a result, the
image processing system can prevent the detection accuracy of the
commodity from deteriorating due to the influence of camera
characteristics, external light or the like. Thus, the image
processing system can more effectively recognize the commodity.
[0179] Also, as the camera captures an image of a commodity shelf
and a calibration plate, which can have a known physical location,
a self-driving robot can adjust its position based on its relative
positioning with respect to the calibration plate.
[0180] While certain embodiments have been described, these
embodiments have been presented by way of example only, and are not
intended to limit the scope of the inventions. Indeed, the novel
embodiments described herein may be embodied in a variety of other
forms; furthermore, various omissions, substitutions and changes in
the form of the embodiments described herein may be made without
departing from the spirit of the inventions. The accompanying
claims and their equivalents are intended to cover such forms or
modifications as would fall within the scope and spirit of the
inventions.
* * * * *