U.S. patent application number 15/736111 was filed with the patent office on 2018-06-21 for method for operating a household appliance, corresponding household appliance and system with a household appliance.
The applicant listed for this patent is BSH HAUSGERAETE GMBH. Invention is credited to MARTIN GEBLER, JOCHEN GRIMMINGER, ASTRID KLINGSHIRN.
Application Number | 20180172343 15/736111 |
Document ID | / |
Family ID | 56098230 |
Filed Date | 2018-06-21 |
United States Patent
Application |
20180172343 |
Kind Code |
A1 |
GRIMMINGER; JOCHEN ; et
al. |
June 21, 2018 |
METHOD FOR OPERATING A HOUSEHOLD APPLIANCE, CORRESPONDING HOUSEHOLD
APPLIANCE AND SYSTEM WITH A HOUSEHOLD APPLIANCE
Abstract
A method includes taking a picture of the interior of a
household appliance by using a camera, identifying at least one
object in the picture, and editing the object. A household
appliance having a processing unit for performing the method and a
system having a household appliance and at least one further
component are also provided.
Inventors: |
GRIMMINGER; JOCHEN;
(DILLINGEN, DE) ; KLINGSHIRN; ASTRID;
(ZUSMARSHAUSEN, DE) ; GEBLER; MARTIN; (PULLACH,
DE) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
BSH HAUSGERAETE GMBH |
MUENCHEN |
|
DE |
|
|
Family ID: |
56098230 |
Appl. No.: |
15/736111 |
Filed: |
May 25, 2016 |
PCT Filed: |
May 25, 2016 |
PCT NO: |
PCT/EP2016/061802 |
371 Date: |
December 13, 2017 |
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
F25D 29/00 20130101;
F25D 2700/06 20130101; F25D 2500/06 20130101 |
International
Class: |
F25D 29/00 20060101
F25D029/00 |
Foreign Application Data
Date |
Code |
Application Number |
Jun 15, 2015 |
DE |
102015210899.6 |
Claims
1-10. (canceled)
11. A method, comprising the following steps: taking a picture of
an interior of a household appliance by using a camera; identifying
at least one object in the picture; and editing the object.
12. The method according to claim 11, which further comprises
carrying out the step of editing the object by creating, modifying
or deleting information associated with the object.
13. The method according to claim 12, wherein the information
includes at least one of the following: a content of the object; an
intended use of the object; an owner of the object; an instruction
relating to the object; a condition associated with the object; or
a predefined action to be executed if a condition is met.
14. The method according to claim 11, wherein the household
appliance is one of the following appliances: a kitchen appliance,
a cooking appliance, or a refrigerator/freezer appliance.
15. The method according to claim 11, which further comprises
carrying out the step of taking the picture by using the camera
when a door of the household appliance is open or closed.
16. A household appliance, comprising a processing unit for
performing the following steps: taking a picture of an interior of
a household appliance by using a camera; identifying at least one
object in the picture; and editing the object.
17. A system, comprising: a household appliance having a camera for
taking a picture of an interior of the household appliance; and at
least one further component having a processing unit for:
identifying at least one object in the picture and editing the
object.
18. The system according to claim 17, wherein said at least one
further component includes a memory configured to be accessed by a
network interface and configured to store the picture by the
camera.
19. The system according to claim 17, wherein said at least one
further component has a display device with an input interface and
a screen, said input interface is configured to edit the object,
and said screen is configured to show the object and
information.
20. The system according to claim 19, wherein said display device
is a smartphone or a computer.
Description
[0001] The invention relates to a method for operating a household
appliance, a corresponding household appliance and a system with a
household appliance.
[0002] It is known that a camera can be used in a household
appliance, e.g. a refrigerator, to record the interior of the
refrigerator or part thereof at specific times. The camera can take
a picture of the interior of the refrigerator, e.g. the food or
other items currently located in the refrigerator, e.g. when the
refrigerator door is opened or closed. Correspondingly, such a
picture can also be taken when the refrigerator is in a closed
state.
[0003] A sensor can be provided for this purpose, for example, and
detect any opening and/or closing of the refrigerator door. This
can be achieved e.g. by means of a magnetic sensor or other type of
switch. The picture taken by the camera is saved in a memory, e.g.
via a wireless or hard-wired connection. The memory can be a memory
of the household appliance or a memory of a computer or computer
network, for example. In particular, the picture can be stored via
a network interface in a central database which is attached via the
Internet, for example.
[0004] If at least one picture of the content of the household
appliance is taken each time it is opened and/or closed, it is then
possible on the basis of the changes between different pictures
produced during the course of using the household appliance to
infer an insertion of goods into the household appliance and a
corresponding removal of goods. This information can be further
processed for various purposes.
[0005] It is problematic in this context that a multiplicity of
reusable containers are present, appearing on the picture but
giving no indication of their content. This applies to
general-purpose resealable containers, for example.
[0006] The object of the invention is to improve an existing
household appliance such that efficient use is also possible with
different containers.
[0007] This object is achieved in accordance with the features in
the independent claims. Developments of the invention are derived
from the dependent claims.
[0008] A method comprising the following steps is specified for the
purpose of achieving said object: [0009] taking a picture of the
interior of a household appliance by means of a camera, [0010]
identifying at least one object in the picture, [0011] editing the
object.
[0012] The method can be executed e.g. entirely or partly in the
household appliance. In particular, the identification of the
object and/or the editing the object can be performed in or at the
household appliance. As further option, these steps can also be
performed separately from the household appliance in at least one
external component which is connected to the household appliance
e.g. via a communication interface. For example, the identification
of the object can be performed in a central computer, which like
the household appliance is connected to the Internet. The editing
of the object can be performed e.g. on or with the aid of the
central computer. As a further option, the editing can also be
performed on another device, e.g. a mobile terminal (e.g.
smartphone). According to a further option, the identification can
be effected at least partially or even entirely by means of a
program which is executed on the terminal with or without user
input.
[0013] The picture can be taken by a camera of the household
appliance. The picture can be stored in a memory of the household
appliance, in an external memory and/or in a network memory. In
principle, any of the application cases cited here by way of
example can be combined as desired. The picture could therefore be
transferred via a WLAN interface from the household appliance via
the Internet directly to a smartphone, which can be connected to
the Internet either via a WLAN interface or other wireless
communication connection (e.g. 3G or 4G). The identification of the
object and the editing of the object could be performed on the
mobile terminal in this example. The mobile terminal can have a
special program for this purpose (a so-called "app").
[0014] According to a development, the editing of the object
comprises: creating, modifying or deleting information which is
associated with the object.
[0015] The household appliance can have a display device
(comprising a screen and an input interface) for the purpose of
editing. It is also possible to use an external display device
which can be linked to the household appliance. As a further
option, the external display device can be connected to a memory in
which the pictures are stored.
[0016] According to a further development, the information
comprises at least one of the following: [0017] a content of the
object; [0018] an intended use of the object; [0019] an owner of
the object; [0020] an instruction relating to the object; [0021] a
condition associated with the object; [0022] a predefined action to
be executed if a condition is met.
[0023] The predefined action can be very diverse. For example,
communication can be automatically established with other services.
In particular, it is possible to detect that at least one item
(e.g. milk) is missing from the refrigerator and reordering of the
item can automatically be triggered. Accordingly, the item (where
present) can optionally occupy a specific space in the refrigerator
or be arranged in a specific region of the refrigerator. It is then
possible automatically to determine when the item is missing or has
been consumed. For example, a timer is started when the item is
removed from the refrigerator. If the item is not replaced within a
predefined time (e.g. 1 hour), it is assumed that the item has been
consumed and the reordering can be triggered. A multiplicity of
options are available for identifying items in the refrigerator
accordingly, and possible options which can be associated with the
identification or non-identification.
[0024] In particular, at least one condition can be associated as
information. This condition can be checked automatically on a
regular basis, e.g. every day, or as a function of specific
parameters, e.g. a temperature, or other objects. If the condition
is met, a predefined action is initiated. One example is the
display of goods whose use-by date has expired and/or will soon
expire. A further example is the output of a warning if the
temperature in the household appliance reaches a critical threshold
for this product.
[0025] According to a development, the household appliance is one
of the following appliances in particular: [0026] a kitchen
appliance, [0027] a cooking appliance, [0028] a fridge/freezer
appliance.
[0029] The fridge/freezer appliance is also referred to as a
refrigerator here.
[0030] According to a further development, the camera is so
arranged that the picture is taken when a door of the household
appliance is opened and/or closed.
[0031] The object is further achieved by a proposed household
appliance comprising a processing unit for performing the method
described here.
[0032] The object is further achieved by a system which comprises a
household appliance and at least one further component, [0033]
wherein the household appliance has a camera for taking a picture
of the interior of a household appliance, [0034] wherein the at
least one further component has a processing unit for [0035]
identifying at least one object in the picture, [0036] editing the
object.
[0037] The processing unit cited here can be embodied in particular
as a processor unit and/or a circuit arrangement that is at least
partly hard-wired or a logical circuit arrangement, which is so
configured that the method can be performed as described here, for
example. Said processing unit may be or comprise any type of
processor or computer or computer with peripherals (memory,
input/output interfaces, input/output devices, etc.) that are
correspondingly required.
[0038] The present explanations relating to the method apply
respectively to the device and the system correspondingly.
[0039] According to a development, the at least one further
component comprises a memory which can be accessed via a network
interface and in which the picture can be stored by the camera.
[0040] According to a further development, the at least one further
component has a display device with an input interface and a
screen, wherein the object can be edited using the input interface
and wherein the object and information can be represented by means
of the screen.
[0041] According to a further development, the display device is a
smartphone or a computer.
[0042] Exemplary embodiments of the invention are illustrated and
explained below with reference to the drawing, in which:
[0043] FIG. 1 shows an exemplary schematic block diagram comprising
a refrigerator with a door, a camera and a processing unit in a
symbolic side view.
[0044] The proposal described here allows a user to identify what
is contained in any container (e.g. a general-purpose resealable
box, packaging, etc.) which is situated in a household appliance.
To this end, the container is recorded by a camera, the picture
undergoes object identification and the container is identified as
the object of the picture. The camera can be arranged in the
household appliance, for example. The picture can be taken when the
door of the household appliance is opened or closed, for
example.
[0045] The identified object is edited, i.e. annotated or
associated with information, for example. The information may
comprise a marking, a text, a symbol, a color, etc. The information
can be associated with a content of the container for the user. It
is therefore possible for the user to identify the content of the
container on the basis of the information.
[0046] The object is edited, e.g. by coloring the identified object
and/or annotating it with a text. For example, the picture can be
supplemented by the information and the information stored in
association with the picture. When looking at the picture, the user
also sees the information and can immediately identify the content
of the container. It is optionally possible for the user to make
the information visible by means of an action, or for the
information to be represented as a result of the action. The
clarity of the representation can be improved if not all
information is represented at all times.
[0047] It is advantageous in this context that the information can
be made available to every user who uses the household appliance.
For example, the information can be represented on a display
device. The display device can be part of the household appliance
or part of a computer, e.g. a smartphone or any (mobile or
stationary) computer. The picture with or without information can
be forwarded e.g. from the household appliance via an interface and
stored in a network. The information can be provided directly at
the household appliance, or it can be supplemented or modified
after the picture (with or without information) has been stored in
the household appliance, in an associated memory and/or in the
network, for example.
[0048] The information can comprise the content of the container,
an intended purpose (e.g. in the context of a recipe), a possible
use, an owner, a condition (e.g. use-by date), or similar.
[0049] The information can be associated with at least one
predefined action: if the user selects a use-by date as
information, a predefined action may be initiated before, upon or
after reaching this use-by date: for example, a message may be
displayed (e.g. on the display device) two days before reaching the
use-by date to the effect that the goods in the container should be
consumed (e.g. the container and the goods can be displayed in this
case with the imminent use-by date or the time until expiry). It is
also possible in this way to display a countdown for goods due to
expire in a forthcoming time period, e.g. the next ten days. It is
thereby possible effectively to ensure that the user can consume
the goods in a timely manner before reaching the use-by date,
and/or that the refrigerator does not contain any goods in respect
of which the user is not aware that the use-by date has
expired.
[0050] In a further example, the temperature of the refrigerator
can be adapted as a function of the information: if information
specified by the user relates to specific goods for which a
preferred temperature (or a temperature range) is stored in a table
of the processing unit, the refrigerator can adapt the temperature
accordingly for a region in which the container is located, for
example.
[0051] It is advantageous that displacement of the object in the
refrigerator can also be detected and therefore in a virtual sense
the information moves with the object in the refrigerator. This
means that at all times the most recent content of the
refrigerator, i.e. the identified objects, can be represented on
the display device with the information. The information about the
object can optionally be deleted or updated. This allows the
identified object to be used in a wide variety of ways as a
container for different goods.
[0052] The information can be associated in different ways with the
objects identified in the picture. For example, the information can
be represented in a manner which is transparent, magnified,
reduced, shifted, pointing to the object with arrows or lines, etc.
It is also possible in a virtual manner to represent the content of
the refrigerator on the display device, by virtually moving the
identified objects on the screen. The information can be recreated
and represented in conjunction with the virtual operations. It is
also possible to represent a (possibly sorted) list which is
independent of the actual position of the object in the
refrigerator, and shows the identified objects with the information
in the form of a table or otherwise sorted, for example. The
information can be used as a sorting criterion for such a
representation.
[0053] It should also be noted that the information can be edited
in different representations on the display device. For example,
the information can be supplemented, entered for the first time or
deleted. The information can also be associated with conditions, or
previously entered conditions can be deleted. Furthermore, the
information can be represented to all users or a selection of
users. For example, it is possible to display the content
"Biscuits" of a container to specific users, and instead of the
content to display different information to other users, e.g.
"Owner: Karl, please do not eat!".
[0054] The information can also comprise suggestions in respect of
storage (place of storage, duration of storage, temperature of
storage, etc.) of an item in the household appliance, in particular
for the purpose of changing the storage.
[0055] The display device can be part of the household appliance.
The display device can also be a conventional computer (e.g.
laptop, personal computer, etc.). Furthermore, a smartphone can be
used as a display device. For example, an application (also known
as an application program or app) can be installed on the
smartphone, so that the smartphone can be used as a display device
for the household appliance.
[0056] The information can be edited via the display device. The
content of the household appliance (current content according to
most recent picture, possibly with reference to previous pictures)
can also be represented virtually on the display device.
Furthermore, it is possible to perform analyses (e.g. in respect of
the use-by date or in respect of possible recipes) using the
display device.
[0057] Accordingly, the display device comprises e.g. a processor
unit which allows the information to be edited and/or processed by
means of a program and with reference to the picture.
[0058] In this context, part of the processing such as e.g.
identification of objects in the pictures of the camera can already
be performed by the household appliance, or pictures can be
transmitted from the household appliance pictures to an external
processing unit, e.g. in a network. Processing steps e.g. for
identifying objects in the pictures can be performed there, either
exclusively or additionally. The results of the processing can be
provided to the different display devices, e.g. via a wireless or
hard-wired interface.
[0059] As an option, the household appliance can have a connection
to the Internet and load the pictures into a memory which is
connected to the Internet. The display devices can also access the
memory via the Internet, and thereby access the pictures stored
there, together with the stored information if applicable.
[0060] The information can be stored centrally, it can be stored
together with or separately from the pictures, it can also be
stored in a memory which is directly connected to or arranged
within the household appliance. Furthermore, the information can be
stored in the display device. Distribution of the information from
one user to at least one other user is possible in all of the cited
cases, and optionally with restrictions relating to the access
(e.g. on the form of user permissions).
[0061] By way of example, FIG. 1 shows a schematic block diagram
comprising a refrigerator 101 with a door 102 in a symbolic side
view. The door 102 is closed in FIG. 1. A sensor 106 detects
whether the door 102 is open or closed. When the door 102 is opened
or closed, the sensor 106 transmits a corresponding signal to a
processing unit 104. The processing unit 104 then activates a
camera 103, whereby this takes a picture of the interior of the
refrigerator 101. By way of example, the processing unit 104 stores
the picture in a memory 107. The memory 107 can be embodied as part
of the processing unit 104 or separate from this.
[0062] A display device 108 can be embodied externally relative to
the refrigerator 101. The display device 108 can also make use of
the processing unit 104, or this can be part of the display device
108. The display device 108 can be arranged e.g. externally on the
door 102 of the refrigerator 101, and therefore operated by the
user when the door 102 is closed (or open). For example, the
display device 108 has an input interface, e.g. a keypad, a (e.g.
touch-sensitive) screen, a pointer device, at least one control
button or other navigation tool. The picture with or without
(editable) information can therefore be represented for the user
via a screen of the display device 108. In particular, the user can
edit the information and, for example, assign information to an
identified object in the picture in this way. This means that
virtual information can be superimposed on the picture on the
screen of the display device, said information supplementing the
identified object, for example.
[0063] The processing unit 104 can be configured, for example, to
identify at least one object 105 in the picture taken by the
camera. The object 105 in the example of FIG. 1 is a resealable
general-purpose opaque box. This box is identified in the picture
and can be edited by the user as an object. Such editing comprises
the assignment of information to the object 105, for example. The
user can therefore associate the information "Biscuits" with the
object 105 by means of the display device 108. This information
"Biscuits" can then be displayed with the picture on the screen of
the display device 108.
[0064] It is also possible, for example, to embody the camera 103
and the processing unit 104 as one component. In particular, such a
component may have a wireless network connection, for example,
whereby the pictures that have been taken are stored directly in a
memory which is attached via the Internet. Such a memory is also
referred to as "cloud storage".
[0065] By way of example, FIG. 1 also shows an interface between
the processing unit 104 and a smartphone 123. It should be noted
that the broken lines in FIG. 1 symbolically represent a transfer
of signals or data. The transfer of data to the smartphone 123 can
be effected e.g. via a radio interface which at least partially
uses the Internet. To this end, the processing unit 104 itself can
have a wireless or hard-wired connection to the Internet.
[0066] The smartphone 123 represents a further example of a display
device. It comprises input means in the form of a (e.g. virtual)
keypad 123 and a screen 121. The identified object of the box 105
can be visualized on the screen of the smartphone, e.g. as an
optional image 122 of the box 105. The user has the means to enter
information for this image 122. In the example of FIG. 1, the
character string "Biscuits" is entered as information 124 and
associated with the image 122.
[0067] This means that each user or an authorized selection of
users can be given the option, when looking at a picture of the
interior of the refrigerator 101, also to find out what is
contained in the object 105. For this purpose, the information
"Biscuits" is also shown with the picture on the display device 108
or 123.
LIST OF REFERENCE SIGNS
[0068] 101 Refrigerator [0069] 102 Door [0070] 103 Camera [0071]
104 Processing unit [0072] 105 Object [0073] 106 Sensor [0074] 107
Memory [0075] 108 Display device [0076] 121 Screen [0077] 122 Image
[0078] 123 Smartphone [0079] 124 Information
* * * * *