U.S. patent application number 16/807269 was filed with the patent office on 2021-09-09 for system and method for using sound to monitor the operation of a washing machine appliance.
The applicant listed for this patent is Haier US Appliance Solutions, Inc.. Invention is credited to David Scott Dunn, Juan Manuel Huerta, Khalid Jamal Mashal.
Application Number | 20210277564 16/807269 |
Document ID | / |
Family ID | 1000004700859 |
Filed Date | 2021-09-09 |
United States Patent
Application |
20210277564 |
Kind Code |
A1 |
Mashal; Khalid Jamal ; et
al. |
September 9, 2021 |
SYSTEM AND METHOD FOR USING SOUND TO MONITOR THE OPERATION OF A
WASHING MACHINE APPLIANCE
Abstract
A washing machine appliance includes a microphone for monitoring
sound generated during operation of the washing machine appliance
and a controller is operably coupled to the microphone. The
controller is configured for obtaining a sound signal generated
during operation of the washing machine appliance and converting
the sound signal into a spectrogram that represents a sound
frequency and a sound amplitude over time. An artificial
intelligence image recognition process is used to analyze the
spectrogram to identify one or more sound signatures that are
associated with particular operating conditions, and operation of
the washing machine appliance is adjusted based at least in part on
the identification of the sound signature.
Inventors: |
Mashal; Khalid Jamal;
(Louisville, KY) ; Dunn; David Scott; (Louisville,
KY) ; Huerta; Juan Manuel; (Louisville, KY) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Haier US Appliance Solutions, Inc. |
Wilmington |
DE |
US |
|
|
Family ID: |
1000004700859 |
Appl. No.: |
16/807269 |
Filed: |
March 3, 2020 |
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
D06F 33/34 20200201;
D06F 2103/18 20200201; D06F 33/44 20200201; D06F 33/40 20200201;
D06F 34/14 20200201; D06F 2103/24 20200201; D06F 2103/26
20200201 |
International
Class: |
D06F 34/14 20060101
D06F034/14; D06F 33/34 20060101 D06F033/34; D06F 33/40 20060101
D06F033/40; D06F 33/44 20060101 D06F033/44 |
Claims
1. A washing machine appliance comprising: a wash tub positioned
within a cabinet and defining a wash chamber; a wash basket
rotatably mounted within the wash tub and being configured for
receiving of a load of articles for washing; a motor operably
coupled to the wash basket for selectively rotating the wash
basket; a microphone for monitoring sound generated during
operation of the washing machine appliance; and a controller
operably coupled to the microphone, the controller being configured
for: obtaining a sound signal generated during operation of the
washing machine appliance using the microphone; generating a
spectrogram from the sound signal, the spectrogram representing a
sound frequency and a sound amplitude over time; identifying a
sound signature by analyzing the spectrogram using an image
recognition process; and adjusting at least one operating parameter
of the washing machine appliance based at least in part on the
identification of the sound signature.
2. The washing machine appliance of claim 1, wherein the image
recognition process uses artificial intelligence (AI) to analyze
the spectrogram.
3. The washing machine appliance of claim 1, wherein the image
recognition process comprises a convolution neural network
(CNN).
4. The washing machine appliance of claim 1, wherein the controller
is further configured for: learning a plurality of sound signatures
associated with various operating conditions.
5. The washing machine appliance of claim 1, wherein the sound
signature is associated with sounds generated from at least one of
a bearing, a belt, the motor, a water valve, a pump, a suspension
system, harmonics of structural components, or undesirable contact
between components or subsystems.
6. The washing machine appliance of claim 1, wherein adjusting the
at least one operating parameter comprises: adjusting an agitation
time or profile, adjusting a water level, limiting a spin speed,
identifying service needs, or providing a user with operating
guidance.
7. The washing machine appliance of claim 1, wherein adjusting the
at least one operating parameter comprises: selecting an operating
cycle based on the sound signature.
8. The washing machine appliance of claim 1, wherein the controller
is further configured for: providing a user notification when the
sound signature indicates that a predetermined operating
characteristic exists.
9. The washing machine appliance of claim 1, wherein the sound
signature is associated with the presence of an unwashable item,
and wherein adjusting the at least one operating parameter
comprises stopping the wash cycle.
10. The washing machine appliance of claim 1, wherein the
controller is further configured for: transmitting the spectrogram
to a remote server for analysis; and receiving analytic feedback
from the remote server.
11. The washing machine appliance of claim 1, wherein the
microphone is positioned outside the cabinet and remote from the
washing machine appliance.
12. A method of operating a washing machine appliance, the washing
machine appliance comprising a wash basket rotatably mounted within
a wash tub, a motor operably coupled to the wash basket for
selectively rotating the wash basket, and a microphone for
monitoring sound generated by the washing machine appliance, the
method comprising: obtaining a sound signal generated during
operation of the washing machine appliance using the microphone;
generating a spectrogram from the sound signal, the spectrogram
representing a sound frequency and a sound amplitude over time;
identifying a sound signature by analyzing the spectrogram using an
image recognition process; and adjusting at least one operating
parameter of the washing machine appliance based at least in part
on the identification of the sound signature.
13. The method of claim 12, wherein the image recognition process
uses artificial intelligence (AI) to analyze the spectrogram.
14. The method of claim 12, wherein the image recognition process
comprises a convolution neural network (CNN).
15. The method of claim 12, further comprising: learning a
plurality of sound signatures associated with various operating
conditions.
16. The method of claim 12, wherein the sound signature is
associated with the presence of an unwashable item, and wherein
adjusting the at least one operating parameter comprises stopping
the wash cycle.
17. The method of claim 12, further comprising: transmitting the
spectrogram to a remote server for analysis; and receiving analytic
feedback from the remote server.
18. An appliance comprising: a microphone for monitoring sound
generated during operation of the appliance; and a controller
operably coupled to the microphone, the controller being configured
for: obtaining a sound signal generated during operation of the
appliance using the microphone; generating a spectrogram from the
sound signal, the spectrogram representing a sound frequency and a
sound amplitude over time; identifying a sound signature by
analyzing the spectrogram using an image processing technique; and
adjusting at least one operating parameter of the appliance based
at least in part on the identification of the sound signature.
19. The appliance of claim 18, wherein the image recognition
process uses artificial intelligence (AI) to analyze the
spectrogram.
20. The appliance of claim 18, wherein the image recognition
process comprises a convolution neural network (CNN).
Description
FIELD OF THE INVENTION
[0001] The present subject matter relates generally to washing
machine appliances, or more specifically, to systems and methods
for monitoring sounds within a washing machine appliance and
analyzing those sounds to identify sound signatures associated with
particular events.
BACKGROUND OF THE INVENTION
[0002] Washing machine appliances generally include a tub for
containing water or wash fluid, e.g., water and detergent, bleach,
and/or other wash additives. A basket is rotatably mounted within
the tub and defines a wash chamber for receipt of articles for
washing. During normal operation of such washing machine
appliances, the wash fluid is directed into the tub and onto
articles within the wash chamber of the basket. The basket or an
agitation element can rotate at various speeds to agitate articles
within the wash chamber, to wring wash fluid from articles within
the wash chamber, etc. During a spin or drain cycle, a drain pump
assembly may operate to discharge water from within sump.
[0003] Notably, it is frequently desirable to monitor sounds
generated by a washing machine appliance during operation, e.g., to
identify unintended objects in a wash load, to diagnose mechanical
failures, or to detect other operating conditions. However,
conventional washing machines lack any sound feedback systems.
Certain washing machine may monitor sounds and provide a
notification when a sound exceeds a certain threshold, but such
systems have limited usefulness and effectiveness.
[0004] Accordingly, a washing machine appliance with features for
improved operation would be desirable. More specifically, a system
and method for monitoring sounds generated by a washing machine
appliance and identifying sound signatures associated with
particular operating conditions would be particularly
beneficial.
BRIEF DESCRIPTION OF THE INVENTION
[0005] Advantages of the invention will be set forth in part in the
following description, or may be apparent from the description, or
may be learned through practice of the invention.
[0006] In accordance with one exemplary embodiment of the present
disclosure, a washing machine appliance is provided including a
wash tub positioned within a cabinet and defining a wash chamber, a
wash basket rotatably mounted within the wash tub and being
configured for receiving of a load of articles for washing, and a
motor operably coupled to the wash basket for selectively rotating
the wash basket. A microphone is provided for monitoring sound
generated during operation of the washing machine appliance and a
controller is operably coupled to the microphone. The controller is
configured for obtaining a sound signal generated during operation
of the washing machine appliance using the microphone, generating a
spectrogram from the sound signal, the spectrogram representing a
sound frequency and a sound amplitude over time, identifying a
sound signature by analyzing the spectrogram using an image
recognition process, and adjusting at least one operating parameter
of the washing machine appliance based at least in part on the
identification of the sound signature.
[0007] In accordance with another exemplary embodiment of the
present disclosure, a method of operating a washing machine
appliance is provided. The washing machine appliance includes a
wash basket rotatably mounted within a wash tub, a motor operably
coupled to the wash basket for selectively rotating the wash
basket, and a microphone for monitoring sound generated by the
washing machine appliance. The method includes obtaining a sound
signal generated during operation of the washing machine appliance
using the microphone, generating a spectrogram from the sound
signal, the spectrogram representing a sound frequency and a sound
amplitude over time, identifying a sound signature by analyzing the
spectrogram using an image recognition process, and adjusting at
least one operating parameter of the washing machine appliance
based at least in part on the identification of the sound
signature.
[0008] In accordance with another exemplary embodiment of the
present disclosure, an appliance is provided including a microphone
for monitoring sound generated during operation of the appliance
and a controller operably coupled to the microphone. The controller
is configured for obtaining a sound signal generated during
operation of the appliance using the microphone, generating a
spectrogram from the sound signal, the spectrogram representing a
sound frequency and a sound amplitude over time, identifying a
sound signature by analyzing the spectrogram using an image
processing technique, and adjusting at least one operating
parameter of the appliance based at least in part on the
identification of the sound signature.
[0009] These and other features, aspects and advantages of the
present invention will become better understood with reference to
the following description and appended claims. The accompanying
drawings, which are incorporated in and constitute a part of this
specification, illustrate embodiments of the invention and,
together with the description, serve to explain the principles of
the invention.
BRIEF DESCRIPTION OF THE DRAWINGS
[0010] A full and enabling disclosure of the present invention,
including the best mode thereof, directed to one of ordinary skill
in the art, is set forth in the specification, which makes
reference to the appended figures.
[0011] FIG. 1 provides a perspective view of an exemplary washing
machine appliance according to an exemplary embodiment of the
present subject matter.
[0012] FIG. 2 provides a side cross-sectional view of the exemplary
washing machine appliance of FIG. 1.
[0013] FIG. 3 illustrates a method for using sounds generated by a
washing machine appliance to identify operating conditions in
accordance with one embodiment of the present disclosure.
[0014] FIG. 4 provides an exemplary spectrogram according to an
exemplary embodiment of the present subject matter.
[0015] Repeat use of reference characters in the present
specification and drawings is intended to represent the same or
analogous features or elements of the present invention.
DETAILED DESCRIPTION
[0016] Reference now will be made in detail to embodiments of the
invention, one or more examples of which are illustrated in the
drawings. Each example is provided by way of explanation of the
invention, not limitation of the invention. In fact, it will be
apparent to those skilled in the art that various modifications and
variations can be made in the present invention without departing
from the scope or spirit of the invention. For instance, features
illustrated or described as part of one embodiment can be used with
another embodiment to yield a still further embodiment. Thus, it is
intended that the present invention covers such modifications and
variations as come within the scope of the appended claims and
their equivalents.
[0017] Referring now to the figures, FIG. 1 is a perspective view
of an exemplary horizontal axis washing machine appliance 100 and
FIG. 2 is a side cross-sectional view of washing machine appliance
100. As illustrated, washing machine appliance 100 generally
defines a vertical direction V, a lateral direction L, and a
transverse direction T, each of which is mutually perpendicular,
such that an orthogonal coordinate system is generally defined.
Washing machine appliance 100 includes a cabinet 102 that extends
between a top 104 and a bottom 106 along the vertical direction V,
between a left side 108 and a right side 110 along the lateral
direction, and between a front 112 and a rear 114 along the
transverse direction T.
[0018] Referring to FIG. 2, a wash basket 120 is rotatably mounted
within cabinet 102 such that it is rotatable about an axis of
rotation A. A motor 122, e.g., such as a pancake motor, is in
mechanical communication with wash basket 120 to selectively rotate
wash basket 120 (e.g., during an agitation or a rinse cycle of
washing machine appliance 100). Wash basket 120 is received within
a wash tub 124 and defines a wash chamber 126 that is configured
for receipt of articles for washing. The wash tub 124 holds wash
and rinse fluids for agitation in wash basket 120 within wash tub
124. As used herein, "wash fluid" may refer to water, detergent,
fabric softener, bleach, or any other suitable wash additive or
combination thereof. Indeed, for simplicity of discussion, these
terms may all be used interchangeably herein without limiting the
present subject matter to any particular "wash fluid."
[0019] Wash basket 120 may define one or more agitator features
that extend into wash chamber 126 to assist in agitation and
cleaning articles disposed within wash chamber 126 during operation
of washing machine appliance 100. For example, as illustrated in
FIG. 2, a plurality of ribs 128 extends from basket 120 into wash
chamber 126. In this manner, for example, ribs 128 may lift
articles disposed in wash basket 120 during rotation of wash basket
120.
[0020] Referring generally to FIGS. 1 and 2, cabinet 102 also
includes a front panel 130 which defines an opening 132 that
permits user access to wash basket 120 of wash tub 124. More
specifically, washing machine appliance 100 includes a door 134
that is positioned over opening 132 and is rotatably mounted to
front panel 130. In this manner, door 134 permits selective access
to opening 132 by being movable between an open position (not
shown) facilitating access to a wash tub 124 and a closed position
(FIG. 1) prohibiting access to wash tub 124.
[0021] A window 136 in door 134 permits viewing of wash basket 120
when door 134 is in the closed position, e.g., during operation of
washing machine appliance 100. Door 134 also includes a handle (not
shown) that, e.g., a user may pull when opening and closing door
134. Further, although door 134 is illustrated as mounted to front
panel 130, it should be appreciated that door 134 may be mounted to
another side of cabinet 102 or any other suitable support according
to alternative embodiments.
[0022] Referring again to FIG. 2, wash basket 120 also defines a
plurality of perforations 140 in order to facilitate fluid
communication between an interior of basket 120 and wash tub 124. A
sump 142 is defined by wash tub 124 at a bottom of wash tub 124
along the vertical direction V. Thus, sump 142 is configured for
receipt of and generally collects wash fluid during operation of
washing machine appliance 100. For example, during operation of
washing machine appliance 100, wash fluid may be urged by gravity
from basket 120 to sump 142 through plurality of perforations
140.
[0023] A drain pump assembly 144 is located beneath wash tub 124
and is in fluid communication with sump 142 for periodically
discharging soiled wash fluid from washing machine appliance 100.
Drain pump assembly 144 may generally include a drain pump 146
which is in fluid communication with sump 142 and with an external
drain 148 through a drain hose 150. During a drain cycle, drain
pump 146 urges a flow of wash fluid from sump 142, through drain
hose 150, and to external drain 148.
[0024] More specifically, drain pump 146 includes a motor (not
shown) which is energized during a drain cycle such that drain pump
146 draws wash fluid from sump 142 and urges it through drain hose
150 to external drain 148.
[0025] A spout 154 is configured for directing a flow of fluid into
wash tub 124. For example, spout 154 may be in fluid communication
with a water supply 155 (FIG. 2) in order to direct fluid (e.g.,
clean water or wash fluid) into wash tub 124. Spout 154 may also be
in fluid communication with the sump 142. For example, pump
assembly 144 may direct wash fluid disposed in sump 142 to spout
154 in order to circulate wash fluid in wash tub 124.
[0026] As illustrated in FIG. 2, a detergent drawer 156 is slidably
mounted within front panel 130. Detergent drawer 156 receives a
wash additive (e.g., detergent, fabric softener, bleach, or any
other suitable liquid or powder) and directs the fluid additive to
wash tub 124 during operation of washing machine appliance 100.
According to the illustrated embodiment, detergent drawer 156 may
also be fluidly coupled to spout 154 to facilitate the complete and
accurate dispensing of wash additive.
[0027] In addition, a water supply valve or control valve 158 may
provide a flow of water from a water supply source (such as a
municipal water supply 155) into detergent dispenser 156 and into
wash tub 124. In this manner, control valve 158 may generally be
operable to supply water into detergent dispenser 156 to generate a
wash fluid, e.g., for use in a wash cycle, or a flow of fresh
water, e.g., for a rinse cycle. It should be appreciated that
control valve 158 may be positioned at any other suitable location
within cabinet 102. In addition, although control valve 158 is
described herein as regulating the flow of "wash fluid," it should
be appreciated that this term includes, water, detergent, other
additives, or some mixture thereof.
[0028] A control panel 160 including a plurality of input selectors
162 is coupled to front panel 130. Control panel 160 and input
selectors 162 collectively form a user interface input for operator
selection of machine cycles and features. For example, in one
embodiment, a display 164 indicates selected features, a countdown
timer, and/or other items of interest to machine users.
[0029] Operation of washing machine appliance 100 is controlled by
a controller or processing device 166 (FIG. 1) that is operatively
coupled to control panel 160 for user manipulation to select
washing machine cycles and features. In response to user
manipulation of control panel 160, controller 166 operates the
various components of washing machine appliance 100 to execute
selected machine cycles and features.
[0030] Controller 166 may include a memory and microprocessor, such
as a general or special purpose microprocessor operable to execute
programming instructions or micro-control code associated with a
cleaning cycle. The memory may represent random access memory such
as DRAM, or read only memory such as ROM or FLASH. In one
embodiment, the processor executes programming instructions stored
in memory. The memory may be a separate component from the
processor or may be included onboard within the processor.
Alternatively, controller 166 may be constructed without using a
microprocessor, e.g., using a combination of discrete analog and/or
digital logic circuitry (such as switches, amplifiers, integrators,
comparators, flip-flops, AND gates, and the like) to perform
control functionality instead of relying upon software. Control
panel 160 and other components of washing machine appliance 100 may
be in communication with controller 166 via one or more signal
lines or shared communication busses.
[0031] During operation of washing machine appliance 100, laundry
items are loaded into wash basket 120 through opening 132, and
washing operation is initiated through operator manipulation of
input selectors 162. Wash tub 124 is filled with water, detergent,
and/or other fluid additives, e.g., via spout 154 and or detergent
drawer 156. One or more valves (e.g., control valve 158) can be
controlled by washing machine appliance 100 to provide for filling
wash basket 120 to the appropriate level for the amount of articles
being washed and/or rinsed. By way of example for a wash mode, once
wash basket 120 is properly filled with fluid, the contents of wash
basket 120 can be agitated (e.g., with ribs 128) for washing of
laundry items in wash basket 120.
[0032] After the agitation phase of the wash cycle is completed,
wash tub 124 can be drained. Laundry articles can then be rinsed by
again adding fluid to wash tub 124, depending on the particulars of
the cleaning cycle selected by a user. Ribs 128 may again provide
agitation within wash basket 120. One or more spin cycles may also
be used. In particular, a spin cycle may be applied after the wash
cycle and/or after the rinse cycle in order to wring wash fluid
from the articles being washed. During a final spin cycle, basket
120 is rotated at relatively high speeds and drain pump assembly
144 may discharge wash fluid from sump 142. After articles disposed
in wash basket 120 are cleaned, washed, and/or rinsed, the user can
remove the articles from wash basket 120, e.g., by opening door 134
and reaching into wash basket 120 through opening 132.
[0033] Washing machine appliance 100 may further include a
microphone 180 that is used for monitoring the sound waves, noises,
or other vibrations generated during the operation of washing
machine appliance 100. For example, microphone 180 may be one or
more microphones, acoustic detection devices, vibration sensors, or
any other suitable acoustic transducers that are positioned at one
or more locations in or around washing machine appliance 100. For
example, according to exemplary embodiments, microphone 180 may be
mounted within cabinet 102. In addition, or alternatively,
microphone 180 may be positioned elsewhere within the room or
residence where washing machine appliance 100 is located. In this
regard, any suitable microphone 180 that is acoustically coupled
with washing machine appliance 100 may be used to monitor sounds
generated by washing machine appliance 100.
[0034] Notably, the sounds generated during operation of washing
machine appliance may be associated with one or more operating
conditions, failure modes, event occurrences, the presence of one
or more distinct items within a wash load, etc. For example, if a
user accidently leaves loose coins or a belt in a wash load, the
noise of these items striking wash basket 120 may create a unique
sound signature, identifiable for example by natural resonant
frequencies, amplitudes, the time-based excitations, the excitation
rate (e.g., the speed at which a particular sound is triggered),
the time decay of the generated sound waves, or any other acoustic
signature or characteristic. As explained in more detail below,
aspects of the present subject matter are directed to systems and
methods for monitoring sounds generated by an appliance, converting
those sounds into a three-dimensional spectrogram, and using
artificial intelligence image recognition processes to identify
sounds signatures in the spectrogram.
[0035] In addition, referring again to FIG. 1, washing machine
appliance 100 may generally include an external communication
system 190 which is configured for enabling the user to interact
with washing machine appliance 100 using a remote device 192.
Specifically, according to an exemplary embodiment, external
communication system 190 is configured for enabling communication
between a user, an appliance, and a remote server or network 194.
According to exemplary embodiments, washing machine appliance 100
may communicate with a remote device 192 either directly (e.g.,
through a local area network (LAN), Wi-Fi, Bluetooth, etc.) or
indirectly (e.g., via a network 194), as well as with a remote
server, e.g., to receive notifications, provide confirmations,
input operational data, transmit sound signals and sound
signatures, etc.
[0036] In general, remote device 192 may be any suitable device for
providing and/or receiving communications or commands from a user.
In this regard, remote device 192 may include, for example, a
personal phone, a tablet, a laptop computer, or another mobile
device. In addition, or alternatively, communication between the
appliance and the user may be achieved directly through an
appliance control panel (e.g., control panel 160).
[0037] In general, network 194 can be any type of communication
network. For example, network 194 can include one or more of a
wireless network, a wired network, a personal area network, a local
area network, a wide area network, the internet, a cellular
network, etc. In general, communication with network may use any of
a variety of communication protocols (e.g., TCP/IP, HTTP, SMTP,
FTP), encodings or formats (e.g. HTML, XML), and/or protection
schemes (e.g., VPN, secure HTTP, SSL).
[0038] External communication system 190 is described herein
according to an exemplary embodiment of the present subject matter.
However, it should be appreciated that the exemplary functions and
configurations of external communication system 190 provided herein
are used only as examples to facilitate description of aspects of
the present subject matter. System configurations may vary, other
communication devices may be used to communicate directly or
indirectly with one or more appliances, other communication
protocols and steps may be implemented, etc. These variations and
modifications are contemplated as within the scope of the present
subject matter.
[0039] While described in the context of a specific embodiment of
horizontal axis washing machine appliance 100, using the teachings
disclosed herein it will be understood that horizontal axis washing
machine appliance 100 is provided by way of example only. Other
washing machine appliances having different configurations,
different appearances, and/or different features may also be
utilized with the present subject matter as well, e.g., vertical
axis washing machine appliances. Moreover, the systems and methods
described herein may be used to monitor sounds generated by any
other suitable appliance or appliances.
[0040] Now that the construction of washing machine appliance 100
and the configuration of controller 166 according to exemplary
embodiments have been presented, an exemplary method 200 of
operating a washing machine appliance will be described. Although
the discussion below refers to the exemplary method 200 of
operating washing machine appliance 100, one skilled in the art
will appreciate that the exemplary method 200 is applicable to the
operation of a variety of other washing machine appliances, such as
vertical axis washing machine appliances. In exemplary embodiments,
the various method steps as disclosed herein may be performed by
controller 166 or a separate, dedicated controller.
[0041] Referring generally to FIG. 3, a method of operating a
washing machine appliance is provided. According to exemplary
embodiments, method 200 includes, at step 210, obtaining a sound
signal generated during operation of a washing machine appliance
using a microphone. For example, continuing the example from above,
microphone 180 may be used to detect noises, sounds, vibrations, or
other acoustic waves generated during the operation of washing
machine appliance 100. In addition, or alternatively, step 210 may
include monitoring the sounds generated by washing machine
appliance 100 while it is not in operation, sounds generated during
a diagnostic procedure, or any other suitable beeps, indicators, or
sound waves that emanate from washing machine appliance 100.
[0042] Step 220 includes generating a spectrogram from the sound
signal. In this regard, for example, controller 166 may be
configured for converting a sound clip or sound recording into a
spectrogram for subsequent analysis. Thus, the original recording
of sound from step 210 may be in the form of noise amplitude versus
time, noise frequency versus time, noise amplitude versus noise
frequency (e.g., a full Fourier transform or FFT), or any other
suitable two-dimensional representation of the measured sound. In
addition, any suitable duration of sound may be measured at step
210 and converted at step 220. For example, according to exemplary
embodiments, the sound signal is between about 0.1 seconds and 10
seconds, between about 1 in 5 seconds, or about 3 seconds.
[0043] Notably, the spectrogram generated at step 220 may be a
three-dimensional representation of sound pressure or amplitude at
a given frequency and time. Specifically, spectrograms may be a
two-dimensional graphs, with a third dimension represented by
colors. According to exemplary embodiments, the spectrogram
represents both a sound frequency and a sound amplitude of over
time. For example, such a spectrogram may be a visual
representation of the spectrum of frequencies of a signal as it
varies with time, sometimes referred to as waterfall diagrams. FIG.
4 provides an exemplary spectrogram that may be generated and
analyzed according to aspects of the present subject matter.
Notably, once the sound signal is converted to a spectrogram,
controller 166 may use various image recognition processes or
processing tools to identify noise sources and operating
conditions, and may use such information for improving machine
performance, e.g., by scheduling maintenance visits, adjusting
operating parameters, providing user notifications, etc. In this
regard, spectrogram images may add the element of time and may use
color temperature to signal intensity or noise amplitude for
improved knowledge of the appliance state or operation.
[0044] Step 230 includes identifying a sound signature by analyzing
the spectrogram using an image recognition process. For example,
image recognition processes that rely on artificial intelligence,
neural networks, or any other suitable known image processing
techniques may be used while remaining within the scope of the
present subject matter. Specifically, using such a spectrogram
image provides several advantages over existing sound recognition
processes.
[0045] For example, the use of a spectrogram provides the potential
to use a variety of sophisticated image recognitions models.
According to an exemplary embodiment, portions of the image
recognition processes may use single-label image convolution neural
networks (CNNs) as the main algorithm to compare/classify
spectrograms. As used herein, the terms image recognition and
similar terms may be used generally to refer to any suitable method
of observation, analysis, image decomposition, feature extraction,
image classification, etc. of the spectrogram generated from sound
signals measured from washing machine appliance 100. It should be
appreciated that any suitable image recognition software or process
may be used to analyze the spectrograms and controller 166 may be
programmed to perform such processes and take corrective
action.
[0046] According to an exemplary embodiment, controller may
implement a form of image recognition called convolutional neural
network ("CNN") image recognition. Generally speaking, CNN may
include taking an input image (e.g., a spectrogram) and using a
convolutional neural network to identify unique signatures in the
image, referred to herein generally as "sound signatures."
According to still other embodiments, the image recognition process
may use any other suitable neural network process.
[0047] In addition, or alternatively, an Adam optimizer may be
used, binary cross-entropy may be used as a loss function, and
softmax as a last layer activation may be used. Any other suitable
image classification technique may be used according to alternative
embodiments. For example, various transfer techniques may be used,
but use of such techniques is not required. If using transfer
techniques learning, a neural network architecture may be
pretrained such as VGG16/VGG19/ResNet50 with a public dataset then
the last layer may be retrained with an appliance specific
dataset.
[0048] In addition, or alternatively, the image recognition process
may detect dryness or other events that depend on comparison of
initial conditions. For example, a dry-initial spectrogram image
may be subtracted from a spectrogram image while clothes are
drying. The subtracted image may be used to train a neural network
with two classes: dry, not dry. If not using any transfer learning
VGG16 may be the neural net architecture of choice. In addition, or
alternatively, two spectrogram images may be stacked, e.g., the dry
initial spectrogram image from the spectrogram image on top and the
spectrogram image while drying on the bottom of the image. In other
words, according to exemplary embodiments, two images could be
concatenated in any suitable manner and order. Moreover, according
to alternative embodiments, two or more images could be combined by
subtracting two spectrogram images or modifying such images in any
other suitable manner. This combined image may be used in a similar
way to train a neural network with two classes: dry, not dry. If
detection of sound events does not require a comparison from the
initial conditions, image combination may be avoided. To detect,
for example, the washer being ON, a wide variety of spectrograms
recording of this event may be collected, label, and trained.
[0049] Notably, additional advantages of the use of spectrograms
include privacy. For example, sound data collected as an image in
inherently more private. In this regard, since the spectrogram
contains no information about the exact, or even approximate, phase
of the signal that it represents, the sound may be protected and
may not be derivable from the spectrogram. For this reason, it may
not be possible to reverse the process and generate a copy of the
original signal from a spectrogram. In addition, a spectrogram
image may allow for more effective memory use since it can be
compressed. Notably, compressing the spectrogram may make it easier
or less data intensive to transmit. Thus, for example, controller
166 may further be configured for transmitting the spectrogram
(e.g., or the compressed spectrogram) to a remote server (e.g.,
such as remote server 194) for analysis. Controller 166 may further
be configured for receiving analytic feedback from remote server
194. In this manner, data processing may be offloaded from
controller 166.
[0050] Notably, controller 166 may further be configured for
learning sound signatures associated with a washing machine
appliance 100. For example, common conditions or operating noises
may be intentionally generated to train a neural network model.
That model may then be used to detect particular sound signatures
associated with particular events. Such sound signatures may be
stored locally on controller 166 or a remote server 194. In
addition, sound signatures may be appliance specific, may be stored
according to a particular model or appliance configuration, or may
be associated with a washing machine appliance or another appliance
in any other suitable manner.
[0051] Step 240 includes adjusting at least one operating parameter
of the washing machine appliance based at least in part on the
identification of the sound signature. In this regard, if a sound
signature associated with a specific condition is identified at
step 230, controller 166 may take corrective action, e.g., by
adjusting one or more operating parameters or implementing some
other action in response to detecting that sound signature.
[0052] As used herein, an "operating parameter" of washing machine
appliance 100 is any cycle setting, operating time, component
setting, spin speed, part configuration, or other operating
characteristic that may affect the performance of washing machine
appliance 100. Thus, references to operating parameter adjustments
or "adjusting at least one operating parameter" are intended to
refer to control actions intended to improve system performance
based on the sound signature or other system parameters. For
example, adjusting an operating parameter may include adjusting an
agitation time or an agitation profile, adjusting a water level,
limiting a spin speed of wash basket 120, identifying service
needs, providing a user with operating guidance, etc. Other
operating parameter adjustments are possible and within the scope
of the present subject matter.
[0053] In addition, according to exemplary embodiments, adjusting
an operating parameter may include providing a user notification
when the sound signature indicates that a predetermined operating
condition exists. For example, according to one exemplary
embodiment, the sound signature may be associated with sounds
generated from one or more of a bearing, a belt, the motor 122, a
water valve (e.g., dripping or stuck in the ON position), a pump, a
suspension system, harmonics of structural components, undesirable
contact between components or subsystems, etc. When a sound
signature is generated that indicates a particular operating
condition, e.g., such as a potential failure of one of these
components, a user notification may be provided via display 164 or
directly to a user's remote device 192 (e.g., a cell phone, via
wireless connection).
[0054] FIG. 3 depicts steps performed in a particular order for
purposes of illustration and discussion. Those of ordinary skill in
the art, using the disclosures provided herein, will understand
that the steps of any of the methods discussed herein can be
adapted, rearranged, expanded, omitted, or modified in various ways
without deviating from the scope of the present disclosure.
Moreover, although aspects of method 200 are explained using
washing machine appliance 100 as an example, it should be
appreciated that these methods may be applied to the operation of
any suitable washing machine appliance.
[0055] This written description uses examples to disclose the
invention, including the best mode, and also to enable any person
skilled in the art to practice the invention, including making and
using any devices or systems and performing any incorporated
methods. The patentable scope of the invention is defined by the
claims, and may include other examples that occur to those skilled
in the art. Such other examples are intended to be within the scope
of the claims if they include structural elements that do not
differ from the literal language of the claims, or if they include
equivalent structural elements with insubstantial differences from
the literal languages of the claims.
* * * * *