U.S. patent application number 17/169343 was filed with the patent office on 2021-05-27 for utilizing context information of environment component regions for event/activity prediction.
This patent application is currently assigned to BrainofT Inc.. The applicant listed for this patent is BrainofT Inc.. Invention is credited to Maurice Chu, Deng Deng, Kunal Lad, Ashutosh Saxena, Ashwini Venkatesh, Jinjing Zhou.
Application Number | 20210160326 17/169343 |
Document ID | / |
Family ID | 1000005387206 |
Filed Date | 2021-05-27 |
View All Diagrams
United States Patent
Application |
20210160326 |
Kind Code |
A1 |
Saxena; Ashutosh ; et
al. |
May 27, 2021 |
UTILIZING CONTEXT INFORMATION OF ENVIRONMENT COMPONENT REGIONS FOR
EVENT/ACTIVITY PREDICTION
Abstract
Sensor data is received from a physical environment sensor. A
component region associated with at least a portion of the sensor
data is identified. A physical environment has been defined to
include a plurality of component regions. Context information
associated with the identified component region is obtained. The
context information is utilized in association with a prediction
model.
Inventors: |
Saxena; Ashutosh; (San
Mateo, CA) ; Zhou; Jinjing; (San Mateo, CA) ;
Chu; Maurice; (Burlingame, CA) ; Deng; Deng;
(Mountain View, CA) ; Lad; Kunal; (Milpitas,
CA) ; Venkatesh; Ashwini; (Milpitas, CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
BrainofT Inc. |
Redwood City |
CA |
US |
|
|
Assignee: |
BrainofT Inc.
Redwood City
CA
|
Family ID: |
1000005387206 |
Appl. No.: |
17/169343 |
Filed: |
February 5, 2021 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
15953238 |
Apr 13, 2018 |
10931758 |
|
|
17169343 |
|
|
|
|
15354115 |
Nov 17, 2016 |
10157613 |
|
|
15953238 |
|
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
H04W 4/029 20180201;
G06N 5/025 20130101; H04W 4/33 20180201; H04W 4/38 20180201; H04L
67/22 20130101; G06N 7/00 20130101; H04L 67/10 20130101; H04W 84/18
20130101; G06N 20/00 20190101; H04W 4/70 20180201; H04L 67/125
20130101 |
International
Class: |
H04L 29/08 20060101
H04L029/08; H04W 4/38 20060101 H04W004/38; G06N 7/00 20060101
G06N007/00; G06N 20/00 20060101 G06N020/00; H04W 4/029 20060101
H04W004/029; H04W 4/33 20060101 H04W004/33; H04W 4/70 20060101
H04W004/70; G06N 5/02 20060101 G06N005/02 |
Claims
1-21. (canceled)
22. A computer-executed method for home automation, comprising:
determining, by a computer, one or more component regions within a
physical environment; determining, for a respective component
region, context information associated with the component region;
receiving sensor data from one or more sensors in the physical
environment; and applying a prediction model to determine a state
of a detected subject in the physical environment based on the
determined component regions, context information associated with
the component regions, and the received sensor data.
23. The method of claim 22, wherein applying the prediction model
further comprises: identifying one or more component regions
associated with the received sensor data; selecting one or more
signal aggregators based on context information associated with the
identified component regions; and aggregating, by at least one
selected signal aggregator, sensor data from different sensors and
corresponding context information to obtain a contextual input for
the prediction model.
24. The method of claim 23, wherein a respective signal aggregator
comprises one or more of: a neural network; an aggregation
function; a rule function; and a machine-learning model trained
across multiple physical environments.
25. The method of claim 22, further comprising: in response to
determining that multiple sensors monitor a same component region,
correlating sensor data provided by the multiple sensors
together.
26. The method of claim 22, wherein the one or more sensors in the
physical environment include at least one of the following: a
switch, a camera, a motion detector, an infrared detector, a light
detector, an accelerometer, a thermal detector, a human pose
detector, a thermometer, an air quality sensor, a smoke detector, a
microphone, a humidity detector, a door sensor, a window sensor, a
water detector, a glass breakage detector, and a power utilization
sensor.
27. The method of claim 22, wherein the component regions include a
hierarchy of component regions.
28. The method of claim 22, wherein the component regions have been
defined, at least in part, using a graph model of the physical
environment.
29. The method of claim 28, wherein determining the context
information associated with the component region comprises
determining the context information from the graph model of the
physical environment.
30. The method of claim 22, wherein the context information
includes one or more of: spatial relationship information
associated with the component region, functional information
associated with the component region, information about an item
associated with the component region, and information about
functional capabilities of the item associated with the component
region.
31. The method of claim 22, wherein determining the state of the
detected subject comprises determining an activity performed by a
human user.
32. A computer system, comprising: a processer; a storage device
coupled to the processor and storing instructions, which when
executed by the processor cause the processor to perform a method
for home automation, the method comprising: determining, by a
computer, one or more component regions within a physical
environment; determining, for a respective component region,
context information associated with the component region; receiving
sensor data from one or more sensors in the physical environment;
and applying a prediction model to determine a state of a detected
subject based on the determined component regions, context
information associated with the component regions, and the received
sensor data.
33. The computer system of claim 32, wherein applying the
prediction model further comprises: identifying one or more
component regions associated with the received sensor data;
selecting one or more signal aggregators based on context
information associated with the identified component regions; and
aggregating, by at least one selected signal aggregator, sensor
data from different sensors and corresponding context information
to obtain a contextual input for the prediction model.
34. The computer system of claim 33, wherein a respective signal
aggregator comprises one or more of: a neural network; an
aggregation function; a rule function; and a machine-learning model
trained across multiple physical environments.
35. The computer system of claim 32, further comprising: in
response to determining that multiple sensors monitor a same
component region, correlating sensor data provided by the multiple
sensors together.
36. The computer system of claim 32, wherein the one or more
sensors in the physical environment include at least one of the
following: a switch, a camera, a motion detector, an infrared
detector, a light detector, an accelerometer, a thermal detector, a
human pose detector, a thermometer, an air quality sensor, a smoke
detector, a microphone, a humidity detector, a door sensor, a
window sensor, a water detector, a glass breakage detector, and a
power utilization sensor.
37. The computer system of claim 32, wherein the component regions
include a hierarchy of component regions.
38. The computer system of claim 32, wherein the component regions
have been defined, at least in part, using a graph model of the
physical environment.
39. The computer system of claim 38, wherein determining the
context information associated with the component region comprises
determining the context information from the graph model of the
physical environment.
40. The computer system of claim 32, wherein the context
information includes one or more of: spatial relationship
information associated with the component region, functional
information associated with the component region, information about
an item associated with the component region, and information about
functional capabilities of the item associated with the component
region.
41. The computer system of claim 32, wherein determining the state
of the detected subject comprises determining an activity performed
by a human user.
Description
CROSS REFERENCE TO OTHER APPLICATIONS
[0001] This application is a continuation-in-part of co-pending
U.S. patent application Ser. No. 15/354,115 entitled CONTROLLING
CONNECTED DEVICES USING A RELATIONSHIP GRAPH filed Nov. 17, 2016,
which is incorporated herein by reference for all purposes.
BACKGROUND OF THE INVENTION
[0002] Network connected devices (e.g., Internet of Things (i.e.,
IoT) devices) allow remote control and automation of the devices
within an environment (e.g., home). However, these devices are
often not capable of being fully autonomous. Often users must
manually operate the devices in a remote control fashion, or users
manually create complex rules that try to mimic desired operations.
For example, the user must be aware of where devices are
specifically located, the capability/range of the device, and how
each device will specifically impact the surrounding environment as
it relates to a desired result. Additionally, when the user is in a
new environment, the user must reconfigure, retrain or relearn the
automation of the devices.
BRIEF DESCRIPTION OF THE DRAWINGS
[0003] Various embodiments of the invention are disclosed in the
following detailed description and the accompanying drawings.
[0004] FIG. 1A is a diagram illustrating an embodiment of a system
for automatically controlling network connected devices.
[0005] FIG. 1B is a diagram illustrating an embodiment of
interactions between components for automatically controlling
network connected devices.
[0006] FIG. 1C is a diagram illustrating an embodiment of sub
components of a system for automatically controlling network
connected devices.
[0007] FIG. 1D is a diagram illustrating an embodiment of
conceptual components included in determining an activity state of
a subject.
[0008] FIG. 1E is a diagram illustrating an example of a history of
contextual inputs.
[0009] FIG. 1F is a diagram illustrating an embodiment of using a
context mapper to at least in part detect an activity.
[0010] FIG. 2A is a flowchart illustrating an embodiment of a
process for associating context information to component
regions.
[0011] FIG. 2B is a flowchart illustrating an embodiment of a
process for automatically learning and applying control rules.
[0012] FIG. 2C is a flowchart illustrating an embodiment of a
process for detecting one more states of a subject.
[0013] FIG. 3 is a flowchart illustrating an embodiment of a
process for determining a likelihood that a detected state is the
correct state.
[0014] FIG. 4 is a flowchart illustrating an embodiment of a
process for automatically discovering an automation rule of a
network connected controllable device.
[0015] FIG. 5 is a flowchart illustrating an embodiment of a
process for determining a selected output result.
[0016] FIG. 6 is a flowchart illustrating an embodiment of a
process for automatically determining an output candidate of a
controllable property of a network connected controllable
device.
[0017] FIG. 7 is a flowchart illustrating an embodiment of a
process for implementing a selected output.
[0018] FIG. 8 is a flowchart illustrating an embodiment of a
process for attempting to discover an improved output.
[0019] FIG. 9A is a diagram illustrating an example neighborhood
level of a hierarchical graph model of a connected device
environment.
[0020] FIG. 9B is a diagram illustrating an example building level
of a hierarchical graph model of a connected device
environment.
[0021] FIG. 9C is a diagram illustrating an example unit level of a
hierarchical graph model of a connected device environment.
[0022] FIG. 9D is a diagram illustrating an example room level of a
hierarchical graph model of a connected device environment.
[0023] FIG. 10 is a flowchart illustrating an embodiment of a
process for initializing a model of a connected device
environment.
[0024] FIG. 11 is a flowchart illustrating an embodiment of a
process for using an environment model to process a command.
DETAILED DESCRIPTION
[0025] The invention can be implemented in numerous ways, including
as a process; an apparatus; a system; a composition of matter; a
computer program product embodied on a computer readable storage
medium; and/or a processor, such as a processor configured to
execute instructions stored on and/or provided by a memory coupled
to the processor. In this specification, these implementations, or
any other form that the invention may take, may be referred to as
techniques. In general, the order of the steps of disclosed
processes may be altered within the scope of the invention. Unless
stated otherwise, a component such as a processor or a memory
described as being configured to perform a task may be implemented
as a general component that is temporarily configured to perform
the task at a given time or a specific component that is
manufactured to perform the task. As used herein, the term
`processor` refers to one or more devices, circuits, and/or
processing cores configured to process data, such as computer
program instructions.
[0026] A detailed description of one or more embodiments of the
invention is provided below along with accompanying figures that
illustrate the principles of the invention. The invention is
described in connection with such embodiments, but the invention is
not limited to any embodiment. The scope of the invention is
limited only by the claims and the invention encompasses numerous
alternatives, modifications and equivalents. Numerous specific
details are set forth in the following description in order to
provide a thorough understanding of the invention. These details
are provided for the purpose of example and the invention may be
practiced according to the claims without some or all of these
specific details. For the purpose of clarity, technical material
that is known in the technical fields related to the invention has
not been described in detail so that the invention is not
unnecessarily obscured.
[0027] An interactive device controller is disclosed. For example,
network connected smart devices of a deployment environment are
controlled autonomously using a controller hub that anticipates
needs of a user and automatically controls smart devices without
requiring the user to manually operate a complex set of the smart
devices in a remote control fashion. The controller hub may
anticipate needs of the user by applying machine learning,
specified preferences of the user, and/or automation rules to
detected sensor inputs (e.g., inputs from camera, microphone,
motion detector, etc.).
[0028] In some embodiments, context information is utilized in
learning and detecting states (e.g., detected user activities) that
trigger one or more actions. For example, rather than relying on
raw sensor information to learn and detect a state that triggers an
appropriate network connected smart device action, detected
information from one or more sensors in an environment is analyzed
in a context of higher level context information known about the
specific environment where the sensors are located (e.g., sensor
information combined with associated context information to
generate and/or as an input to a prediction model). This allows
sensor data to be interpreted as contextual events that are used by
one or more models to detect a state/activity of a user. By using
the higher level contextual events as inputs to these models, the
models can be used across various different environments by using
various different mappers/translators for different sensor data of
the different environments.
[0029] In some embodiments, an environment is divided into
component regions. Specific context information is associated with
each component region. For example, information about spatial
geometry, functional properties, and/or acoustic information
associated with each component region that has been specified,
detected, and/or learned is associated with corresponding component
regions. When an input is received (e.g., sensor data), one or more
component regions associated with the input are identified and
associated context information is obtained. Then using at least the
context information, a state (e.g., user activity) in the
environment is determined. The determined state of the user may be
used to trigger a network connected smart device.
[0030] However in certain situations, it may be difficult to
definitively determine that an action to control a smart device
(e.g., turn on a lightbulb) should be performed. For example, it
may be difficult to definitively determine a state of a user (e.g.,
current action being performed by a user) given sensor inputs
(e.g., camera image) when the action is triggered by the state of
the user. In some embodiments, rather than simply perform or not
perform the action, the user is prompted to confirm whether the
action should be performed. For example, the action has been
automatically identified (e.g., user has not specifically requested
that the action be performed but the system has automatically
identified the action as desirable) by the system to be potentially
performed in anticipating needs of the user. However because the
system is not sufficiently sure the action should be performed
given the uncertainty in analysis results of sensor inputs, the
system prompts the user via an audio output of a speaker whether
the system should perform the action. If the user confirms that the
action should be performed, the action is performed.
[0031] In some embodiments, sensor information about one or more
states of an environment is received. For example, sensors are
located in an enclosed physical location (e.g., home, building,
etc.) and sensor inputs are received for analysis to determine
various states of the environment (e.g., state of devices,
temperature, humidity, brightness, motion, states of one or more
humans or animals in the environment, etc.). In some embodiments,
one or more states are associated with a measure of certainty that
the state is correct based on the received sensor inputs. In some
embodiments, the one or more states are determined using context
information about one or more component regions associated with the
sensor information.
[0032] Based at least in part on the states detected using sensor
information, an expected value for automatically performing an
action as compared to an expected value for potentially performing
the action is evaluated. For example, the expected value is a
measure of a degree of desirability associated with the option. The
expected value is specific to the particular associated action. The
expected value may take into consideration potential benefit of
choosing an associated option for the action given the input
states, potential harm of not choosing the associated option for
the action given the input states, a history of past option
selections for the action, annoyance to a user for choosing the
associated option, etc. among other factors. In some embodiments,
an expected value is determined for each output handling type for
the action given the determined state(s). These output handling
types may include automatically performing the action (e.g., not
subject to a user interactive qualification), determining that the
action should be potentially performed subject to the user
interactive qualification, and not performing the action. The user
interactive qualification may be associated with inquiring a user
whether the action should be performed and only performing the
action if the user provides a confirmatory indication. In another
example, the user interactive qualification may be associated with
providing warning that the action will be performed unless
otherwise instructed and not performing the action if instructed
otherwise. The user interactive qualification may be a
prequalification of performing the action and a qualification
indication from a user may allow or prevent execution of the
action.
[0033] In some embodiments, not performing the action is associated
with waiting to perform the action because a user will likely
manually perform the action. In some embodiments, the evaluating
the expected values includes comparing the expected values
associated with each output handling type associated with a
particular action to select the expected value (e.g., highest or
lowest value) that corresponds to the most desirable result.
[0034] In some embodiments, in the event it is determined that the
expected value of automatically taking the action is insufficient
to trigger automatically performing the action without being
subject to the user interactive qualification, a user is provided
an interactive inquiry to request a response from the user on
whether the action should be performed or prevented from being
performed.
[0035] FIG. 1A is a diagram illustrating an embodiment of a system
for automatically controlling network connected devices. Devices
102 include one or more network connected devices including sensor
devices and controllable devices (e.g., IoT devices). Examples of
one or more controllable devices included in devices 102 include a
switch, a door lock, a thermostat, a light bulb, a kitchen/home
appliance, a camera, a speaker, a garage door opener, a window
treatment (e.g., roller shades, curtains, etc.), a fan, an
electrical outlet, an audio/visual device, a light dimmer, an
irrigation system, a pet feeder, and any other device able to be
connected to a computer network. Examples of one or more sensor
devices included in devices 102 include a switch, a camera, a
motion detector, an infrared detector (e.g., passive infrared
sensor), a light detector, an accelerometer, a thermal
detector/camera, a human pose/motion camera/detector, a
thermometer, an air quality sensor, a smoke detector, a microphone,
a humidity detector, a door sensor, a window sensor, a water
detector, a glass breakage detector, a power utilization sensor,
and any other sensor.
[0036] Hub 104 communicates with devices 102 via a wired and/or
wireless connection. For example, a device of devices 102
communicates with hub 104 via a communication interface configured
to utilize WiFi, Bluetooth, Bluetooth Low Energy, Zigbee, Z-Wave,
power-line communication, Ethernet, and/or any other wireless or
wired communication protocol. In some embodiments, hub 104 and
devices 102 have been deployed in the same environment. For
example, devices 102 include devices that have been deployed in a
home and hub 104 wirelessly communicates with the devices within
the home via a short range wireless signal. In some embodiments,
hub 104 facilitates communication and/or control of devices 102
with user device 108 and/or via network 110. For example, user
device 108 connects to hub 104 via a direct wireless connection
(e.g., Bluetooth, Wifi, etc.) and/or via network 110 to control and
view information about a device of devices 102. In another example,
user device 108 connects to network 110 to access devices 102 via
hub 104 and/or server 106. In some embodiments, one or more
devices/sensors of devices 102 report information to hub 104 that
in turn provides the reported information to server 106 via network
110 for processing. In some embodiments, one or more devices and/or
sensors of devices 102 directly communicate with network 110
without being routed via hub 104. For example, a device includes a
Wifi and/or cellular radio to directly access network 110.
[0037] Examples of user device 108 include a laptop computer, a
smartphone, a tablet computer, a desktop computer, a wearable
device, a smartwatch, and any other electronic computing device of
a user. In some embodiments, information from user device 108 is
provided to server 106 and/or hub 104 to enable automation of
devices 102. For example, a GPS location detected by user device
108 is provided to hub 104 and/or server 106 as a sensor input to
determine and/or trigger device automation rules. In some
embodiments, information associated with a device of devices 102 is
provided to user device 108. For example, a detected alert of a
device of devices 102 is provided to user device 108 via hub 104
and/or server 106. Server 106 may be a part of a cloud computing
infrastructure to process and/or store information associated with
devices 102 to enable control and management of devices 102. In
some embodiments, hub 104 functions as an interface and a control
hub to automatically control devices 102. Although storage elements
have not been explicitly shown in FIG. 1A, any of the shown
components may include a local storage and/or access a locally
connected and/or remote storage. For example, a local and/or remote
storage stores an environment model that maps relationships among
devices 102 and the component regions of the environment as well as
context information associated with component regions of the
environment.
[0038] In some embodiments, using the system shown in FIG. 1A,
machine learning is performed to automatically learn how users want
to interact with one or more controllable devices of devices 102 to
autonomously and automatically adjust properties of the
controllable devices. In some embodiments, powerful statistical
models of user preferences and behaviors are automatically built.
For example, sensors of devices 102 and user device 108 are
utilized along with the environment model and/or context
information associated with one or more component regions
associated with the devices to learn about a user's presence,
activities, and behavioral preferences. Controllable devices of
devices 102 may be automatically managed and controlled based on
the learned data. For example, rather than simply using
controllable devices of devices 102 in a remote control fashion by
requiring a user to explicitly provide control commands or user
defined control rules, devices are automatically controlled based
on learned user preferences without explicit user triggering of the
devices.
[0039] In some embodiments, machine learning (e.g., local and/or
cloud-based) is utilized to integrate input from many different
devices/sensors (e.g., devices 102) to build a unique model of a
user's presence, activities, and behavior. For example, an
environment such as home is installed with devices 102 that can be
controlled remotely or locally. The sensors of devices 102 may
provide data about the presence and motion of people in an
environment, measurements of the environmental properties such as
light, temperature and humidity of the house, motion of subjects,
and video of different locations of the environment. The machine
learning may take into account the component regions of the
environment and associated context information to account for
locational, functional, and behavior properties of the
devices/sensors and their data during the learning process.
[0040] In some embodiments, learning is performed to learn how to
automatically control the controllable devices and automatically
adjust the controllable devices of devices 102 to match the
automatically determined preferences of the users. For example,
sensors detect the environmental conditions that trigger changes in
the controllable device states (e.g., behavior and interaction with
switches, thermostats, and other devices) and the environmental
conditions are given context using the environment model and
context information of component regions, and devices of devices
102 are commanded autonomously in accordance with automatically
learned preferences of the users.
[0041] In some embodiments, machine learning is performed for a
particular environment by a local hub. For example, hub 104
performs learning for the users of its deployed environment
including devices 102 using a model of the environment. In some
embodiments, rather than performing machine learning at a local
hub, at least a portion of the machine learning is performed
remotely (e.g., cloud-based). For example, machine learning for the
environment of hub 104 is performed by server 106 that is aware of
the environment model and component region context information of
the local environment. In some embodiments, a backend server
performs learning across various different environments and their
component regions and utilizes the result to aid in automatically
configuring each different environment and its component regions.
For example, an environment model includes a hierarchy of nodes
that represent and connect together various component regions
(e.g., graph model of a building includes node elements for
individual units of the building) and learning is performed across
the entire graph of various component regions.
[0042] In some embodiments, hub 104 and/or server 106 includes one
or more inference engines that convert sensor data received from
one or more devices of devices 102 into state representations
(e.g., state of a person's behavior, location, etc.). For example,
the inference engine utilizes machine learning algorithms that rely
on statistical and/or deep learning techniques. In some
embodiments, hub 104 and/or server 106 includes a "vision engine"
(e.g., ML Inference) that receives images/video from one or more
camera sensors and analyzes the images/video using vision
algorithms to infer a subject's (e.g., human, pet, etc.) location,
behavior, poses (e.g., skeletal tracking), and activities (e.g.,
spatial and motion features). In some embodiments, camera video
data is analyzed to learn hand gestures of a person that control
connected devices to a desired state. In one embodiment, the
gestures are learned from statistical demonstrations by the person.
In one embodiment, the gestures are learned using deep learning. In
some embodiments, output of vision data and the data from other
sensor devices are utilized to build a semantic representation
containing information about a person's presence, activities, and
behavioral preferences. State information may be determined and/or
given context using the environment model and/or context
information about one or more associated component regions and
higher level information about one or more states may be
extrapolated using the environment model and/or context
information.
[0043] In some embodiments, in the system shown, the data is sent
using an event-driven database architecture. For example, the
sensor data is first converted into a compact feature vector before
streaming. In some embodiments, the sensor data is cleaned, before
streaming, by statistical learning models that model the different
types of noise in the system.
[0044] In some embodiments, hub 104 and/or server 106 includes one
or more "rule engines" that utilize a detected state to trigger one
or more automatically determined automation rules. For example, the
rule engine consists of rules with pre-conditions and
post-conditions and if a certain pre-condition state of the
environment is detected, a rule is triggered to cause one or more
network connected controllable devices to be placed in the
post-condition state. For example, if it is detected that a person
is reading a book on a desk, then the system will turn on reading
lights. The pre-conditions of a rule may be based on one or more
properties of the environment model.
[0045] In some embodiments, server 106 stores various types of
information that can be utilized to automatically determine
automation rules. For example, sensor data at various points in
time as well as device control events of devices 102 (e.g.,
pressing a light switch, changing a temperature of a thermostat,
etc.) are logged. By learning the preferences/configurations
desired by users, the preferences/configurations may be recreated
when the conditions in the environment match the previously
recorded conditions. In some embodiments, statistical models (e.g.,
transition model) of the control action of devices are determined.
For example, when a network connected light bulb is turned on or
off, the associated environmental conditions (e.g., user location,
action, time of day, etc.) and associated controllable device
status are stored. In some embodiments, which state vector maps to
which control action of controllable devices is learned. In some
embodiments, a learning method learns the weight of a function that
maps the value of the state vector to the state change of
controllable devices. In some embodiments, a learning method uses
deep learning in order to learn a multi-staged non-linear mapping
from the state vector to the state changes of controllable devices.
In some embodiments, feedback from the user regarding an action of
an automatically triggered rule is received in the form of a
reward, and reinforcement learning is used to automatically modify
the control rule. In some embodiments, server 106 stores the
environment model for one or more deployment environments.
[0046] In some embodiments, certain alarming or important events
are detected and the data is stored. For example, a break-in is
detected as an unusual event, and in response, output alerts are
generated including an alert that is sent to a smartphone and
activation of a siren. In another example, a baby climbing out of a
crib is detected. Unlike manually set alarms, the automatically
detected unusual events may be detected based on statistical
modeling of events using sensor data.
[0047] In various embodiments, examples of events that are able to
be detected include: presence events (e.g., house-level presence,
room-level presence, smaller area presence [e.g., sitting areas,
sleeping-areas, standing areas, etc.], etc.); people/human events
(e.g., face detection/recognition, voice finger printing, human
location/pose detection, human emotion detection [e.g., irritated,
etc.], etc.); pet events (e.g., pet activity); environment events
(e.g., stove/oven on/off, TV use, ventilation fan use, refrigerator
door open/close, package outdoor, etc.); and activity events (e.g.,
taking a shower, taking a bath, using toilet, watching TV, getting
up from bed, detecting quality of sleep, performing in bed
activity, detecting wake up time, sleeping in bed, drinking/eating,
sitting down at dining table, closing the door and going out,
opening the door and coming in, using cabinet, using refrigerator,
using microwave, using oven, using sink, making coffee, taking
medicine, loading laundry, controlling lighting, sitting down on
sofa, sitting down at desk, watering plant, cleaning floor, etc.).
One or more of the events may be associated with an intensity level
(e.g., low-motion, very-active motion, etc.).
[0048] In some embodiments, in response to detecting an event, an
output response (e.g., sound, user alert, light alert, wearable
alert, etc.) is generated. For example, when it is detected that a
stove is left on beyond a threshold amount of time and/or when no
human subject is detected as present, the stove is automatically
turned off and/or an output alert is generated. In another example,
when a water leak is detected, an automatic output alert is sent
and a water valve is automatically turned off. In another example,
when it is detected that a person has fallen and further movement
of the person is not detected, an alert to an emergency contact
person and/or an emergency authority is automatically sent. In
another example, if a presence of a person is detected in a living
room during morning times, curtains are automatically opened. In
another example, if it is detected that humidity is above a
threshold value, a fan is automatically turned on. In another
example, a humidifier is automatically switched on/off to maintain
a preferred humidity. In another example, when a learned preferred
time is reached, a coffee maker is automatically turned on. In
another example, a dishwasher is automatically scheduled to be
operated at a time when energy rates are relatively lower. In
another example, light intensity is automatically adjusted based on
a time of day (e.g., lights turned on a lower intensity when a
subject wakes up in the middle of the night to use the bathroom.).
In another example, music is automatically turned on when it is
detected that a subject is eating dinner. In another example, when
it is detected that ambient temperature and humidity are above
threshold values and a subject is detected as sitting, a fan is
automatically turned on. In another example, based on a detected
pattern of a subject walking into a location, pouring water into a
container, reaching for an object where a medicine is located, and
drinking water, a user is informed that a subject has taken a
medication. In another example, based on a detected pattern of a
subject sleeping in a bed, traveling to a bathroom, flushing noise
detection, and travel back to the bed, it is detected that a
subject went to and used a toilet at night. In another example,
based on a detected pattern of activities of a certain subject
(e.g., child) within a time period, a user is informed that the
subject has completed a routine (e.g., morning routine). In another
example, based on a detection that a subject rapidly drops to a
ground, a user is informed that the subject has likely fallen. In
another example, based on a detection that a subject is making a
loud noise that is not typical, a user is informed about the
subject.
[0049] In certain situations, it may be difficult to definitively
determine that an action (e.g., change a property of a device) to
control a device (e.g., controllable device of device 102) should
be performed. For example, it may be difficult to definitively
determine input states (e.g., states in a state vector) given
sensor inputs. Thus, if the output of an evaluation of an
automation rule is simply to change or not change a property of a
controllable device, uncertainty of environmental states provided
to the automation rule may result in either a property of the
controllable device being changed when it is not desirable to be
changed (e.g., undesirably aggressive triggering of change) or a
property of the controllable device being not changed when it is
desirable to be changed (e.g., undesirably conservative triggering
of change). In some embodiments, rather than an evaluation of an
automation rule simply resulting in performing and not performing
the action, the user is prompted to interactively confirm or deny
triggering of the action if input states to the automation rule
cannot be determined with a sufficient level of certainty. For
example, hub 104 prompts the user via an audio output of a speaker
whether the system should perform the action automatically
identified as desirable to be performed. If the user confirms that
the action should be performed, the action is performed. Example
results of handling an action triggered by an automation rule
include: performing the action, providing a warning that the action
will be performed then performing the action (e.g., with a user
opportunity to cancel performance of the action), inquiring a user
whether to perform the action (e.g., only performing the action if
the user confirms), determining that the user will likely manually
perform the action (e.g., not perform action and wait), and not
performing the action.
[0050] One or more of the following may be included in network 110:
a direct or indirect physical communication connection, mobile
communication network, Internet, intranet, Local Area Network, Wide
Area Network, Storage Area Network, a wireless network, a cellular
network, and any other form of connecting two or more systems,
components, or storage devices together. Additional instances of
any of the components shown in FIG. 1A may exist. For example,
multiple hubs may be deployed and utilized in a single environment.
Multiple user devices may also be utilized to receive user
associated information and/or control devices of the same
environment. Server 106 may be one of a plurality of distributed
servers that process network connected device data. In some
embodiments, components not shown in FIG. 1A may also exist.
[0051] FIG. 1B is a diagram illustrating an embodiment of
interactions between components for automatically controlling
network connected devices. Cloud-based machine learning module 120
communicates with rules engine 122. Rules engine 122 receives input
from sensors 124 and controls devices 126. Examples of cloud-based
machine learning module 120 include one or more modules for
performing machine learning discussed in conjunction with FIG. 1A.
Examples of rules engine 122 include one or more "rule engines" of
hub 104 and/or server 106 of FIG. 1A. Sensors 124 and/or controls
devices 126 may be included in devices 102 of FIG. 1A. Cloud-based
machine learning module 120 communicating with rules engine 122 may
utilize a model of the environment (e.g., graph model) including
sensors 124 and devices 126 to discover and invoke rules.
[0052] FIG. 1C is a diagram illustrating an embodiment of sub
components of a system for automatically controlling network
connected devices. In some embodiments, components 130 of the cloud
shown in FIG. 1C may be included in server 106 of FIG. 1A. In some
embodiments, components 132 of the local hub shown in FIG. 1C may
be included in hub 104 of FIG. 1A.
[0053] FIG. 1D is a diagram illustrating an embodiment of
conceptual components included in determining an activity state of
a subject. In some embodiments, the components of FIG. 1D may be at
least in part implemented on any device of devices 102, user device
108, hub 104, and/or server 106 of FIG. 1A.
[0054] Diagram 140 shows a plurality of input signals. These input
signals represent sensor data from sensors (e.g., from devices 102
of FIG. 1A, sensors 124 of FIG. 1B, etc.). The input signals are
processed using context mappers/translators that each can route at
least a portion of the input signal to one or more appropriate
signal aggregators. Based on a location of the environment where
the corresponding input signal was sensed/detected and associated
context information and/or based on which user/subject is
associated with the corresponding input signal and associated
context information, context mappers map an input signal and
associated context information to one or more appropriate signal
aggregators. For example, if an input signal is from a sensor that
is known to monitor a specific component region of an environment,
the context information associated with the specific component
region is utilized by the context mapper to route/map the input
signal from this sensor to one or more appropriate signal
aggregators along with the associated context information. In one
example, if "input signal 1" is the sound signal detected by a
microphone in a "kitchen" component region of a home environment,
it is routed to a "kitchen water sound detector" signal aggregator
and if "input signal 2" is a camera image detected by a camera in
the "kitchen" component region, it is routed to a "kitchen human
detection" signal aggregator. In another example, if a specific
user is known or detected to be associated with an input signal,
the context information associated with the specific user is
utilized by the context mapper to route/map the input signal to one
or more appropriate signal aggregators. In one example, if "input
signal 3" is a motion detection alert detected by a motion sensor
and a user of the house is known to have a pet, this input signal
is not routed to a "people in the house" signal aggregator because
of the user context of having a pet.
[0055] The shown signal aggregators take one or more of the input
signals and output a corresponding contextual input (e.g., water
sound detected in the kitchen, a female person in the kitchen,
etc.). By using the signal aggregators, sensor data from various
different sensors can be aggregated together to output a combined
cohesive contextual input that provides higher semantic level
context to otherwise raw sensor data. For example, portions of
detector sensor data are augmented with context information
associated with component regions corresponding to the sensor data
of the input signal(s) to allow activity state prediction models to
make better predictions based on the higher level contextual input
rather than simply raw sensor data. This allows the activity state
prediction models to work universally for different input data from
sensors in different deployment environments when they are
associated with same or similar context. Examples of the signal
aggregators include neural networks, aggregation functions, other
machine learning models, and rule functions. These signal
aggregators can also be learned models that are trained across
different deployment environments. For example, a "kitchen water
sound detector" signal aggregator has been trained using training
data from different apartments of different users.
[0056] Each of the shown universal activity prediction models takes
one or more of the appropriate contextual inputs and contributes in
outputting a probability of detection for one or more candidate
detected activities. For example, a probability that "washing
dishes in the kitchen" activity has been detected is determined.
Based on the calculated probabilities of the different candidate
detected activities, one or more of the candidate activities are
selected as being detected (e.g., meets a threshold criteria).
[0057] Some examples of activities that may be detected include:
wakeup activity (e.g., gets out of bed), device control action
activity (e.g., curtains open, lights turn on, music plays), in-bed
activity (e.g., sit up in bed and check the news on phone, get out
of bed and put dirty pajamas in laundry), morning routine activity
(e.g., brush teeth, shave apply hair gel hair blow dry),
bath/shower activity, cooking preparation activity (e.g., get pan
out of cabinet), obtaining an item from fridge activity (e.g., get
eggs and cheese from fridge, grab juice from fridge, pour, put
away), stovetop cooking activity (e.g., make the omelet on the
stove), plating food activity (e.g., transfer omelet from pan to
plate), making coffee activity (e.g., start coffee maker), eating
breakfast activity (e.g., sit down at table and eat breakfast),
leaving home environment activity (e.g., open door, leave house,
close door, lock door), package delivery activity, pet detected
activity, returning to home environment activity (e.g., unlock
door, open door, enter house, close door, lock door), turning on TV
activity (e.g., voice command or remote control command to turn on
TV), retrieving item from cabinet activity (e.g., retrieve bottle
opener), watching TV activity (e.g., TV volume control), phone call
activity (e.g., telephone conversation), oven cooking activity
(e.g., put the fish into the oven), cooking activity (e.g., start
kitchen timer), stovetop cooking activity (e.g., stir food in pot),
finished cooking activity (e.g., sit down at table and eat dinner),
finish eating activity (e.g., take plate and glass to sink),
dishwashing activity (e.g., load dishwasher), loading laundry
washer activity, laundry washer completion activity, using sink
activity (e.g., wash hands), brushing teeth activity, washing face
activity, walking to bedroom activity, working/reading on bed
activity, tossing/turning in bed activity, sleeping activity,
etc.
[0058] In some embodiments, the activity prediction models take
into account a history and temporal relation of previous contextual
inputs to determine a probability for one or more candidate
detected activities. FIG. 1E is a diagram illustrating an example
of a history of contextual inputs. As shown in FIG. 1E, the shown
temporal history 150 of the contextual inputs influences/leads to
detections of and/or probability calculations for the "take
medicine" activity and "use the toilet at night" activity.
[0059] FIG. 1F is a diagram illustrating an embodiment of using a
context mapper to at least in part detect an activity. Context
mapper 160 of FIG. 1F may be at least in part implemented on any
device of devices 102, user device 108, hub 104, and/or server 106
of FIG. 1A. In some embodiments, context mapper 160 includes the
context mapper shown in FIG. 1D. Context mapper 160 receives input
data from detectors (e.g., camera images), and based on which image
portion a motion and/or a user was detected and known component
region(s) corresponding to with the image portion, an activity of a
user is detected.
[0060] FIG. 2A is a flowchart illustrating an embodiment of a
process for associating context information to component regions.
The process of FIG. 2A may be at least in part performed by any
device of devices 102, user device 108, hub 104, and/or server 106
of FIG. 1A.
[0061] At 202, an environment is divided into component regions.
The environment includes a deployment environment where sensors are
located to detect one or more subjects that will be interacting
with the environment. Based on the detection by the sensors,
network connected devices are automatically controlled. Each of the
component regions is identified by a location identifier (e.g., one
or more location coordinates, an area identifier, a relative
location identifier, etc.). For example, a component region is
defined by location coordinates identifying a polygon/polyhedral
boundary region. In another example, a component region is defined
by a radius or other distance value away from a point or
feature/object that encompasses the component region (e.g., area
within 36 inches away from a sofa furniture item is included in a
sofa component region). The component region may be a point
location, a two-dimensional area, or a three-dimensional space/area
(e.g., bounds of the 3D area specified by 3D location coordinates)
in various embodiments. One or more of the component regions may at
least in part overlap with one another and may be hierarchically
related to one another (e.g., "kitchen" component region is
hierarchically higher than "kitchen sink area" component region).
The component regions are spatially related to one another and
spatial relationships between them may be specified by an
environment model of the environment (e.g., a graph model as
further discussed in FIGS. 9A-9D). Examples of the environment
include a home, an apartment, a house, a commercial establishment,
a building, etc. Examples of the component region include
space/room of a home (e.g., a bedroom, a kitchen, a dining room, a
bathroom, a laundry room, a garage, a living room floor region, a
bedroom wall region, etc.), area on/near/surrounding a home feature
(e.g., a vanity region, a sink region, a stove region, a shower
region, etc.), area on/near/surrounding an object in the
environment (e.g., a sofa region, a bed region, a TV region, a desk
region, etc.), etc.
[0062] In some embodiments, a specification of the component
regions of the environment is received. For example, an externally
generated specification (e.g., configured by an administrator,
detected and generated by an external device, etc.) is received. In
some embodiments, the component regions of the environment are
detected automatically. For example, sensors deployed/installed in
the environment automatically detect component regions of the
environment (e.g., using a machine learning model trained using
training data gathered using previous sensor data and/or provided
from a preselected training data). In some embodiments, for each
sensor included in an environment, the component regions detected
by the sensor are identified/configured. For example, for a camera
sensor, how each portion of an image captured by the camera maps to
a specific component region is determined. In another example, for
a microphone sensor, a relative location of the microphone sensor
within the environment and the acoustic pattern for the microphone
(e.g., from which component regions' sound can be detected by the
microphone sensor) given its location within the environment are
identified. In another example, for a motion detector, component
regions that can be sensed by the motion detector are identified.
By knowing how and which component regions each sensor detects, all
of the sensed data for a same component region can be correlated
together in determining a change/state in the component region.
Additionally, given a known relationship between the component
regions, data from sensors covering different component regions can
be analyzed together in light of the known relationship.
[0063] At 204, context information is associated with each
component region of at least a portion of the component regions.
The context information includes spatial relationship information
and/or functional information associated with the corresponding
component region. For example, the context information includes
information about functional capabilities (e.g., sitting, sleeping,
cooking, washing, storage, etc.) associated with a component
region. In another example, the context information includes
information about items associated with the component region (e.g.,
items in the component region, items stored in a storage unit in
the component region, etc.). In another example, the context
information includes information about functional capabilities of
the items associated with the component region. In another example,
the context information includes information about a spatial
relationship of the component region with other component regions
(e.g., hierarchical relationship, adjacency information, connection
information, graph relationship, etc.).
[0064] In some embodiments, a specification of the context
information of the component regions of the environment is
received. For example, an externally generated specification (e.g.,
configured by an administrator, detected and generated by an
external device, etc.) is received. In some embodiments, the
context information for the component regions of the environment is
detected automatically. For example, sensors deployed/installed in
the environment automatically detect context information for the
component regions of the environment (e.g., using a machine
learning model trained using training data gathered using previous
sensor data and/or provided from a preselected training data). For
example, a test acoustic sound is emitted and acoustic pattern
context information for various different context regions is
determined using microphones for the various different context
regions.
[0065] At 206, the component region information and associated
context information are utilized. For example, the component region
information and associated context information are utilized to
train a prediction model and/or perform a prediction of a state.
Context information may be utilized in learning and detecting
states/activities that trigger one or more actions associated with
network connected devices. For example, rather than relying on raw
sensor information to learn and detect a user state to trigger an
appropriate network connected smart device action, detected
information from one or more sensors in an environment is analyzed
using appropriate context information of a context region
associated with the sensor(s) to determine a detected
state/activity that is utilized to train a prediction model and/or
make a prediction using a prediction model. This allows sensor data
to be translated into semantically higher level events that are
used by one or more prediction models to detect a state/activity of
a user. By using the higher level events as inputs to these
prediction models, the prediction models can be used across various
different environments by using various different
mappers/translators for different sensor inputs of the different
environments.
[0066] FIG. 2B is a flowchart illustrating an embodiment of a
process for automatically learning and applying control rules. The
process of FIG. 2B may be at least in part performed by any device
of devices 102, user device 108, hub 104, and/or server 106 of FIG.
1A. In some embodiments, at least a portion of the process of FIG.
2B is performed in 206 of FIG. 2A.
[0067] At 212, sensor data is received. In some embodiments, the
received sensor data includes data from one or more sensor devices
of devices 102 of FIG. 1A. For example, data from a switch, a
camera, a motion detector, an infrared detector (e.g., passive
infrared sensor), a light detector, an accelerometer, a thermal
detector/camera, a human pose/motion camera/detector, a
thermometer, an air quality sensor, a smoke detector, a microphone,
a humidity detector, a door sensor, a window sensor, a water
detector, a glass breakage detector, a power utilization sensor,
and any other sensor monitoring an environment is received. In some
embodiments, the sensor data includes data from a user device
(e.g., user device 108 of FIG. 1A). For example, data from a sensor
on the user device (e.g., location sensor, GPS, accelerometer,
heart rate sensor, orientation sensor, microphone, gyroscope, etc.)
is received. In another example, the data from the user device
includes status data and/or user specified data. In some
embodiments, the sensor data includes data from one or more
controllable devices. For example, a status, a configuration, a
functional state, a parameter, and/or any other data of a
controllable device of devices 102 of FIG. 1A is received. In some
embodiments, the sensor data is received periodically. For example,
a sensor device sends currently detected sensor data periodically.
In some embodiments, the sensor data is received dynamically. For
example, the sensor data is received when sensor data has been
detected. In some embodiments, the received sensor data is received
from a plurality of sensor devices. In some embodiments, the sensor
data is received at a hub (e.g., hub 104 of FIG. 1A) and the sensor
data is shared/sent to another hub and/or sent to the computing
cloud for processing (e.g., sent to server 106 of FIG. 1A).
[0068] At 214, one or more states are identified using the sensor
data. In some embodiments, the sensor data is analyzed to identify
component regions of the environment detected in the received data
(e.g., selected from component regions of 202 of FIG. 2A identified
for the environment) and for each of the identified component
regions, associated context information (e.g., determined for
component regions in 204 of FIG. 2A), if any, is obtained. For
example, specific sensor data from a sensor has been identified to
detect a specific subset of component regions of the environment
and the specific sensor data is analyzed in the context of context
information associated with the specific subset of component
regions of the environment for the sensor. In one example, when a
person (and the specific body pose of the person) is detected in a
portion of an image captured by a camera, it is identified which
component region the portion of the image corresponds to and the
detected presence/body pose of the person is associated with the
identified component region and its associated context information
when provided as input to train a prediction model or to generate a
prediction using a prediction model. In another example, when a
voice command is detected by a microphone, it is identified which
component region a user that originated the voice command is likely
located in and the voice command is associated with the identified
component region and its associated context information when
provided as input to train a prediction model or to generate a
prediction using a prediction model.
[0069] In some embodiments, machine learning is performed using the
identified component regions, the associated context information of
the identified component regions, and/or the received sensor data.
In some embodiments, performing machine learning includes utilizing
a recursive hidden Markov model and/or expectation maximization. In
some embodiments, each state is associated with discrete categories
of information desired to be detected using machine learning. For
example, an activity currently being performed by a person/animal,
if present, is desired to be detected among a plurality of possible
activities able to be detected. This may be achieved by analyzing
camera video/image data in the context of associated context
information of the identified component regions for the video/image
to detect a person and activity performed by the person. In some
embodiments, a state for each detected subject is determined. In
some embodiments, by reducing the sensor data in the context of
associated context information of the identified component regions
to one or more specific states, the sensor data is reduced to
meaningful variable values that can be utilized to determine one or
more automation rules. In some embodiments, the state may be in a
form of a vector. For example, the state vector includes a grouping
of values. In some embodiments, the state vector includes one or
more of the following: a time value, a weather forecast, a date
value, and other data associated with time and/or environment
conditions.
[0070] In many cases, it is difficult to determine a specific state
with complete accuracy. For example, using the sensor data, it may
be difficult to determine the exact location and activity of a
subject. In some embodiments, a likelihood/probability that the
determined state is correct is determined. For example, a certain
state may be one of a plurality of different possible states and
the probability that each possible state is the correct state is
determined. In some embodiments, in order to determine the
probability, machine learning (e.g., statistical and/or deep
learning) may be performed. For example, statistical and/or deep
learning models of correlations between sensor associated data and
a potential state, a previous state and the potential state (e.g.,
transition model), and associations between different states/state
components are built and utilized to determine an overall
likelihood/percentage for each candidate state.
[0071] In some embodiments, the observations in time are utilized
as statistical inputs and utilized to estimate a state vector
evolving in time. The state vector may be the output of an
inference engine that converts the sensor data into information
about a detected subject presence, activities, etc. In one
embodiment, the dynamic processes of the subjects in the house
(e.g., human and pet) and their motions are modeled. In one
embodiment, deep learning is utilized to learn non-linear
relations. In some embodiments, sounds captured from microphone
sensors are utilized to infer the number of people, their location,
and their activities. In one embodiment, statistical models are
utilized to relate sound to the aforementioned state vector.
[0072] In some embodiments, a state vector including the activity
state (e.g., general activity state--reading, sleeping, cooking,
etc. and detailed activity state--reading-opening-book,
placing-book-down, etc.) of a subject is estimated. In one
embodiment, the state vector includes other relevant information
such as time-of-the-day, weather, number of people in the house,
subject locations, and current activities being performed. In some
embodiments, the state vector includes one or more controllable
device states. In some embodiments, the state vector includes a
listing of subjects with each region (e.g., room) of an
environment. In some embodiments, a learning method learns the
weight of a function that maps the values of the sensor data to the
state vector. In some embodiments, a learning method utilizes deep
learning to learn a multi-staged non-linear mapping from the sensor
data to the state vector.
[0073] In some embodiments, one or more states are identified using
an environment model of the environment. For example, an
environment graph model stored in server 106 and/or hub 104 of FIG.
1A is utilized. An example of the graph model is shown in FIGS.
9A-9D. Because the graph model identifies nodes and relationships
between nodes, an identified state and/or detected sensor data can
be augmented with context information from the graph model. For
example, physical location information about sensors and their
relationship to each other and component regions of the physical
environment can be utilized to determine additional location
information of detected sensor data because information of the
physical placement of the sensors (e.g., location, orientation,
etc.) can be used to provide context on the sensed information. In
some embodiments, additional states determined from the environment
model and/or context information of component regions included in
the environment model are included in the state vector. In some
embodiments, the environment model enables states and/or sensor
data to be abstracted. For example, sensor data can be generalized
to higher level information about regions associated with the type
of region where the sensor data was obtained.
[0074] In some embodiments, data from the sensors and/or cameras is
analyzed in order to predict what activities the person may perform
or locations the person may go to. Based on the activity and/or
location prediction, anticipatory actions may be taken. For
example, if a user starts moving towards a reading desk, it is
predicted that the user will go to the reading desk to read and a
network connected light will turn on even before the user starts
reading.
[0075] At 216, one or more automation rules are discovered based on
the identified state(s). For example, once it has been observed
that an identified state is correlated with a certain controllable
device state/status/action, a rule that places the controllable
device into the associated state/status/action when the associated
state is detected is created. Correlation between a determined
state and a detected controllable device state/status/action may be
identified and once the correlation reaches a threshold, an
automation rule is dynamically created and/or updated. In some
embodiments, a correlation between a group of states and/or a range
of state values with a controllable device state/status/action is
identified and utilized to generate an automation rule. In some
embodiments, the probability measure of each state may be utilized
when determining the correlations and/or automation rules. In some
embodiments, a history of determined states and associated
probability values and co-occurring controllable device
states/status/actions over time are stored and analyzed using
machine learning (e.g., statistical and/or deep learning) to
discover correlations. In the event a measure of correlation is
above a threshold value, a corresponding automation rule may be
created/updated. In some embodiments, automation rules are
continually added/updated based on new correlations that are
discovered.
[0076] In some embodiments, the rules in the rule engine that
control the devices are automatically learned. In one embodiment,
the rules are formulated as a reinforcement learning problem in
order to learn the policy. In one embodiment, the execution of the
policy is performed using proportional-derivative controllers. The
rule engine may take actions (e.g., changing the state of the
devices) based on triggering state vector and sensor information.
In one embodiment, parameters of a machine learning algorithm are
modified using online stochastic gradient algorithms. The rule
engine may take into account data from a user device, web services,
weather services, calendar events, and other data sources to learn
and/or trigger rules.
[0077] In some embodiments, discovering the one or more automation
rules includes utilizing the context information associated with
corresponding identified component regions of the environment
(e.g., obtained using the graph model of the environment). For
example, automation rules may be based on specific component
regions and/or associated context information (e.g., based on
specific nodes, areas/regions, relationships, and/or edges of an
environment graph model). In some embodiments, the context
information associated with corresponding identified component
regions enables more generalized rules to be discovered. For
example, without reliable and generalizable location information
and/or relationships of devices, only detected sensor/device based
rules are discovered. However, with context information of
component regions, more generalized rules that are applicable to
general location concepts/descriptions (e.g., kitchen) and/or
relationships between devices are able to be discovered. By
allowing more generalized rules to be discovered, invoked, and/or
utilized, rules become applicable in more situations and locations.
For example, a rule specific to a particular user but generalized
to a broader area/region may be applied in different environments
as the particular user travels to different environments with
different environment models but with a same labeled area/region
(e.g., rule to turn on lights at night when the user enters a
kitchen area can be applied across all kitchens visited by the user
even though actual light devices and their configurations are
different across different kitchens). In some embodiments, by
allowing actions to be generalized to broader concepts using
identified component regions (e.g., generalized to broader location
context), rule discovery may take place across various different
types of deployment environments (e.g., by same user and/or
different users).
[0078] At 218, an automatically discovered rule is invoked based on
the determined state. For example, a triggering condition of the
automatically discovered rule is the identified state, and the rule
is invoked to modify a property (e.g.,
state/status/function/action, etc.) of a controllable device to be
as specified by the rule. In some embodiments, the triggering
condition of the rule is a plurality of different states and/or a
range of states. In some embodiments, a plurality of rules is
applied/triggered based on the determined state. In some
embodiments, the applied rule is inconsistent with another rule to
be applied. For example, one rule specifies that a light switch
should be turned on while another rule specifies that the light
switch should be turned off. In order to resolve conflicts between
rules, each rule may be associated with a preference value that
indicates a measure of preference of the rule over another rule. In
some embodiments, feedback from the user regarding an action of an
automatically triggered rule is received in the form of a reward,
and reinforcement learning is used to automatically modify the
control rule.
[0079] In some embodiments, invoking the rules includes utilizing
component regions of an environment model. For example, one or more
triggering conditions and/or desired results are based on component
regions. Rather than requiring the condition to refer to a specific
device, the condition may reference a generalized concept that can
be identified based on component regions (e.g., specified using the
nodes and edges of an environment graph model). This allows an
example rule to turn on a light in the "kitchen" component region
rather than having to specify a specific device identifier of a
specific light bulb.
[0080] In some embodiments, an optimization function is utilized to
modify a property of a controllable device. For example, at each
periodic interval, state inputs are received (e.g., user states,
device/sensor states, environmental states, states of 214, etc.)
and utilized to select the optimal property setting of the
controllable device given the state inputs. This includes utilizing
an optimization function (e.g., cost function) to evaluate
optimization evaluation values (e.g., learned or specified cost
values) of candidate property settings of the controllable device
to select and implement the optimal candidate property setting. One
or more components of the optimization evaluation values may vary
according to one or more dynamic parameters.
[0081] FIG. 2C is a flowchart illustrating an embodiment of a
process for detecting one more states of a subject. The process of
FIG. 2C may be at least in part performed by any device of devices
102, user device 108, hub 104, and/or server 106 of FIG. 1A. In
some embodiments, at least a portion of the process of FIG. 2C is
performed in 206 of FIG. 2A. In some embodiments, the process of
FIG. 2C corresponds to the utilization of the conceptual components
shown in FIG. 1D.
[0082] At 222, sensor data is received. In some embodiments, the
received sensor data includes data from one or more sensor devices
of devices 102 of FIG. 1A. For example, data from a switch, a
camera, a motion detector, an infrared detector (e.g., passive
infrared sensor), a light detector, an accelerometer, a thermal
detector/camera, a human pose/motion camera/detector, a
thermometer, an air quality sensor, a smoke detector, a microphone,
a humidity detector, a door sensor, a window sensor, a water
detector, a glass breakage detector, a power utilization sensor,
and any other sensor monitoring an environment is received. In some
embodiments, the sensor data includes data from a user device
(e.g., user device 108 of FIG. 1A). For example, data from a sensor
on the user device (e.g., location sensor, GPS, accelerometer,
heart rate sensor, orientation sensor, microphone, gyroscope, etc.)
is received. In another example, the data from the user device
includes status data and/or user specified data. In some
embodiments, the sensor data includes data from one or more
controllable devices. For example, a status, a configuration, a
functional state, a parameter, and/or any other data of a
controllable device of devices 102 of FIG. 1A is received. In some
embodiments, the sensor data is received periodically. For example,
a sensor device sends currently detected sensor data periodically.
In some embodiments, the sensor data is received dynamically. For
example, the sensor data is received when sensor data has been
detected. In some embodiments, the received sensor data is received
from a plurality of sensor devices. In some embodiments, the sensor
data is received at a hub (e.g., hub 104 of FIG. 1A) and the sensor
data is shared/sent to another hub and/or sent to the computing
cloud for processing (e.g., sent to server 106 of FIG. 1A).
[0083] At 224, the sensor data is provided to one or more
corresponding signal aggregators selected based on component
region(s) and/or context information associated with at least a
portion of the sensor data. For example, the sensor data is
processed using one or more context mappers/translators that each
can route at least a portion of the sensor data to one or more
appropriate signal aggregators. The selection of the corresponding
signal aggregator(s) for a particular sensor input data portion may
be based on a location (e.g., a component region identified in 202
of FIG. 2A) of the environment where the particular sensor input
data portion was sensed/detected and/or based on which user/subject
is associated with the corresponding input sensor data. For
example, if the sensor data is from a sensor that is known to
monitor a specific component region of an environment, the context
information (e.g., context information of 204 of FIG. 2A)
associated with the specific component region is utilized by the
context mapper to route/map the input signal from this sensor to
one or more appropriate signal aggregators along with the
associated context information. In one example, if "input signal 1"
is the sound signal detected by a microphone in a "kitchen"
component region of a home environment, it is routed to a "kitchen
water sound detector" signal aggregator and if "input signal 2" is
a camera image detected by a camera in the "kitchen" component
region, it is routed to a "kitchen human detection" signal
aggregator. In another example, if a specific user is known or
detected to be associated with the sensor data, the context
information associated with the specific user is utilized by the
context mapper to route/map the input signal to one or more
appropriate signal aggregators along with the associated context
information. In one example, if "input signal 3" is a motion
detection alert detected by a motion sensor and a user of the house
is known to have a pet, this input signal is not routed to a
"people in the house" signal aggregator because of the user context
of having a pet.
[0084] At 226, outputs from signal aggregators are provided as
contextual inputs to one or more prediction models. The aggregators
take one or more of the routed sensor signals along with the
associated context information and output a corresponding
contextual input (e.g., water sound detected in the kitchen, a
female person in the kitchen, etc.). By using the signal
aggregators, sensor data from various different sensors can be
aggregated together to output a combined cohesive contextual input
that provides higher semantic level context (e.g., using component
region and context information of the process of FIG. 2A) to
otherwise raw sensor input data. For example, portions of the
sensor data are augmented with context information associated with
component regions corresponding to the sensor input data to allow
prediction models to make better predictions based on the higher
level contextual input rather than simply raw sensor data. This
allows the prediction models to work universally for different
input data from sensors in different deployment environments but
associated with the same context. Examples of the signal
aggregators include neural networks, aggregation functions, other
machine learning models, and rule functions. These signal
aggregators can also be learned models that are trained across
different deployment environments. For example, a "kitchen water
sound detector" signal aggregator has been trained using training
data from different apartments of different users. An example of
the prediction model is a machine learning model trained in 214 of
FIG. 2B.
[0085] At 228, the prediction models are utilized to determine one
or more states. Each of the prediction models takes one or more of
the appropriate contextual inputs and contributes in determining
whether one or more states have been detected. For example,
activity states of subjects are to be detected and a detection
probability is calculated for each one of one or more candidate
activities (e.g., a probability that "washing dishes in the
kitchen" has been detected is determined). Based on the calculated
probabilities of the different potential candidate detected
activities, one or more of the candidate activities are selected as
being detected (e.g., meets a threshold criteria). The prediction
models may take into account a history and temporal relation of
previous contextual inputs to determine a probability for one or
more candidate detected activities.
[0086] FIG. 3 is a flowchart illustrating an embodiment of a
process for determining a likelihood that a detected state is the
correct state. The process of FIG. 3 may be at least in part
implemented on any device of devices 102, user device 108, hub 104,
and/or server 106 of FIG. 1A. In some embodiments, at least a
portion of the process of FIG. 3 is performed in 214 of FIG. 2C. In
some embodiments, at least a portion of the process of FIG. 3 is
performed in 228 of FIG. 2B. In some embodiments, the process of
FIG. 3 is performed periodically. In some embodiments, the process
of FIG. 3 is performed dynamically. For example, the process of
FIG. 3 is performed when new sensor data is received.
[0087] At 312, candidate states of an actual state are determined.
For example, candidate states that may correspond to a newly
received sensor data are identified. In some embodiments, the
candidate states are possible states of an actual current state of
a subject (e.g., human, animal, pet, robot, other living or
non-living object, etc.). For example, because the exact current
state (e.g., location and activity currently being performed) of a
subject may be difficult to determine from sensor data with
complete accuracy, candidates states (e.g., each candidate state
including a location component and an activity component) for a
subject are identified.
[0088] In some embodiments, determining the candidate states
includes identifying all possible states that can be associated
with the received sensor data. For example, all possible predefined
activities of a subject that can be identified using data from a
camera are identified. In some embodiments, determining the
candidate states includes identifying the most likely candidate
states. For example, rather than identifying all possible states,
the most likely candidate states are identified. In some
embodiments, the most likely candidate states are identified by
analyzing associated sensor data received in 202 of FIG. 2A, 212 of
FIG. 2B, and/or 222 of FIG. 2C. In some embodiments, determining
the candidate states includes identifying a subject associated with
newly received sensor data and identifying the last determined
state for the subject. In some embodiments, the most likely
candidate states are identified based on a previous current state.
For example, for a given previous state (e.g., a location of a
subject), only certain states are eligible to become the new
current state (e.g., only locations adjoining the previous
location) and these states are identified.
[0089] In some embodiments, a single state includes a plurality of
sub states. In some embodiments, each state includes an identifier
of a subject, a coarse location of the subject (e.g., which room of
a house/building), a specific location of the subject within the
coarse location (e.g., on the bed of a bedroom), whether the
subject is present within an environment, a type of the subject
(e.g., human vs. pet, specific individual, etc.), a coarse activity
of the subject (e.g., reading), and the specific activity of a
subject (e.g., opening a book). In some embodiments, each candidate
state includes a state of a controllable object. In some
embodiments, an activity state of a subject is one of predefined
activities that can be detected (e.g., detected based on
observed/training data).
[0090] In some embodiments, determining the candidate states
includes identifying states based on an environment model (e.g.,
graph model shown in FIGS. 9A-9D) and/or context information about
one or more associated component regions. For example, received
sensor data is given context by associating with the sensor data,
context information of component regions detected in the sensor
data using a graph model (e.g., information/properties of edges
connected to the node of the sensor that detected the data and
information/properties of other nodes connected to the node of the
sensor). Possible states may be identified and/or eliminated based
on the environment model and/or context information about one or
more associated component regions.
[0091] At 314, for each of the candidate states, a likelihood that
the candidate state corresponds to a received sensor data is
determined. For example, a probability that a received sensor data
corresponds to a particular candidate state is determined. In some
embodiments, the likelihood identifies a probability that the
candidate state is the actual state given the sensor data. In some
embodiments, this likelihood is determined using machine learning.
For example, statistical and/or deep learning processing has been
utilized to analyze a training data set of example associations
between different sensor data and corresponding states to determine
associated probabilities of association between different sensor
data and different states. In some embodiments, the likelihood that
the candidate state corresponds to the received sensor data is
determined using a predetermined analysis algorithm. For example, a
computer vision pattern recognition algorithm is utilized to
analyze camera sensor data and the algorithm provides the
likelihood. In some embodiments, the likelihood is determined using
an environment model and/or context information about one or more
associated component regions.
[0092] At 316, for each of the candidate states, a likelihood that
the candidate state is the next state after a previously identified
state is determined. For example, a probability that the candidate
state is the actual state after a previously determined state of a
subject is determined. In some embodiments, this likelihood is
determined using machine learning. For example, statistical and/or
deep learning processing has been utilized to analyze observed
state transitions between different states to determine a
transition model of probabilities for each potential candidate
state given a previous state. In one example, a motion detector
sensor has been installed in each room of a house. The relative
locations of the rooms of the house may be automatically determined
by using machine learning to observe the pattern of sensor triggers
as subjects move from one room to another room. Once the
connections between the rooms are known, given a current room
location of a subject, the possible adjoining rooms are known and
each likelihood that the subject will visit a next room of the
possible connected rooms may be determined. For example, given the
previous state that indicates a location of a subject, the next
state is limited to adjoining rooms that are reachable given the
determined/observed rate of movement of the subject and elapsed
time between the sensor data of the states. In some embodiments,
the likelihood that the candidate state is the next state is
determined using the environment model.
[0093] At 318, for each of the candidate states, a concurrent state
component correlation is determined. For example, certain candidate
state components are more likely included together in the correct
state than another grouping of candidate state components. In some
embodiments, determining the concurrent state correlation includes
determining a probability that one component of the candidate state
is included in the correct/actual state given another component of
the candidate state. For example, a candidate state includes a
location state component and an activity state component, and the
probability of the specific activity state component given the
specific location state component is determined. In some
embodiments, determining the concurrent state correlation includes
determining a plurality of probabilities each associated with a
different pair combination of state components of the candidate
state. In some embodiments, the concurrent state component
correlation is determined using the environment model.
[0094] At 320, for each of the candidate states, an overall
likelihood that the candidate state is the actual state is
determined. For example, for each candidate state, the overall
probability that the candidate state is the correct state of a
subject is determined. In some embodiments, determining the overall
state includes multiplying together one or more of the
probabilities determined in 314, 316, and 318. For example, at
least a first probability that the candidate state corresponds to a
received sensor data, and a second probability that the candidate
state is the next state after a previously identified state are
multiplied together to obtain the overall likelihood. In some
embodiments, the candidate states are sorted based on their overall
likelihoods and the candidate state with the best overall
likelihood is selected as the actual/correct state.
[0095] FIG. 4 is a flowchart illustrating an embodiment of a
process for automatically discovering an automation rule of a
network connected controllable device. The process of FIG. 4 may be
at least in part implemented on any device of devices 102, user
device 108, hub 104, and/or server 106 of FIG. 1A. In some
embodiments, at least a portion of the process of FIG. 4 is
performed in 206 of FIG. 2A and/or 216 of FIG. 2B. In some
embodiments, the process of FIG. 4 is performed periodically. In
some embodiments, the process of FIG. 4 is performed dynamically.
For example, the process of FIG. 4 is performed when a new current
state is identified.
[0096] At 402, identified states are correlated with corresponding
controllable device properties. For example, a state that has been
identified as being the actual state of a subject in 312 of FIG. 3
is correlated with a corresponding status, a configuration, a
functional state, a property, a parameter, and/or any other data of
a controllable device of devices 102 of FIG. 1A. In some
embodiments, the identified states are correlated with
corresponding controllable device properties by analyzing a stored
history of identified states and controllable device properties to
identify controllable device properties that correspond with
identified states. For example, corresponding pairings between an
identified state (e.g., state vector) and a corresponding
controllable property (e.g., status, configurations, functional
states, parameters, and/or any other data) of a controllable device
are determined. In some embodiments, correlating the identified
states with corresponding controllable device properties includes
performing machine learning to discover correlations. For example,
statistical and/or deep learning techniques are utilized to
discover temporal correlations between identified states and
controllable device properties. In some embodiments, the identified
states include state vectors that may include one or more of the
following: a time value, a weather forecast, a date value, and
other data associated with time and/or environment conditions. In
some embodiments, a historical probability that an identified state
corresponds to a specific controllable device property is
determined.
[0097] At 404, identified states are correlated with clusters of
one or more identified states. In some embodiments, similar
identified states (e.g., state value within a range) are clustered
together and correlated with a controllable device state. For
example, identified device states that are associated with physical
locations that are close in range with one another are clustered
together. This cluster of states is correlated together with one or
more corresponding controllable device properties. In some
embodiments, a cluster probability that a cluster of one or more
identified states corresponds to the same controllable device
property is determined. The cluster probability may identify a
historical probability that any identified state included in the
cluster corresponds to the controllable device property. In some
embodiments, the cluster probability is determined by at least
multiplying together individual probabilities of each identified
state (e.g., probability determined in 320 of FIG. 3) of the
cluster. In some embodiments, the clusters are identified using an
environment model (e.g., graph model shown in FIGS. 9A-9D) and/or
context information about one or more associated component regions.
For example, context information (e.g., information of nodes/edges
of a graph model of the environment) that corresponds to the
identified states is identified and the cluster is identified based
on the context information (e.g., the proximity within the graph
model).
[0098] At 406, in the event a historical probability meets a
threshold, an associated automation rule is created. For example,
if a historical probability determined in 402 and/or a cluster
probability determined in 404 is greater than a threshold value
(e.g., 80%), a corresponding automation rule is stored in a rule
database. In some embodiments, the automation rule identifies that
if an identified state (e.g., included in the cluster of identified
states) is detected, the corresponding controllable property
setting is to be recreated/implemented (e.g., property of
corresponding controllable device(s) modified to be the rule
specified in controllable device property setting). In some
embodiments, the automation rule is updated periodically. For
example, the automation rule is associated with an expiration time
and the rule is to be renewed or deleted upon expiration. In some
embodiments, creating the associated automation rule includes
setting or modifying one or more optimization evaluation values
identifying a relationship between an identified state and a
corresponding controllable device property. For example, a measure
of user desirability of the corresponding particular controllable
device property setting given the identified state is determined
and saved for use when utilizing an optimization function to
automatically determine and control the controllable device.
[0099] FIG. 5 is a flowchart illustrating an embodiment of a
process for determining a selected output result. The process of
FIG. 5 may be at least in part implemented on any device of devices
102, user device 108, hub 104, and/or server 106 of FIG. 1A. In
some embodiments, at least a portion of the process of FIG. 5 is
included in 206 of FIG. 2A, 218 of FIG. 2B, and/or 228 of FIG. 2C.
In some embodiments, the process of FIG. 5 is performed
periodically. For example, the process of FIG. 5 is repeated at a
regular interval of time. In some embodiments, the process of FIG.
5 is performed dynamically. For example, the process of FIG. 5 is
performed when a new input or state is identified. In some
embodiments, the process of FIG. 5 is repeated for each
controllable property of each controllable device to be
automatically controlled.
[0100] At 502, a plurality of inputs that indicate states is
received. For example, each input identifies an associated state of
the input. In some embodiments, the received inputs correspond to
automatically controlling a controllable device. For example, the
states of the inputs are to be analyzed to determine a new control
setting and an action is to be potentially performed to implement
the new control setting on the controllable device. In some
embodiments, the received inputs include one or more states
identified in 214 of FIG. 2B. In some embodiments, the received
inputs include one or more states (e.g., vector of states)
identified using the process of FIG. 3. Each received input may
indicate corresponding states of one or more subjects (e.g., user
location, user activity, user preference, user type, user profile,
user category, user proficiently level, user knowledge level, user
knowledge of system feature, etc.), devices (e.g., sensor data,
controllable device setting, device configuration, etc.), and/or
environments (e.g., time, date, weather, humidity, air quality,
geographical location, etc.). In some embodiments, the received
states include previous states (e.g., history of previous states of
subjects, devices, sensors, etc.). In some embodiments, the
received states includes a state identifying a previous user
provided indication received in response to an inquiry or
indication of a user interactive qualification. In some
embodiments, one or more of the inputs that indicate the states
were determined by analyzing received sensor information. For
example, sensor information from one or more devices of devices 102
of FIG. 1A and/or sensors 124 of FIG. 1B is received and utilized
to determine the states. In some embodiments, at least one of the
plurality of inputs indicates a scene mode. For example, various
scene modes (e.g., breakfast scene, sleeping scene, reading scene,
movie scene, away mode, etc.) define a desired group of
controllable device settings for a particular activity or status of
one or more users. The scene mode may be manually specified by a
user and/or automatically determined.
[0101] In some embodiments, at least one of the plurality of inputs
indicates a history of: states, outputs, output types, settings,
and/or configurations. For example, the input identifies one or
more previously determined outputs identifying controllable device
property settings. In some embodiments, one or more of the inputs
states are associated with a measure of confidence that the
corresponding state is the correct state. For example, it may be
difficult to determine the exact correct state based on received
sensor information and candidate states are identified along with
associated confidence score values.
[0102] In some embodiments, receiving the inputs includes selecting
inputs to be utilized among a larger group of possible inputs. For
example, it is determined which inputs are relevant in
automatically determining an output control setting for a property
of a controllable device. The selection of inputs to be utilized
may have been preconfigured (e.g., determined based on a predefined
configuration for a particular controllable property of a
controllable device). For example, a programmer has manually
configured which inputs are to be analyzed when automatically
determining a particular output (e.g., control setting of a
controllable device). In some embodiments, the selection of inputs
to be utilized is based at least in part on an automation rule that
was created in 406 of FIG. 4. In some embodiments, the selection of
inputs to be utilized is based at least in part on locations of
devices associated with the inputs relative to a controllable
device. For example, based on a mapping of physical locations of
devices/sensors within an installation environment (e.g.,
determined using a graph model of the environment), states of a
device that are located in the same room and/or physically near the
controllable device to be controlled are automatically selected as
inputs. In various embodiments, the selection of inputs to be
utilized is automatically determined based on machine learning.
[0103] At 504, based on the indicated states, a plurality of output
candidates automatically controlling a controllable device is
evaluated. For example, the indicated states are utilized as inputs
to one or more evaluation rules discovered in 216 and invoked in
218 of FIG. 2B to automatically control the controllable device by
at least in part identifying the output candidates and/or the
desirability of the output candidates.
[0104] In some embodiments, a plurality of output candidates is
evaluated. For example, a plurality of candidates for a
controllable property setting of a controllable device (e.g.,
candidate settings of a property of the controllable device) is
evaluated. Each output candidate is not only associated with a
particular controllable property setting, it is associated with a
particular output handling type. Example output handling types
associated with performing an action to modify a controllable
property of a controllable device include: performing the action
(e.g., action performed without being subject to user interactive
qualification), providing a warning that the action will be
performed then performing the action (e.g., with a user interactive
opportunity to cancel performing of the action), inquiring a user
whether to perform the action (e.g., only performing the action if
the user interactively confirms performing the action), determining
that the user will likely manually perform the action (e.g., not
performing action and waiting), and not performing the action.
[0105] In some embodiments, the evaluation of the plurality of
output candidates is based at least in part on automatically
detected/learned and/or user/programmer specified
preferences/rules. For example, which optimization function to
utilize and/or values (e.g., optimization evaluation/expected
values) that indicate preference/relationships between different
configuration settings are automatically selected/determined using
machine learning and/or preferences/rules (e.g., determined using
the process of FIG. 4).
[0106] In some embodiments, the plurality of output candidates is
evaluated according to an optimization function. For example, a
plurality of candidates for a controllable property setting of a
controllable device (e.g., candidate settings of a property of the
controllable device) is evaluated according to the optimization
function. The optimization function depends on the received inputs
and one or more parameters of the one or more received inputs.
Examples of the optimization function include a cost function, a
maximization function, a risk function, a value function, a
probability function, a decision function, or any other function
that can be utilized to compare the plurality of output candidates.
In some embodiments, evaluating the output candidates includes
determining a measure of expected desirability of each output
candidate (e.g., optimization expected value) for each
configuration candidate. For example, for each output candidate, a
candidate optimization expected value identifying a desirability of
the candidate output is determined. By comparing the optimization
expected values using the optimization function, one of the output
candidates may be selected as the optimum output to be
implemented.
[0107] In some embodiments, evaluating the output candidates
includes evaluating, for each output candidate (e.g., each output
candidate is associated with a particular output handling type and
particular configuration setting candidate for a particular
controllable property of the controllable device), a measure of
expected desirability of the output candidate for each of at least
a portion of the received input state. For example, the measure of
desirability varies for each different output candidate even though
the input state is the same. In some embodiments, in order to
determine the measure of desirability for a particular output
candidate, an input state component evaluation value (e.g.,
component expected value) corresponding to the output candidate is
determined for each input state and the input state component
evaluation values for the output candidate are summed/combined
(e.g., averaged) to determine the overall measure of desirability
for the output candidate (e.g., total expected value). In other
words, component values corresponding to each input state are
summed/combined (e.g., weighted averaged) for each output candidate
to determine the expected value for each output candidate. In one
example, a configuration setting candidate for a motorized curtain
device is a curtain open setting and a curtain closed setting. The
output handling types of this example include automatically
performing an action to place the device in the desired setting
(e.g., open or closed curtain), warning a user prior to
automatically performing the action, inquiring a user whether to
perform the action, and waiting for a user to manually perform the
action. In this example, eight total expected values are determined
(e.g., four output handling types multiplied by two configuration
setting candidates=eight output candidates). For each of these
total expected values, component expected values for each input
state with respect to the output candidate were determined (e.g.,
using discovered automation rules) and combined (e.g., using an
optimization function) to determine the total expected value.
[0108] In some embodiments, rather than evaluating every output
handling type for every configuration setting candidate, a desired
configuration setting candidate is first chosen among all
configuration setting candidates, and then output handling types
for the desired configuration setting candidate are evaluated for
the desired configuration setting candidate and not for not
selected configuration setting candidates. For example, a total
overall measure of desirability for each configuration setting
candidate is determined (e.g., using automatically discovered
automation rules) by combining individual component measures of
desirability for each configuration setting candidate corresponding
to each input state for the corresponding configuration setting
candidate. By comparing the total overall measure of desirability
for the different configuration setting candidates (e.g., using an
optimization function), the desired configuration setting candidate
is chosen. Then, given the desired configuration setting candidate,
different output handling types are evaluated by determining total
expected values for each of the different output handling types for
the desired configuration setting candidate. For each of these
total expected values, component expected values for each input
state with respect to the corresponding output handling type are
determined (e.g., using discovered automation rules) and combined
(e.g., using an optimization function) to determine the total
expected value for the corresponding output handling type. This
effectively narrows the output candidates to the different output
handling types of the selected desired configuration setting
candidate.
[0109] In some embodiments, the measures of desirability (e.g.,
expected values) may take into consideration potential
benefit/risk/cost of choosing an associated output candidate given
the input states, potential harm of not choosing the associated
output candidate given the input states, a history of past output
candidates selected, annoyance to a user for choosing the
associated option, etc. among other factors.
[0110] In one example, when determining the expected value of an
output candidate associated with output handling of interactively
inquiring to a user, whether to perform an action to implement the
associated configuration setting, the expected value of this output
candidate takes into consideration the annoyance of the inquiry to
a user in light of a detected state of the user (e.g., high
annoyance to user if user is in sleeping state). In another
example, when determining the expected value of an output candidate
associated with output handling of waiting for a user to manually
perform the action, the expected value associated with this output
candidate takes into consideration the amount of effort required by
the user to manually perform the action (e.g., low effort if user
is physically near a physical switch and user typically manually
operates the switch).
[0111] In another example, when determining the expected value of
an output candidate associated with output handling of informing a
user with a warning prior to performing the action to provide the
user an opportunity to interactively prevent triggering of the
action, the associated expected value of this output candidate
takes into consideration severity of the action (e.g., highly
consequential actions such as opening bedroom curtains at night
should not be automatically performed unless input states can be
determined with high confidence) and previously selected outputs
(e.g., if a user has not responded to previous requests to provide
a response on whether to perform the action, it is more likely that
the action is performed after a warning).
[0112] In another example, when determining the expected value of
an output candidate associated with performing the action without
being subject to a user interactive qualification, the associated
expected value of this output candidate takes into consideration
whether a user has previously indicated a desire to perform
actions. For example, a user may indicate a desire to perform any
action to increase comfort/satisfaction of the user (e.g., user
provides a voice command to "make me happy"), and it is more likely
that the action will be performed without qualification (e.g.,
without warning or user confirmation requirement) given the user's
awareness and desire for the system to perform actions to increase
user satisfaction. In some embodiments, the user indication of the
desire to perform any action to increase comfort/satisfaction of
the user is associated with a detected user command that is
ambiguous or not understood by the system. For example, even if a
user command is not fully understood, it indicates a desire by the
user to perform one or more actions to increase comfort and
satisfaction of the user.
[0113] In some embodiments, one or more component evaluation values
(e.g., component expected values) are dynamically determined. For
example, rather than using a predetermined static mapping from the
output candidate and input state to a component evaluation value,
the component evaluation value may vary according to one or more
other variables. In one example, a component evaluation value
decays over time from a time of a specified event according to a
decay function parameterized by a parameter that utilizes a time
value as an input. When taking into account a history of previous
states in the input states, the decay of the component evaluation
value may be utilized to emphasize effects of recent historical
states while deemphasizing older historical states. In some
embodiments, one or more component evaluation values are based on
an amount of time passed from a previous output event. For example,
there exists a limit on a total number of user interactive
inquiries provided to a user within a specified recent time window
and one or more component evaluation values become larger or
smaller depending on whether this limit has been reached.
[0114] In some embodiments, rather than only evaluating each
individual input state separately for a particular output
candidate, a cluster of input states is evaluated together as a
group for a particular output candidate. For example, certain input
states are determined to be correlated together (e.g., based on an
environment model and/or context information about one or more
associated component regions) and for an output candidate, a
component optimization evaluation value is determined for the
cluster of input states (e.g., based on an environment model and/or
context information about one or more associated component regions)
as a single entity rather than separately for individual input
states.
[0115] At 506, a selected output candidate is selected among the
output candidates based on the evaluation. For example, an output
that optimizes the optimization function is selected among the
output candidates. In some embodiments, the selected output
candidate specifies a selected configuration setting of a property
of the controllable device. In some embodiments, selecting the
output includes selecting an output candidate based on a comparison
of the output candidate optimization evaluation values (e.g.,
compare optimization evaluation values using the optimization
function). For example, a cost value (e.g., expectation value) has
been determined for each output candidate and a cost minimization
function is utilized to select the output candidate with the
minimal cost value.
[0116] In various embodiments, by utilizing the optimization
function, conflicts between automation correlations/policies/rules
are resolved. For example, two or more automation
correlations/policies/rules are to be activated because the
received input state triggers these automation
correlations/policies/rules, but they may specify conflicting
controllable device settings that cannot be all implemented at the
same time (e.g., one rule specifies an "on" configuration setting
while another rule specifies an "off" configuration setting). By
comparing various associated optimization values associated with
each possible configuration setting, the most optimal configuration
setting given the input states may be determined.
[0117] At 508, the selected output candidate is implemented. In
some embodiments, an action associated with the selected output
candidate is performed, if applicable. For example, if a
controllable device configuration setting of the selected output
candidate is a change from a current setting, the action is
performed by providing an instruction to the controllable device to
change to the controllable device configuration setting of the
selected output candidate, if applicable, based on the output
handling type of the selected output candidate. Another example of
the action includes sending an alert/notification to a remote user
based on information captured using one or more sensor devices.
[0118] In some embodiments, the action associated with the selected
output candidate is to be performed at a time in the future (e.g.,
action to turn on coffee maker 9 am tomorrow). For example,
performing the action includes scheduling one or more instructions
provided or to be provided to the controllable device to change a
setting of the controllable device at a specified time in the
future. In some embodiments, the action associated with the
selected output candidate is to be repeated (e.g., repeated at a
specified interval). For example, performing the action includes
scheduling one or more instructions provided or to be provided to
the controllable device to change a setting of the controllable
device periodically (e.g., open curtains every weekday morning at 9
am). In some embodiments, a controllable device configuration
setting of the selected output candidate indicates a configuration
setting that affects a future state or behavior of a controllable
device. For example, the controllable device configuration setting
of the selected output candidate indicates a desired brightness
intensity whenever a lightbulb is switched on in the future. In
some embodiments, the action and/or the selected output candidate
are associated with controlling a plurality of configuration
settings for a plurality of controllable devices. For example, the
action triggers instructions to be sent to a plurality of different
controllable device to change to their controllable device
configuration settings.
[0119] Example output handling types associated with performing an
action include: performing the action (e.g., action performed
without being subject to user interactive qualification), providing
a warning that the action will be performed then performing the
action (e.g., with a user interactive opportunity to cancel
performing of the action), inquiring a user whether to perform the
action (e.g., only performing the action if the user interactively
confirms performing the action), determining that the user will
likely manually perform the action (e.g., not performing action and
waiting), and not performing the action. Depending on the output
handling type associated with the selected output candidate,
implementation of the selected output candidate is handled
differently. An example of the different types of processes is
shown in FIG. 7.
[0120] At 510, feedback about the output is received. For example,
in the event the output result resulted in an undesirable output
result (e.g., configuration setting of the controllable device), a
user may specifically override the configuration setting by
explicitly changing the configuration setting (e.g., adjust control
switch of the network connected controllable device). In some
embodiments, a user may confirm via a user indication whether a
change in output configuration settings was desired. For example,
using a user device, a user may specify whether a recent output
resulted in an automation behavior desired by the user. In some
embodiments, a lack of an action by a user is a confirmation
feedback that the output was not undesirable. For example, a
relationship between an input state and the selected output is
assumed to be more acceptable by a user the longer and/or more
often the user has not provided feedback or modified the
configuration setting given the same input state.
[0121] At 512, one or more reference optimization evaluation values
are updated, if applicable, based at least in part on the received
feedback. For example, one or more reference values (e.g.,
evaluation metric values to be referenced for future evaluations in
determining an expected value) identifying an association between
an input state and a particular output candidate (e.g.,
configuration setting of the controllable device) are updated using
the received feedback. For example, a positive indication from a
user that the output was desired reduces an associated optimization
evaluation value and a negative indication from the user that the
output was not desired increases the associated optimization
evaluation value.
[0122] FIG. 6 is a flowchart illustrating an embodiment of a
process for automatically determining an output candidate of a
controllable property of a network connected controllable device.
The process of FIG. 6 may be at least in part implemented on any
device of devices 102, user device 108, hub 104, and/or server 106
of FIG. 1A. In some embodiments, at least a portion of the process
of FIG. 6 is performed in 206 of FIG. 2A and/or 218 of FIG. 2B. In
some embodiments, at least a portion of the process of FIG. 6 is
included in 504 and/or 506 of FIG. 5.
[0123] At 602, a candidate configuration setting of a controllable
device property to be automatically controlled is selected for
evaluation. For example, in order to automatically determine the
configuration setting of the property of the controllable device,
each candidate configuration setting for the property is to be
evaluated based on states (e.g., states of users, sensors, devices,
environment, configuration modes/groupings, etc. received in 502)
of a network connected device installation environment (e.g.,
apartment unit) and each candidate configuration setting is
selected iteratively for evaluation. In some embodiments, selecting
the next candidate configuration setting includes selecting a next
item in a list of candidate configuration settings for a property
of the controllable networked device. In some embodiments, the list
of candidate configuration settings includes all possible
configuration settings of the property of the controllable
networked device. Examples of the property of the controllable
networked device include a switch state, a value setting, an
intensity level, whether a notification should be sent, and any
other controllable property. In various embodiments, the
controllable network connected device is included in devices 102 of
FIG. 1A or devices 126 of FIG. 1B.
[0124] The list of candidate configuration settings for the
property may be preconfigured. For example, a subset from all
possible configuration settings for the property has been selected
to be included in the list. In some embodiments, the list of
candidate configuration settings for the property to be iterated is
dynamically determined. For example, candidate configuration
setting values are bounded by one or more limits that dynamically
vary based on one or more variables. For example, the maximum
intensity output of a light bulb is limited based on a time of day
(e.g., bulb is allowed to output the entire range of intensities
during daylight hours but is bounded to only output 50% of maximum
intensity during night hours) and the list of candidate
configuration settings only includes intensity values within the
time-based limit.
[0125] At 604, for each applicable input state, a component
evaluation value for the selected candidate configuration setting
is determined. For example, for each applicable input state of
input states received in 502 of FIG. 5 to automatically control the
property of the controllable networked device, the component
evaluation value for the selected candidate configuration setting
is determined.
[0126] In some embodiments, there exists a data structure that
stores for each different pair of input state and candidate output
(e.g., for a particular configuration setting), a corresponding
component evaluation value that quantifies the relationship between
the input state and the candidate output. For example, the
component evaluation value identifies the magnitude of the
desirability of the selected candidate configuration setting for a
particular input state. In some embodiments, the component
evaluation value is based at least in part on automatically
detected/learned and/or user/programmer specified
preferences/rules. For example, which optimization function to
utilize and/or component optimization evaluation/expected values
that indicate preferences/relationships between a particular
configuration setting are automatically selected/determined using
machine learning and/or preferences/rules (e.g., determined using
the process of FIG. 4).
[0127] In some embodiments, the component evaluation value is
determined based on an environment model and/or context information
about one or more associated component regions. For example,
context information (e.g., properties of relevant edges between
relevant nodes of a graph model of the environment) can be used to
identify desirability of the selected candidate configuration
setting. In one example, the component evaluation value is a cost
value to be evaluated using a cost function. In some embodiments,
the component evaluation value is at least in part calculated using
a formula. In some embodiments, the evaluation value has been
automatically determined. For example, the evaluation value has
been determined using machine learning based on detected data and
user feedback. In some embodiments, the evaluation value has been
at least in part predefined. For example, a user and/or programmer
has specified the evaluation value.
[0128] In some embodiments, determining the component evaluation
value for an input state includes utilizing one or more parameters
to dynamically determine the component evaluation value. For
example, rather than using a predetermined static value, the
component evaluation value may vary according to one or more other
variables that are inputs to the parameters. In some embodiments, a
component evaluation value decays over time from a time of a
specified event according to a decay function parameterized by a
parameter that utilizes a time value as an input. For example, a
light intensity of a light bulb is to decay gradually after a light
switch has been manually turned on and the component evaluation
value associated with a full brightness light intensity setting
decays according to a parameter decay function that utilizes as its
input a time since a switch was turned on. In some embodiments, a
component evaluation value varies according to a parameter based on
a number of users detected in an environment (e.g., evaluation
value for a particular switch state and a candidate light level
configuration setting is based on the number of users in a
room).
[0129] In some embodiments, the one or more input states and/or
parameters are based on one or more previous outputs or
configuration settings. For example, it is not desirable to rapidly
change configuration settings within a short period of time (e.g.,
might damage a motor, light bulb, etc.) and multiple changes to
different configuration settings in a short period of time are
discouraged by taking into consideration previous configuration
setting changes when determining the component evaluation value
(e.g., add an adjustment factor to a base component evaluation
value when many configuration changes have been implemented for the
same property of the controllable device within a short period of
time).
[0130] In some embodiments, rather than only evaluating each
applicable input state separately for a particular component
evaluation value, a cluster of input states is evaluated together
as a group for a particular component evaluation. For example,
certain input states are determined to be correlated together
(e.g., based on an environment model and/or context information
about one or more associated component regions) and a component
optimization evaluation value is determined for the cluster of
input states as a single entity rather than separately for
individual input states.
[0131] At 606, the determined component evaluation values are
combined to determine a total evaluation value (e.g., total
expected value) for each output handling type of the selected
potential configuration setting. The output handling type of the
selected candidate configuration setting specifies how and whether
an action to change a device setting to the associated selected
candidate configuration setting is handled, if applicable (e.g.,
how controllable device property setting change is handled).
Example output handling types associated with performing an action
to modify a property of a controllable device include: performing
the action (e.g., action performed without being subject to user
interactive qualification), providing a warning that the action
will be performed then performing the action (e.g., with a user
interactive opportunity to cancel performing of the action),
inquiring a user whether to perform the action (e.g., only
performing the action if the user interactively confirms performing
the action), determining that the user will likely manually perform
the action (e.g., not performing action and waiting), and not
performing the action.
[0132] In some embodiments, the determined component evaluation
values for the selected candidate configuration setting are
summed/combined together. In some embodiments, determining the
total evaluation values includes determining a weighted sum of one
or more of the determined component optimization evaluation values.
In some embodiments, determining the total evaluation value
includes determining an average of one or more of the determined
component evaluation values.
[0133] In some embodiments, determining the total evaluation value
includes determining a weighted value of component evaluation
values. For example, because the exact correct current state may be
difficult to determine with complete accuracy, one or more of the
received input states are associated with one or more weight
factors (e.g., percentage value) representing a determined
likelihood that the input state is the actual correct input state.
This may allow different candidate input states for the same actual
input state to be received and evaluated independently then
weighted and combined together (e.g., weighted average) based on
the weight factors. For example, a received input indicates that
there is a 70% likelihood that a user is in a sitting state and a
30% likelihood that the user is in a standing state. Prior to being
summed with other optimization evaluation values, the determined
component evaluation value for the sitting state is multiplied by
0.7 and the determined component value for the standing state is
multiplied by 0.3. The weight value may dynamically change based on
one or more factors (e.g., based on time or other environmental
variable/state).
[0134] In some embodiments, the summed or averaged result of the
component evaluation values or weighted component evaluation values
are scaled and/or adjusted (e.g., adjustment value added or
subtracted) to determine a total evaluation value for each output
handling type of the selected potential configuration setting. For
example, a scaling or adjustment value is specific to each
different output handling type of a particular configuration
setting and varies based on one or more of the input states. For
example, for each of one or more input states, a corresponding
scaling or adjustment value specific to the particular
configuration setting and output handling type is determined and
applied (e.g., applied to component evaluation values and/or total
evaluation values) to determine the final total evaluation value
(e.g., total expected value) for each output handling type of the
selected potential configuration setting.
[0135] In some embodiments, the scaling and/or adjustment values
may take into consideration potential benefit/risk/cost of choosing
an associated output handling type for the selected potential
configuration setting, potential harm of not choosing the
associated output handling type for the selected potential
configuration setting, a history of past output handling types
selected, annoyance to a user for choosing the associated output
handling type, etc. among other factors.
[0136] In one example, when determining the scaling and/or
adjustment value associated with an output handling type of
interactively inquiring a user whether to perform an action to
implement the associated configuration setting, the scaling and/or
adjustment value of this output handling type takes into
consideration the annoyance of the inquiry to a user in light of a
detected state of the user (e.g., high annoyance to user if user is
in sleeping state). In another example, when determining the
scaling and/or adjustment value associated with an output handling
type corresponding to waiting for a user to manually perform the
action, the scaling and/or adjustment value associated with this
output handling type takes into consideration the amount of effort
required by the user to manually perform the action (e.g., low
effort if user is physically near a physical switch and user
typically manually operates the switch).
[0137] In another example, when determining the scaling and/or
adjustment value associated with an output handling type associated
corresponding to informing a user with a warning prior to
performing the action, the associated scaling and/or adjustment
value of this output candidate takes into consideration severity of
the action (e.g., highly consequential actions such as opening
bedroom curtains at night should not be automatically performed
unless input states can be determined with high confidence) and
previously selected outputs (e.g., if a user has not responded to
previous requests to provide a response on whether to perform the
action, it is more likely that the action is performed after a
warning).
[0138] In another example, when determining the scaling and/or
adjustment value of an output handling type associated with
performing the action without being subject to a user interactive
qualification, the associated scaling and/or adjustment value of
this output handling type takes into consideration whether a user
has previously indicated a desire to perform actions. For example,
a user may indicate a desire to perform any action to increase
comfort/satisfaction of the user (e.g., user provides a voice
command to "make me happy"), and it is more likely that the action
will be performed without qualification (e.g., without warning or
user confirmation requirement) given the user's awareness and
desire for the system to perform actions to increase user
satisfaction.
[0139] In some embodiments, it is determined whether there exists
an additional candidate configuration setting of the property of
the controllable networked device to be selected for evaluation.
For example, it is determined whether there exists a next candidate
configuration setting to be evaluated from a list of candidate
configuration settings to be evaluated. If it is determined that
there exists an additional candidate configuration setting to be
selected for evaluation, the process returns to 602.
[0140] At 608, an optimization function is utilized to select a
selected candidate configuration setting and output handling type
to be implemented.
[0141] In some embodiments, the optimization function to be
utilized is predetermined. In some embodiments, the optimization
function to be utilized is selected among a plurality of
optimization functions based on one or more factors (e.g., property
to be controlled, input states, user preference, etc.). In some
embodiments, utilizing the optimization function includes comparing
the total evaluation values (e.g., expected values) for each of the
candidate configuration settings and output handling types using
the optimization function to select the most optimal total
evaluation value. For example, the optimization function is a cost
optimization function and each of the determined total evaluation
values represents cost values, and the selected candidate
configuration setting is the candidate configuration setting with
the lowest cost value. Other examples of the optimization function
include a maximization function, a risk function, a value function,
a probability function, and a decision function, among others. In
some embodiments, the optimization function specifies that the
candidate configuration setting and output handling type with the
largest output candidate optimization evaluation value is to be
selected. In some embodiments, the optimization function specifies
that the candidate configuration setting and output handling type
with a local minimum/maximum output candidate optimization
evaluation value is to be selected. In various embodiments,
utilizing the optimization function includes identifying a
statistically relevant value among the determined total evaluation
values using a criteria defined by the optimization function and
identifying the corresponding selected candidate configuration
setting and output handling type as the setting to be implemented
on the controllable device in a manner of the output handling
type.
[0142] In some embodiments, the selected potential configuration
setting and output handling type are provided for implementation in
508 of FIG. 5 and the process returns to 510 of FIG. 5. In some
embodiments, the selected candidate configuration setting is
provided to the controllable device only if the selected candidate
configuration setting indicates a different configuration setting
than a current configuration setting of the controllable device
property being automatically controlled.
[0143] In one illustrative example of the process of FIG. 6, in
order to determine whether to automatically open the living room
curtains that are motorized by a controllable network connected
device motor, the candidate configuration settings of a curtain
open setting and a curtain close setting are evaluated separately.
The received input state indicates that there is a 70% chance that
a user is in the living room (e.g., 70% chance of living room
state) and a 30% chance that the user is in the bedroom (e.g., 30%
chance of bedroom state). When the curtain open setting is
evaluated, component evaluation values of -1 for the living room
state and -11 for the bedroom state are combined to determine a
combined evaluation value of -4 (e.g., (0.7)*(-1)+(0.3)*(-11)). For
the living room state, this combined evaluation value of -4 is
scaled/adjusted for each of three different output handling types
to identify the total evaluation value of -10 for output handling
type 1, -5 for output handling type 2, and -1 for output handling
type 3. For the bedroom state, the combined evaluation value of -4
is scaled/adjusted for each of the three different output handling
types to identify the total evaluation value of -20 for output
handling type 1, -20 for output handling type 2, and -20 for output
handling type 3. After combining the total evaluation values for
the different states, the combined total evaluation values of -13
(e.g., (0.7)(-10)+(0.3)(-20)) for output handling type 1, -9.5
(e.g., (0.7)(-5)+(0.3)(-20)) for output handling type 2, and -6.7
(e.g., (0.7)(-1)+(0.3)(-20)) for output handling type 3 result for
the curtain open setting. Applying a similar process to the curtain
close setting, combined example total evaluation values of -21 for
output handling type 1, -12 for output handling type 2, and -10 for
output handling type 3 result for the curtain close setting. A cost
optimization function to be utilized specifies that the
configuration setting candidate with the lowest total evaluation
value is to be selected and the curtain open setting with output
handling type 3 (i.e., -6.7 is the lowest value) is selected as the
selected output candidate to be implemented because it had the
lowest total evaluation value.
[0144] FIG. 7 is a flowchart illustrating an embodiment of a
process for implementing a selected output. The process of FIG. 7
may be at least in part implemented on any device of devices 102,
user device 108, hub 104, and/or server 106 of FIG. 1A. In some
embodiments, at least a portion of the process of FIG. 7 is
included in 508 of FIG. 5.
[0145] At 702, a selected output is received. In some embodiments,
the selected output is the output candidate selected in 506 of FIG.
5.
[0146] At 704, it is determined whether an output handling type of
the selected output corresponds to performing an action without
being subject to user interactive qualification. For example, it is
determined whether the output handling type corresponds to
performing the action without providing a user warning or requiring
user confirmation to perform the action. If at 704, it is
determined that the output handling type of the selected output
does correspond to performing the action without user interactive
qualification, at 710, the action is performed, if applicable.
[0147] For example, at 710, if a controllable device configuration
setting of the selected output candidate is a change from a
corresponding current setting of a property of a controllable
device, the action is performed by providing an instruction to the
controllable device to change to the specified device configuration
setting of the selected output candidate. In some embodiments, the
action is not associated with a persistent property state of the
controllable device and the action is always performed in 710. For
example, performing the action includes sending an
alert/notification to a remote user based on information captured
using one or more sensor devices. In some embodiments, performing
the action includes performing the action gradually in partial
steps over a period of time. For example, if the action is to
change a volume intensity of a speaker to a new target intensity
level, the volume intensity of the speaker is changed gradually
over time towards the new target intensity level. While gradually
performing the action, an indication may be received from a user to
stop/pause the gradual performance of the action. For example, a
user indicates to stop/pause the volume change during a gradual
increase of audio volume at a level desired by the user. If the
indication is received to stop/pause the gradual performance of the
action, the action is stopped/paused and the current setting at the
point of the stop indication is maintained. In some embodiments, if
the indication is received to reverse the action, the action is
action is reserved (e.g., provide an instruction to the
controllable device to revert back to a previous device
configuration setting).
[0148] If at 704, it is determined that the output handling type of
the selected output does not correspond to performing the action
without being subject to user interactive qualification, at 706, it
is determined whether the output handling type of the selected
output corresponds to providing a warning indication that the
action will be performed. If at 706 it is determined that the
output handling type of the selected output does correspond to
providing the warning indication that the action will be performed,
at 712, the warning indication is provided. For example, the
warning indication provides a hint to a user that action will be
performed.
[0149] The warning indication may include an interactive visual
warning (e.g., turn on warning lights, flash lights, change light
intensity, change light color, an electronic display warning,
etc.), an auditory warning (e.g., voice warning via speakers, audio
tone, change audio/volume intensity etc.), a warning message (e.g.,
email, text message, chat message, application notification, etc.),
a vibration warning, a mechanical warning (e.g., activate motor,
open or close window shade, etc.) and any other type of warning
notification. In some embodiments, the warning notification may
gradually increase in intensity (e.g., light intensity, color,
volume, message severity, etc.) over time as it gets closer to
performing the action. In some embodiments, providing the warning
indication includes gradually or partially performing the action.
For example, if the action is to change a light intensity of a
lightbulb to a new target intensity level, the light intensity of
the lightbulb is changed partially towards the new target intensity
level. In another example, if the action is to fully open a
motorized curtain, the curtain is opened slightly to provide an
indication that the curtain will be fully opening. In some
embodiments, the warning indication provides a time value
indicating when the action will be performed. For example, a
message is provided to a user that lights will automatically turn
on in three minutes unless otherwise instructed.
[0150] At 714, it is determined whether a cancelation indication
was received during a waiting period. For example, after the
waiting period has passed the process proceeds to 710. During the
waiting period, a user is able to provide an indication to cancel
or stop performing of the action. The cancelation indication may be
provided via voice command, movement gesture (e.g., detected by a
camera), a physical switch activation, a computer input, an
application input, a digital message, and any other user input. If
the cancelation indication was received in 714, it is determined
not to perform the action in 720. If the cancelation indication was
not received in 714, the process proceeds to 710. In some
embodiments, in the event providing the warning indication in 712
included gradually or partially performing the action, the received
cancelation indication may indicate to undo the gradually or
partially performed action. If the cancelation indication indicates
to undo the gradually or partially performed action, at 720, the
gradually or partially performed action is reserved (e.g., provide
an instruction to the controllable device to revert back to a
previous device configuration setting). In some embodiments,
although a cancelation indication is not received in 714, an
indication is received to stop a performance of the action being
performed in gradual steps prior to complete completion of the
action. If the indication is received to stop the gradual
performance of the action before the end of the waiting period
(e.g., the waiting period is the amount of time required to
completely complete the gradual performance of the action), the
gradual performance of the action is paused and the current
controllable device setting at the point of the indication is
maintained in 710.
[0151] If at 706 it is determined that the output handling type of
the selected output does not correspond to providing the warning
indication that the action will be performed, at 708, it is
determined whether the output handling type of the selected output
corresponds to inquiring a user whether to perform the action. If
at 708 it is determined that the output handling type of the
selected output does correspond to inquiring whether to perform the
action, at 716, a user interactive inquiry is sent to the user to
inquire whether to perform the action. The inquiry may be sent via
a visual inquiry (e.g., turn on lights, flash lights, change light
color, inquiry on an electronic display, etc.), an auditory inquiry
(e.g., voice question via speakers, audio tone, etc.), an inquiry
message (e.g., email, text message, chat message, application
notification, etc.), vibration inquiry, and any other type of user
inquiry. In some embodiments, the inquiry may gradually increase in
intensity (e.g., light intensity, color, volume, message severity,
etc.) over time if the user does not provide a response.
[0152] After 716, at 718, it is determined whether a positive
indication to perform the action is received. The user may provide
the positive indication via voice command, movement gesture (e.g.,
detected by a camera), a physical switch activation, a computer
input, an application input, and any other user input. If at 718
the positive indication is received, the process proceeds to 710.
If at 718 the positive indication is not received (e.g., negative
indication or no indication within a specified time period), the
process proceeds to 720.
[0153] If at 708 it is determined that the output handling type of
the selected output does not correspond to inquiring whether to
perform the action, at 709, it is determined whether the output
handling type of the selected output corresponds to waiting for a
user to manually perform the action. If at 709 it is determined
that the output handling type of the selected output does
correspond to waiting for a user to manually perform the action,
the process proceeds to 720. In some embodiments, if at 709 it is
determined that the output handling type of the selected output
does correspond to waiting for a user to manually perform the
action, the process waits for a specified amount of time to allow
the user to manually perform the action, and if the user does not
perform the action within the specified amount of time, the process
may proceed to 710, 712 or 716 (not shown). Although four types of
output handling types have been shown in FIG. 7, additional or
other output handling types may exist in various other embodiments.
For example, if at 709 it is determined that the output handling
type of the selected output does not correspond to waiting for a
user to manually perform the action, other tests may be performed
to identify other output handling types.
[0154] In some embodiments, one or more aspects of a user response
received in any of the steps of FIG. 7 is utilized in automatically
controlling a controllable device. For example, one or more aspects
of the user response (e.g., a tone of the response, a detected user
annoyance in the response, voice prosody, content of the response,
etc.) is utilized as an input state in 502 of FIG. 5, in selecting
the selected output candidate in 506 of FIG. 5, as an input to the
optimization function utilized in 608 of FIG. 6, etc.
[0155] FIG. 8 is a flowchart illustrating an embodiment of a
process for attempting to discover an improved output. The process
of FIG. 8 may be at least in part implemented on any device of
devices 102, user device 108, hub 104, and/or server 106 of FIG.
1A. In some embodiments, at least a portion of the process of FIG.
8 is performed in 206 of FIG. 2A, 218 of FIG. 2B, 506 of FIG. 5,
and/or 608 of FIG. 6. In some embodiments, at least a portion of
the process of FIG. 8 is repeated each time an output to control a
controllable device is to be automatically determined.
[0156] At 802, a chance threshold of outputting a test output is
identified. For example, when utilizing an optimization function to
select the most optimal output (e.g., using the process of FIGS. 5
and/or 6), the selected output is selected based on determined
candidate optimization evaluation values of the output candidates.
However, if the candidate optimization evaluation values do not
accurately capture the desirability of an output candidate, the
selected output candidate may be suboptimal. Although the
optimization evaluation values mapping to results and inputs may be
adjusted and improved based on feedback (e.g., user manually
changes the suboptimal setting to the optimal setting and machine
learning is utilized to learn from this change), the user may not
provide valuable feedback to improve output results if the output
is satisfactory despite being not optimal. For example, although
the optimal fan speed setting is low speed, a user may not provide
feedback to indicate this optimal speed if a current different fan
setting (e.g., high speed) is satisfactory.
[0157] In some embodiments, in an attempt to discover a more
optimal output (e.g., output configuration setting of a
controllable device), an output that may not be the most optimal
according to an optimization is provided as the output in limited
instances to discover potentially more optimal outputs. For
example, rather than using the most optimal output selected by an
optimization function, a random output and/or second most optimal
output is provided as the output to discover whether this output is
potentially a more optimal output. In some embodiments, the chance
threshold identifies the target probability that an output that was
not selected as the most optimal output is to be provided as the
output. In some embodiments, the chance threshold is dynamically
determined. For example, by dynamically varying the chance
threshold, a beta tester could be exposed to more test outputs that
may not be most optimal while a commercial user that desires
reliability is exposed to relatively less test outputs that may not
be most optimal. In another example, rather than continually
exposing a user to successive test outputs that may not be most
optimal, the number of test outputs provided within a time interval
is limited by varying the learning selection chance threshold.
[0158] In some embodiments, identifying the chance threshold
includes dynamically selecting the chance threshold value based at
least in part on one or more of the following: one or more input
states (e.g., received in 502 of FIG. 5), a time since a last test
output, a profile of a user, a correctness result/feedback of the
last test output, a current time/date, and a type of property
and/or controllable device to be controlled, etc.
[0159] At 804, it is determined whether a selection criteria based
on the identified chance threshold has been met. For example, it is
determined whether a test output that is not an optimization
function selected output should be outputted based on a randomized
selection criteria evaluated using the chance threshold. In some
embodiments, in order to determine whether a test output is to be
provided, a random value is selected and it is determined whether
the random value is within a range bounded by a limit of the chance
threshold. (e.g., if the random value is less than the chance
threshold value, the chance threshold is met or otherwise it is not
met).
[0160] If at 804 it is determined that the selection criteria has
not been met, at 806 a selected output selected to be optimal using
an optimization function is outputted. For example, an output
selected by an optimization function to be optimal (e.g., in 506 of
FIG. 5, 608 of FIG. 6, etc.) is provided as an output to control a
controllable device.
[0161] If at 804 it is determined that the selection criteria has
been met, at 808 a test output is outputted. For example, among
potential output candidates (e.g., candidate configuration setting
of a property of a controllable device to be controlled) that have
been identified (e.g., identified in 602 of FIG. 6), one of the
output candidates is randomly selected and outputted. The output
candidates that are eligible to be randomly selected to be
outputted may be limited based on an analysis performed using an
optimization function. For example, only the top 50% of output
candidates as ranked by the optimization function are eligible to
be selected for output as the test output. In another example, a
second most optimal output as determined based on the optimization
function is provided as the test output.
[0162] At 810, feedback about the output is received. For example,
in the event the output resulted in an undesirable configuration
setting of the property of the controllable device, a user may
specifically override the configuration setting by explicitly
changing the configuration setting (e.g., adjust control switch of
the device). In some embodiments, a user may confirm via a user
indication whether the output configuration setting is a desired
configuration setting. For example, using a user device, a user may
specify whether a recent output resulted in an automation behavior
desired by the user (e.g., optimization component value(s) modified
based on indication). In some embodiments, a lack of an action by a
user is a confirmation feedback that the output was not
undesirable. For example, a relationship between an input state and
the output is assumed to be more acceptable by a user the longer
and/or more often the user has not provided feedback or modified
the configuration setting given the same input state.
[0163] At 812, one or more reference optimization evaluation values
are updated, if applicable, based at least in part on the received
feedback. For example, one or more reference values (e.g., values
to be referenced for future evaluations) identifying an association
between an input state and a particular output (e.g., configuration
setting of the controllable device) are updated using the received
feedback. For example, a positive indication from a user that the
output was desired reduces an associated optimization evaluation
value and a negative indication from the user that the output was
not desired increases the associated optimization evaluation
value.
[0164] FIGS. 9A-9D are diagrams illustrating various example levels
of a hierarchical deployment environment model. For example, the
environment model is stored in any device of devices 102, user
device 108, hub 104, and/or server 106 of FIG. 1A for use in rule
discovery and implementation as well as device control. In some
embodiments, the model of a deployment environment is a model of
devices/sensors and the physical environment and their connections
in graph data format. The devices/sensors and the physical
environment locations (e.g., component regions) are represented as
nodes in the shown graph model and connections (e.g., context
information) between the nodes are represented as edges. A device
node in the graph model does not necessarily have to be a "smart"
or network "connected" or an "IoT" device. For example, a normal
"dumb" light bulb is represented as a node in the graph model
because it affects lighting of a room/environment represented in
the graph model. Other nodes in the graph model may be able to
sense and/or control this light bulb. For example, the light bulb
node may be connected to another device node in the model that can
sense the state of the bulb (e.g., light sensor node) and/or has
control over the light bulb (e.g., smart network connected light
switch). The graph model of the examples is hierarchical and a node
at one level may represent one or more nodes and edges in a lower
level that inherit properties (e.g., context information) of the
higher level node. Edges may exist across levels and between nodes
of different hierarchical levels/groupings. Each node (e.g., each
component region) may be associated with one or more properties
(e.g., context information) that each identify a description, a
configuration, a setting, an identification, and/or any other
property about the node. Each edge may be associated with one or
more properties (e.g., context information) that identify a
relationship between the nodes connected by the edge. For example,
relationships that identify a physical location/property (e.g.,
physical connection type edge), a functional property (e.g.,
functional connection type edge), a behavioral property (e.g.,
behavioral activity connection type edge), and/or any other
relationship property are identified as a property of an edge. The
graph model levels shown in FIGS. 9A to 9D are merely examples and
have been simplified to illustrate the examples clearly. Not all
nodes and edges have been shown and additional nodes and edges may
exist. Node properties (e.g., context information) may include
physical properties such as lighting, temperature, humidity,
geographic locations (e.g., the rooms/areas in the home),
dimensions, vibrations, pressure, locations, sunlight direction,
locations of the devices, sound levels, etc. For example, the
functional property of a curtain node is a curtain open function
(and affects lighting and heat) and its effects on the environment
are modeled through the edges in the graph that identify lighting
and temperature relationships to the function.
[0165] Edges may define relationships between the nodes. Physical
and functional edge properties (e.g., context information) may
include the lighting relationship between two location nodes (e.g.,
light in one area affects lighting in a neighboring room), the
temperature relationship between two nodes (e.g., temperature
change in one area affects temperature change in a neighboring
area, less if there is an insulating wall in the middle), the
relationship between geographic locations (e.g., layout of the
home, living room is walkable from dining room, but not directly to
bathroom as one needs to pass through a corridor node), sound
relationships (e.g., music or voice in one area will affect sound
in other areas), the relationship between two devices (e.g., a
switch may control three bulbs, or a location node is visible from
two cameras), etc. For example, when a curtain (a device node) is
opened, its lighting effects are modeled by a light property edge
that identifies a change in light level of a living room node from
light entering from a node representing an outside environment.
Behavioral edge properties may be associated with activities (e.g.,
a person may be working in the dining room but his "usage" of the
area would include the adjoining living room). For example, a
person may be cooking and the kitchen and dining room would connect
for the cooking activity. In another example, a child's cellphone
would not have any functional effect on the parent's bedroom and
only the authorized owners could have access to the camera feeds in
certain locations.
[0166] FIG. 9A is a diagram illustrating an example neighborhood
level of a hierarchical graph model of a connected device
environment. As shown in the graph model, not only can nodes
represent locations/physical buildings, they can represent shared
resources and devices of the devices shared by locations/buildings
of the level. Graph model level 902 shows nodes 906 and 908 that
represent buildings of a neighborhood. Node 904 represents an
automatic gate for the neighborhood of the buildings and node 910
represents a server managing device automation for the neighborhood
(e.g., server 106 of FIG. 1A). Building node 906 may be associated
with properties that describe the physical location coordinate of
the building, a geographical orientation of the building, a number
of units included in the building, activities that can be performed
in association with the building, a height of the building,
capabilities of the building, etc. The connections between the
nodes (i.e., edges) shown in graph model level 902 represent
physical location connections/relationships between the nodes. For
example, an edge between the nodes may represent an existence of a
physical passage, location association, and/or wire connection.
Each edge may be associated with one or more properties. For
example, an edge may be associated with a property value that
identifies a strength, distance, and/or time value associated with
a distance/location between connected nodes. Although graph 902
only shows the physical connection type edges, other types of edges
not shown may exist as shown by examples in other figures. For
example, a functional type edge identifies a functional connection
(e.g., network connection) between nodes (e.g., the graph model
identifies a detected connection speed between buildings/servers
and the graph model may be utilized to identify the fastest network
path between the nodes). In another example, a behavioral type edge
identifies a connection that exists for certain types of behavioral
activities (e.g., connection based on people, activities,
ownership, authorization, privacy, etc.). Graph model level 902
shows one level of the hierarchical graph model and a higher level
of the hierarchical graph model may exist. For example, graph model
level 902 represents one neighborhood node of a plurality of
neighborhood nodes connected together in a higher level of the
graph model.
[0167] FIG. 9B is a diagram illustrating an example building level
of a hierarchical graph model of a connected device environment.
Graph model level 912 shown in FIG. 9B represents a lower level of
the level shown in FIG. 9A. Specifically, level 912 represents sub
nodes and edges of building 906 of FIG. 9A. Nodes 914, 916, 918,
and 920 represent apartment units of building 906 of FIG. 9A. The
device management server for all of these apartments of the
building is represented by node 922. The connections between the
nodes shown in graph model level 912 represent physical location
connections/relationships between the nodes. Although graph level
912 only shows the physical connection type edges, other types of
edges not shown may exist.
[0168] FIG. 9C is a diagram illustrating an example unit level of a
hierarchical graph model of a connected device environment. Graph
model level 924 shown in FIG. 9C represents a lower level of the
graph level shown in FIG. 9B. Specifically, level 924 represents
sub nodes of apartment 918 of FIG. 9B. Not only can nodes represent
buildings, units, and devices, they can also represent physical
rooms and physical regions. Locations within an apartment unit are
represented by nodes 926-936. Water heater node 925 represents a
water heater device of the unit. The connections between the nodes
shown in graph model level 924 represent physical location
connections/relationships between the nodes. Graph level 924 shows
the physical connection type edges as solid line connections. Graph
level 924 also shows example behavioral type edges as dotted line
connections between nodes. The dotted line between kitchen 926 and
dining room 930 represents a behavioral connection between the
rooms/areas for a particular behavioral activity. This behavioral
edge has been associated with a property that identifies that for
the dining activity, the privacy between the kitchen and the dining
room should be set at a medium privacy setting of value 0.5. This
privacy property value may be utilized to adjust the associated
devices to maintain a medium level of privacy when it is detected
that a dining activity is being performed.
[0169] The dotted line between living room 934 and bedroom 936
represents a behavioral connection between the rooms/areas. This
behavioral edge has been associated with a property that identifies
that for a studying activity, the privacy between the living room
and the bedroom should be set at a high privacy setting of value
1.0. This privacy property value may be utilized to adjust
associated devices to maintain a high level of privacy between the
rooms (e.g., minimize noise) when it is detected that a studying
activity is being performed. Behavioral edges may relate to
activities of a person and/or animal/pet.
[0170] FIG. 9D is a diagram illustrating an example room level of a
hierarchical graph model of a connected device environment. Graph
model level 940 shown in FIG. 9D represents a lower level of the
graph level shown in FIG. 9C. Specifically, graph level 940
represents sub nodes of kitchen 926 of FIG. 9C. Not only can nodes
represent devices and unit partitions, they can also represent sub
regions of a region and items (e.g., furniture) located in the
environment. Node kitchen area 950 represents the general kitchen
location/region. Kitchen island region 952 and counter region 954
represent sub regions of kitchen area 950. Counter region 954 is
further divided into sub regions sink region 956 and stove region
958. Lighting specific to the sink region is represented by light
960 that is controlled by switch 964. A fan specific to the stove
region is represented by fan 962 that is controlled by switch 966.
A camera in the kitchen is represented by camera 942. A microphone
and speaker in the kitchen are represented by module 944. A motion
sensor in the kitchen is represented by motion sensor 946. A window
shade control device in the kitchen is represented by node 948.
[0171] The solid edge connection lines between the nodes shown in
graph model level 940 represent physical location
connections/relationships between the nodes. For example, camera
942 is connected to kitchen area 950 for being installed/mounted in
the region of the kitchen area. The dashed edge connection lines
between the nodes in graph model level 940 represent functional
connections/relationships between the connected nodes. For example,
the functional edge between camera 942 to kitchen island region 952
identifies that the image capture function of the camera is able to
capture an image of at least a region of kitchen island region 952.
A property associated with this edge may identify the exact portion
of the kitchen island region that is able to be captured by the
camera. Additionally, dashed edge connections between light 960 and
kitchen island region 952, sink region 956, and stove region 958
identify that the luminescence function of the light reaches the
connected regions. Each of these functional edges is associated
with a property that identifies a measure of intensity of the light
that reaches each connected region. The functional edge to sink
region 956 is identified with the property intensity value "1.0" to
indicate a strong luminescence, the functional edge to stove region
958 is identified with the property intensity value "0.5" to
indicate a medium luminescence, and the functional edge to kitchen
island region 952 is identified with the property intensity value
"0.2" to indicate a low luminescence. Another example of a
functional edge includes an audio/sound relationship between
nodes.
[0172] FIG. 10 is a flowchart illustrating an embodiment of a
process for initializing a model of a connected device environment.
The process of FIG. 10 may be at least in part implemented on any
device of devices 102, user device 108, hub 104, and/or server 106
of FIG. 1A. In some embodiments, the model is the graph model
stored in any device of devices 102, user device 108, hub 104,
and/or server 106 of FIG. 1A. In some embodiments, an example of
the graph model is the graph model of FIGS. 9A-9D.
[0173] At 1002, an environment model is initialized. In some
embodiments, initializing a model includes establishing a graph
model for a connected device environment to be established. For
example, a user specifies devices/sensors and their locations and
connections within an environment to build the model. In some
embodiments, initializing the model includes receiving a
specification of the model. For example, using software, a user
specifies a configuration of an environment including regions and
connected devices within the environment.
[0174] Environment models may vary across different installation
environment and different homes. However the machine learning may
be utilized to apply at least a portion of a model from one
installation environment to another installation environment. For
example, patterns in the graph model may be found and applied to
various installation environments, so that the learning from one
home can be transferred across to another home. In other words, the
machine learning is utilized to identify behaviors/processes across
similar parts of the model and thus generates common rules across
different installation environments and models.
[0175] In some embodiments, one or more templates are utilized to
initialize the model. For example, a user is provided prebuilt
graph model templates and the user is able to adjust and combine
templates as needed to specify at least a portion of the graph
model instance. In some embodiments, network connected devices are
to be deployed across very similar and/or same environments. For
example, multiple buildings/apartment units may be based on the
same layout and an installer has determined the types and specific
installation locations of network connected devices and sensors to
be installed and duplicated in each unit. Because the design has
been standardized, a model template may be generated for the
same/similar unit and applied across all same/similar units. In
another example, model templates for environment portions (e.g.,
rooms, regions, etc.) may be available to use to build a graph
model for the deployment environment. The model template may be
customizable for each deployment instance. For example, when using
a model template, elements of the template may be
modified/specified to account for deviations/variations of each
unit.
[0176] In some embodiments, initializing the model includes
associating specific physical devices to representative nodes in
the graph model. For example, although a template graph model
includes placeholders for devices/sensors in the template graph
model, the device/sensor nodes have not been specifically
associated with real world deployed/installed devices/sensors and
the user/installer may specify identifiers of specific real world
devices/sensors for these placeholders. In the example where
network connected devices are to be deployed across similar units,
an installer may install physical sensors/devices in each unit
based on the model template for the unit. In some embodiments,
based on a graph model template, the physical sensors/devices are
associated with nodes in the graph model template automatically.
For example, each sensor/device performs one or more tests in
conjunction with other sensors/devices to determine the
corresponding graph model node of the sensor/device. In one
example, if there are two light bulbs device nodes in the graph
model template and two network connected light bulbs have been
installed in the environment, it is required to determine which
physical light bulb corresponds to which light bulb node in the
graph model template. In this example, one of the light bulbs is
turned on while the other is turned off and feedback data from
sensors in the deployed environment is utilized to deduce the
location of the turned on light bulb and its corresponding node and
the node of the turned off light bulb is identified by process of
elimination.
[0177] In some embodiments, properties of nodes and/or edges of the
graph model are varied based on a location and/or orientation of a
unit. For example, an apartment unit on a third floor of a building
has different privacy concerns as compared to a first floor unit
and a privacy property setting value associated with a window
treatment device node and/or edges connecting the node is modified
based on the location and/or orientation of the environment. In
another example, an amount of sunlight and/or noise that enters
through a window may vary based on the direction/orientation of a
window and a property setting value associated with a window
treatment device and/or its edges is modified based on the location
and/or orientation of the associated window.
[0178] At 1004, the model is verified. For example, automatic tests
are performed to ensure that a correct model has been initialized
for the deployment environment (e.g., closed loop verification). In
some embodiments, automatic verification is performed by performing
one or more tests (e.g., test to ensure each physical device/sensor
is associated with correct corresponding node in the graph model).
For example, for each controllable device/sensor, one or more
controllable properties are activated and their effects are
detected using the sensor to determine whether the detected effects
are as expected according to the graph model (e.g., verify
component regions of the environment and associated context
information). In one example, when an environment affecting device
(e.g., sound, light, etc.) is instructed to be turned on, the
sensors that should be able to detect its effects according to the
model (e.g., according to edges between corresponding nodes of the
graph model) should detect the effects if the model is correct. In
some embodiments, a user/installer is instructed to perform one or
more actions (e.g., walk a specific path, turn on switch, etc.) and
the sensors of the environment are utilized to verify that detected
data fits with the model. In the event an error is detected, the
error may be fixed automatically (e.g., switch device association
between two nodes) and/or an error message may be provided to allow
a user/installer to correct the error.
[0179] In some embodiments, using the model, one or more
devices/sensors are verified periodically. For example,
sensors/devices may fail during use and correct functioning of
devices/sensors is verified periodically by using the graph model
to identify for a device/sensor being tested, which other
devices/sensors are able to detect/affect the device/sensor being
tested and verify that these devices/sensors are able to detect the
correct operation of the function of the device/sensor being
tested. In some embodiments, periodic verification is utilized to
detect maintenance issues. For example, verification tests for
certain common maintenance issues may be performed periodically
based on the model.
[0180] In some embodiments, by using the model, most efficient
sensor/device placement locations and/or locations of any
additional sensors/devices required to provide desired coverage may
be identified. For example, based on a desired set of rules and/or
functions desired to be enabled in an environment, a graph model
may be analyzed to identify where additional nodes can be placed in
the graph model (with which associated properties), where a node
can be eliminated, and/or which property values of edges can be
modified (e.g., improve quality). The result of the analysis may
then be implemented in the physical environment to achieve the
desired coverage.
[0181] FIG. 11 is a flowchart illustrating an embodiment of a
process for using an environment model to process a command. The
process of FIG. 11 may be at least in part implemented on any
device of devices 102, user device 108, hub 104, and/or server 106
of FIG. 1A. In some embodiments, the graph model is the graph model
stored in any device of devices 102, user device 108, hub 104,
and/or server 106 of FIG. 1A. In some embodiments, an example of
the model is the graph model of FIGS. 9A-9D. In some embodiments,
at least a portion of the process of FIG. 11 is included in 206,
214, 216, 218, 224, 226, 228, 314, 316, and/or 318 of FIGS. 2A-3,
404 of FIGS. 4, 502 and/or 504 of FIG. 5, 604 of FIG. 6, and/or
1004 of FIG. 10.
[0182] At 1102, a command is received. In some embodiments, the
command is a user command. For example, a user activates a switch
or provides an instruction. The user command may be a voice command
detected by a microphone. In some embodiments, the command is an
automation command (e.g., invoked based on preset or automatically
determined rules). In some embodiments, the command is detected.
For example, a sensor has been triggered. In some embodiments, the
command is not an explicit user command. For example, the command
is automatically invoked based on sensor detected environmental
properties. In some embodiments, the command is an invocation of a
rule. In some embodiments, the command is a discovery of a new
rule. In some embodiments, the command is a test command to test an
aspect of a device/sensor.
[0183] At 1104, a graph model of the environment is applied to
identify nodes of the graph model that are related to the command.
For example, the graph model specifies relationships (e.g., context
information) between devices/sensors as well as physical regions
(e.g., component regions) and this relationship information is
utilized to identify the related devices/sensors as well as their
command related relationships. In some embodiments, the identified
nodes are a subset of all nodes of the graph model. In some
embodiments, the identifying the nodes includes identifying a node
directly associated with the command (e.g., node corresponding to a
device specified by the command) and then determining nodes that
are connected to the identified node with a relationship that is
associated with the command. For example, when a command to open a
garage door is received, the node corresponding to a garage door
opener is identified in the graph model and then other nodes
connected to the garage door opener node with a command related
relationship are identified (e.g., identify all lighting related
nodes that are able to illuminate a garage area of the garage door
opener to turn on related lights). In some embodiments, the graph
model is hierarchical and the command may relate to one or more
levels of the graph hierarchy. For example, nodes/edges of a lower
level inherit properties from a directly corresponding higher level
node.
[0184] In some embodiments, identifying the nodes includes
identifying a node associated with a physical area/region (e.g.,
component region) of the command. For example, the command is
associated with a physical area/region (e.g., turn on lights in the
kitchen command) and the node of the graph model associated with
the physical area/region is identified. Then other nodes that are
connected to the physical area/region and also associated with the
command can be identified based on graph model edge connections of
the other nodes with the physical area/region. The physical
area/region may be directly specified in the command and/or
inferred (e.g., location of a user that provided the command). The
physical area/region may be a room, a location within a room, a
unit, a building, an outdoor location, a sub region, a location
surrounding a specific item, and/or any other identifiable
location.
[0185] In some embodiments, by using context information of the
graph, more generalized and intuitive commands are able to be
utilized than otherwise without context information of the graph.
For example, rather than requiring a user to know specific device
names and configurations of devices, the user is able to provide a
command in relation to location and/or relationship concepts of the
graph model. Based on the identified location/relationship, the
applicable devices/nodes are able to be automatically determined.
Additionally, the context information allows more generalized and
context aware rules to be automatically discovered. For example,
without the location context information of the graph model, state
inputs utilized during rule discovery/use exist in isolation.
However, with the location context information of the graph model,
an input state can be generalized (e.g., to a wider region/area
rather than only for a specific device) based on the context
information to discover/utilize rules for the generalized
area/region (e.g., discover rules for how devices in a kitchen
should behave vs. prior limited discovery of rules for only
specific devices). By allowing more generalized rules to be
discovered, invoked, and/or utilized, rules become applicable in
more situations and locations. For example, a rule specific to a
particular user but generalized to a broader area/region may be
applied in different environments as the particular user travels to
different environments with different graph models but with a same
labeled area/region (e.g., rule to turn on lights at night when the
user enters a kitchen area can be applied across all kitchens
visited by the user even though actual light devices and their
configurations are different across different kitchens). In some
embodiments, by allowing actions to be generalized to broader
concepts using the graph model (e.g., generalized to broader
location context), rule discovery may take place across various
different types of deployment environments (e.g., by same user
and/or different users).
[0186] In some embodiments, the graph model includes edges (e.g.,
connections between nodes) that identify a physical relationship
between the nodes and these edges are utilized in identifying the
nodes. For example, these types of edges and any associated
property (e.g., magnitude value) that identifies physical
location/relationship between nodes (e.g., devices/sensors,
location areas, etc.) are utilized in processing the command. The
location of devices/sensors and their physical relationships to one
another may be utilized to identify nodes that are relevant to
processing the command and/or provide physical context information
in processing information of the nodes. For example, location
triangulation is performed to identify a location of an event by
processing camera and/or microphone data of multiple sensors given
their known respective locations with each other. In another
example, the location information allows joint control of
physically co-located devices/sensors based on physical
relationships identified in the graph model without requiring exact
knowledge of all of the specific devices affected (e.g., command to
turn on all lights in a particular region). In some embodiments,
the location information of sensor placement is utilized to
determine whether a detected movement of the sensor is associated
with a human user or a pet. For example, a sensor mounted low to
the ground may detect human and pet movement while a sensor mounted
high and away from detecting low ground movements is less likely to
detect pet movement.
[0187] In some embodiments, the physical relationships of the nodes
provide outdoor environment context. For example, the graph model
captures a relationship between an outdoor weather, time/date of
day, and/or sun position. In one example, based on a
location/position/orientation of a window and the time and solar
position information at least in part obtained using the graph
model, an expected amount of sunlight entering the window is
determined and utilized to determine whether a thermostat near the
window is correctly measuring an indoor temperature or is
abnormally affected by direct sunlight shining on the thermostat.
If multiple temperature sensors exist, the one that is most likely
accurate may be selected for use. In another example, a bathroom
fan can efficiently control humidity of the bathroom using graph
model data by detecting whether a high humidity is only detected in
the bathroom (e.g., due to a user bathing) or an entire apartment
unit is humid (e.g., detected at a sensor located at a sufficiently
remote location from the bathroom as determined using the graph
model) and the bathroom fan is only turned on when a sufficient
humidity difference is detected between the bathroom sensor and the
remote sensor.
[0188] In some embodiments, the graph model includes edges that
identify functional relationships (e.g., context information)
between the nodes and these edges are utilized in identifying the
nodes. For example, these types of edges and any associated
property (e.g., functional context identification and magnitude
value of connection) that identifies functional relationships
(e.g., context information) between nodes are utilized in
processing the command. Examples of the functional relationships
include light and sound relationships. For example, amount of light
and/or noise outputted/controlled by a node that affects another
node is represented as an edge connection with an associated
magnitude value. In one example, when a user provides commands to
make a room darker, the location of the user is determined and all
of the nodes connected to the location with a functional edge
identifying a light relationship are identified and instructed to
reduce the light output.
[0189] In some embodiments, the graph model includes edges that
identify behavioral relationships (e.g., context information)
between the nodes and these edges are utilized in identifying the
nodes. For example, these types of edges and any associated
property (e.g., behavioral activity identification, associated
property of the activity, and magnitude value of connection) that
identifies behavioral activity relationships between nodes are
utilized in processing the command. Examples of the behavioral
activity relationships include relationships associated with
activities/behaviors of people (e.g., activity, ownership,
authorization, privacy, etc. of human users) and/or animals/pets.
For example, each behavioral relationship type edge is specific to
a specified activity (e.g., working, dining, studying, TV watching,
bathing, etc.) and each behavioral relationship type edge is only
applicable for the specified activity. Different contexts and
behavioral activities may result in different device settings. For
example, a reading activity may give rise to a bright light setting
while a sleeping activity may give rise to a low brightness
setting.
[0190] In some embodiments, an activity associated with the command
is determined and identifying the nodes of the graph model that are
related to the command includes identifying edges for the
determined activity. For example, it is detected that a user is
sleeping and privacy/light edges associated with the sleeping
activity are identified and utilized to adjust devices accordingly
based on the identified edges that are connected to the user's
location in the graph model. In another example, one or more
behavioral edges are associated with one or more particular users
and/or types of users and allow enabling/disabling of functionality
based on whether an authorized user and/or type of user provided
the command and/or is detected.
[0191] At 1106, the identified nodes are activated in response to
the command. For example, a desired result of the command is
achieved by enabling/disabling one or more features (e.g., modify
states) of the appropriate identified nodes. In some embodiments,
activating the nodes includes turning on/off a device/sensor of one
or more of the identified nodes. In some embodiments, activating
the nodes includes determining one or more states of the identified
nodes. For example the states can be used to learn and/or activate
one or more rules. In some embodiments, activating the nodes
includes modifying one or more device settings. In some
embodiments, activating the nodes includes determining one or more
related edges and/or nodes of the identified nodes. In some
embodiments, activating the nodes includes evaluating sensor data.
In some embodiments, activating the nodes includes modifying the
graph model based on the identified nodes. For example, one or more
properties of nodes and/or edges may be modified or one or more
nodes/edges may be added or removed from the graph model. In some
embodiments, activating the nodes includes determining how each
node relates to the command and how each node should be invoked, if
applicable, to process the command and invoking the appropriate
nodes. In some embodiments, activating the nodes includes
identifying one or more rules associated with the command and the
identified nodes and invoking the identified rules.
[0192] Although the foregoing embodiments have been described in
some detail for purposes of clarity of understanding, the invention
is not limited to the details provided. There are many alternative
ways of implementing the invention. The disclosed embodiments are
illustrative and not restrictive.
* * * * *