U.S. patent application number 14/159318 was filed with the patent office on 2014-05-15 for interactivity with a mixed reality.
This patent application is currently assigned to Nant Holdings IP, LLC. The applicant listed for this patent is Nant Holdings IP, LLC. Invention is credited to Ronald H. Cohen.
Application Number | 20140132632 14/159318 |
Document ID | / |
Family ID | 37809447 |
Filed Date | 2014-05-15 |
United States Patent
Application |
20140132632 |
Kind Code |
A1 |
Cohen; Ronald H. |
May 15, 2014 |
Interactivity With A Mixed Reality
Abstract
Methods of interacting with a mixed reality are presented. A
mobile device captures an image of a real-world object where the
image has content information that can be used to control a mixed
reality object through an offered command set. The mixed reality
object can be real, virtual, or a mixture of both real and
virtual.
Inventors: |
Cohen; Ronald H.; (Pasadena,
CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Nant Holdings IP, LLC |
Culver City |
CA |
US |
|
|
Assignee: |
Nant Holdings IP, LLC
Culver City
CA
|
Family ID: |
37809447 |
Appl. No.: |
14/159318 |
Filed: |
January 20, 2014 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
12505726 |
Jul 20, 2009 |
8633946 |
|
|
14159318 |
|
|
|
|
11972062 |
Jan 10, 2008 |
7564469 |
|
|
12505726 |
|
|
|
|
PCT/US06/33811 |
Aug 29, 2006 |
|
|
|
11972062 |
|
|
|
|
60712590 |
Aug 29, 2005 |
|
|
|
Current U.S.
Class: |
345/633 |
Current CPC
Class: |
A63F 13/10 20130101;
A63F 13/323 20140902; A63F 2300/695 20130101; A63F 2300/8094
20130101; H04N 21/4223 20130101; A63F 13/06 20130101; A63F 13/213
20140902; A63F 2300/1093 20130101; G09G 5/18 20130101; A63F 13/332
20140902; G09G 5/377 20130101; A63F 2300/5573 20130101; G06T 19/00
20130101; G06T 15/00 20130101; A63F 2300/8082 20130101; H04M
1/72544 20130101; G06F 3/0484 20130101; A63F 13/65 20140902; A63F
2300/538 20130101; H04M 2250/52 20130101; A63F 13/12 20130101; G06F
3/167 20130101; A63F 13/655 20140902; H04N 21/6582 20130101; A63F
2300/6018 20130101; A63F 13/52 20140902; G06T 19/006 20130101; A63F
13/25 20140902; G06T 19/00 20130101; G06T 19/006 20130101 |
Class at
Publication: |
345/633 |
International
Class: |
G06T 19/00 20060101
G06T019/00 |
Claims
1. A method of interacting with a mixed reality, comprising:
obtaining, by a mixed reality system, image data relating to a
real-world object; identifying, by the mixed reality system,
attributes associated with the real world object based on the image
data; obtaining, by the mixed reality system, a command set for the
real-world object based on the attributes, the command set obtained
from a database storing a plurality of command sets for multiple
mixed reality objects, where the command sets comprise information
addresses corresponding to commands that control the real-world
object; presenting, via a physical computer interface, a virtual
interface comprising the command set to a user, where the command
set comprising a non-visible interactive command set is
superimposed on an image of a corresponding real-world interface
part of the corresponding real-world object and is configured to
stay in proper relation to the corresponding real-world object
within a mixed reality view; and executing, by the mixed reality
system, at least one command from the non-visible interactive
command set upon the user interacting with the image of the
corresponding real-world interface part by issuing a call to a
corresponding information address of the at least one command.
2. The method of claim 1, wherein the real-world object comprises
at least one of an appliance, a consumer electronic appliance, a
kitchen appliance, a business office appliance, a clinic appliance,
a printer, a set-top box, a television, a vending machine, a radio,
and a stereo.
3. The method of claim 1, wherein the real-world object is an
element of an electronic game.
4. The method of claim 3, wherein the attributes indicates the
real-world object has a value in the game.
5. The method of claim 3, wherein the attributes indicates the
real-world object is a goal object of the game.
6. The method of claim 1 wherein, the real-world object is part of
a multi-part object.
7. The method of claim 1, wherein the attributes comprise at least
one of the following a name, a class, and an environmental
characteristic.
8. The method of claim 1, wherein the at least one command is one
of an "on" command and an "off" command.
9. The method of claim 1, wherein the execution of the at least one
command results in changing a channel.
10. The method of claim 1, wherein the execution of the at least
one command results in playing music.
11. The method of claim 1, wherein the execution of the at least
one command inputs data associated with a virtual weapon.
12. The method of claim 1, wherein the execution of the at least
one command inputs data associated with virtual travel.
13. The method of claim 1, wherein the image data comprises a still
image.
14. The method of claim 1, wherein the image data comprises a video
image.
15. The method of claim 1, further comprising using a sensor
associated with the mixed reality system to interact with the
software.
16. The method of claim 15, wherein the sensor provides information
associated with a physical location.
17. The method of claim 15, wherein the sensor provides information
associated with at least one of an orientation and an
acceleration.
18. The method of claim 1, further comprising superimposing the
virtual interface on the physical interface where the virtual
interface relates to the real-world object.
Description
[0001] This application is a Continuation of U.S. application Ser.
No. 12/505,726, filed Jul. 20, 2009, which is a Continuation of
U.S. application Ser. No. 11/972,062 filed Jan. 10, 2008, which is
a Continuation-in-Part of PCT Application Serial No. PCT/US06/33811
filed Aug. 29, 2006, which claims the benefit of U.S. Provisional
Application No. 60/712,590 filed Aug. 29, 2005. These and all other
extrinsic references are incorporated herein by reference in their
entirety. Where a definition or use of a term in an incorporated
reference is inconsistent or contrary to the definition of that
term provided herein, the definition of that term provided herein
applies and the definition of that term in the reference does not
apply.
FIELD OF THE INVENTION
[0002] The field of the invention is mixed reality
technologies.
BACKGROUND
[0003] Virtual reality systems are having an ever increasing impact
on our daily lives. Virtual world games, for example, continue to
dominate the gaming industry and are migrating to mobile devices of
all sorts including cell phones. Just as virtual worlds are
impacting our real lives, the real world will have an ever growing
impact on our virtual lives. The distinction between virtual worlds
and the real world will continue to become ever more blurry
resulting in an ever present mixed reality. People living with the
various mixed realities will interact with mixed reality objects
whether they are real, virtual, or a mix of both. To date there are
no easy, intuitive methods for a person to command or control mixed
reality objects.
[0004] Google Maps.TM. provides a method for people to interact
with a mixed reality where real-world satellite images are
superimposed over computer generated images showing location or
road names. Although Google Maps is useful for interacting with
maps and for virtual travel, it does not address command or control
of mixed reality objects.
[0005] U.S. Pat. No. 6,502,000 to Arnold et al. offers a system for
controlling real-world devices through a communication link between
a controlled device and a controlling device but does not address
the need for interacting with virtual objects of a mixed reality.
The Arnold approach is useful for closed ecosystems where all
devices use a common control protocol. However, the approach does
not lend itself to a universal, heterogeneous environment where
mixed reality objects employ multiple command or control
protocols.
[0006] The mixed reality system described by U.S. Pat. No.
6,604,049 to Yokota teaches that a person can interact with a mixed
reality by accessing and displaying information about a real-world
object on a virtual reality display device worn by the person. A
camera captures images of surrounding objects and sends the image
data to a remote server for processing. The server then returns
information associated with the objects for display on the user's
display screen. Although useful for obtaining information about
real-world objects, the system does not teach how to command or
control mixed reality objects. Additionally, it is unlikely
individuals will purchase or wear the contemplated virtual reality
system as opposed to using a system the individual already owns,
possibly a cell phone or PDA.
[0007] U.S. Publication No. 2002/0090985 to Tochner et al. further
advances mixed reality interactions by teaching that a player of a
virtual world game can interact with the real-world by establishing
a correspondence between virtual objects and real-world objects.
Still, Tochner does not provide for the command or control of mixed
reality objects.
[0008] The applicant's patent U.S. Pat. No. 7,016,532 and patent
applications U.S. Publication Nos. 2004/0208372, 2006/0002607,
2006/0110034, and 2006/0181605 teach that one can use a mobile
device to capture image data associated with a real-world object to
access information about the real-world object. The image data is
used to identify network addresses where information associated
with the real-world objects can be obtained. The network addresses
can also be used to perform an action relating to the real-world
objects. Although the applicant's previous filings teach performing
an action associated with a returned network address and
controlling real-world objects, those filings do not teach how to
command or control mixed reality objects.
[0009] What has not yet been appreciated is that a mobile device
can provide an interface to command or control mixed reality
objects by capturing an image of a real-world object. Thus, there
is still a need for systems and methods for interacting with mixed
realities.
SUMMARY OF THE INVENTION
[0010] The present invention provides apparatus, systems and
methods in which a mobile device can be used to command or control
objects within mixed realities.
[0011] One can interact with a mixed reality by capturing an image
of a real-world object using a mobile device where content from the
image is used to derive a command set for a mixed reality object
that is accessible by the mobile device. The mixed reality object
can include a virtual object in a virtual world, a second
real-world object, or an object having presence within a virtual
world and the real world. Command sets are contemplated to have
commands including changing channels, turning a device on or off,
or inputting data.
[0012] As used herein, "real-world objects" are those that have
mass and extent in the physical world, and "virtual objects" are
those that do not have mass and extent in the physical world. The
term "mixed reality" means a reality having at least one real-world
object and at least one virtual object, which a user of the mixed
reality space can perceive as interacting with one another in some
manner.
[0013] Mixed realities cover the spectrum from slight modifications
of the physical world where virtual elements are superimposed on
the real world, to fully virtual worlds where a real-world object
has a corresponding presence in the virtual world. In general, a
mixed reality requires support from an electronic interface,
possibly a cell phone or PDA, to view or access the mixed reality.
The real-world itself is not a mixed reality because there are no
virtual elements. Many computer games per se are virtual realities,
but they are not mixed-realities because they are self contained
and do not share real-world elements. Other computer games
supporting virtual worlds can represent a mixed-reality when a
real-world object has a corresponding presence within the game
world including trading cards, money, avatars, or other
representations of real-world elements. An example of a mixed
reality weighted heavily toward the virtual end of the spectrum
includes, Linden Lab's Second Life.RTM. where real-world dollars
correspond to money in Second Life.
[0014] A "mixed reality object" is an object within a mixed reality
capable of interacting with other objects in the mixed reality. A
physical television per se is a real-world object and not a mixed
reality object, even though it can be used to display virtual or
mixed reality content. The physical television set can be part of a
mixed reality when it has a corresponding presence in a mixed
reality space. For example, if a user turns off the physical
television, the corresponding presence in the mixed reality space
could be shown as having been turned off.
[0015] Various objects, features, aspects and advantages of the
inventive subject matter will become more apparent from the
following detailed description of preferred embodiments, along with
the accompanying drawings in which like numerals represent like
components.
BRIEF DESCRIPTION OF THE DRAWING
[0016] FIG. 1A is a schematic of a user using a mobile device to
capture image data of a real-world object.
[0017] FIG. 1B is a schematic of the mobile device of FIG. 1,
displaying an image of a mixed reality world that includes a
portion of the image data, and command sets associated with one or
more objects in the mixed reality objects.
[0018] FIG. 2 is a schematic of methods for interacting with a
mixed reality.
DETAILED DESCRIPTION
[0019] Mixed Reality Environment
[0020] In FIG. 1A, user 100 interacts with a mixed reality
environment that includes real-world object 120.
[0021] In a preferred embodiment, user 100 uses mobile device 110
as a command or control interface for mixed reality objects that
are local to mobile device 110. Although the terms "local" and
"locality" are context dependent, as used herein two objects are
considered "local" to each other when they are physically or
virtually near each other. Physically local objects are generally
within about 100 meters of each other. More preferably, objects are
considered local when they are within 10 meters of each other. It
is also contemplated that a user could interact with objects that
are more than 100 meters from mobile device 110.
[0022] Virtual objects are preferably within visual range of each
other in the mixed reality. It is also contemplated that virtual
objects could be hidden from view, yet remain in visual range. For
example, a virtual pet that can only be seen using mobile device
110 might be hiding behind a real-world car, yet the virtual pet is
still virtually local to mobile device 110.
[0023] Although a preferred embodiment calls for mobile device 110
to command or control local mixed reality objects, the techniques
disclosed can also be applied to remote mixed reality objects which
also fall within the scope of the inventive subject matter.
[0024] Mobile device 110 comprises display 113, camera 115, and
network connection 119. Through the use of these elements, mobile
device 110 generally becomes an interface portal into the mixed
reality utilizing support from software running at least partially
on external computer systems. One should appreciate that mobile
device 110 or its abstracted interface, can be considered an object
within the mixed reality.
[0025] Display 113 is contemplated to include a GUI (Graphical User
Interface) that can display image data or user interfaces. User 100
interacts with the mixed reality through the interface rendered by
display 113. Typically, display 113 comprises an LCD (Liquid
Crystal Display) as used in computers, cell phones, PDAs, or other
similar devices. Additionally mobile device 110 preferably runs a
browser application or other GUI for rendering graphics or command
interfaces on display 113. In some embodiments, display 113 renders
the real-world object along with virtual objects that are objects
in the mixed reality. It is also contemplated that display 113
could comprise non-visual displays for use by other senses. For
example, a blind person could use a tactile display to interact
with a mixed reality as opposed to a visual-based display or a deaf
person could use an auditory interface system to interact with a
mixed reality.
[0026] Camera 115 captures image data of real-world object 120 by
capturing still images, video images, streamed images, or other
image data. CCD cameras as typically used in hand-held device are a
contemplated form of camera. Any camera capable of electronically
capturing image data would also be suitable. The image data can be
captured through manually snapping a single still frame, recording
audio or video, or simply by having the camera stream real-time
images. In some embodiments, camera 115 has advantageous features
including zoom, infrared sensitivity, or ultraviolet sensitivity
which can increase the amount of information obtained relating to
real-world object 120.
[0027] Image data of real-world object 120 captured by camera 115
combines with additional data obtained over network connection 119,
possibly from computer 130 or real-world object 120, to create a
interactive view of the mixed reality. The additional data can also
be obtained by mobile device 110 and can include time, temperature,
location (e.g. latitude or longitude), or other information.
[0028] Network connection 119 preferably utilizes standard Internet
related protocols including TCP, UDP, IP, HTTP, wired protocols,
wireless protocols, or other communication protocols; although all
communication protocols are contemplated. Mobile device 110 uses
network connection 119 to communicate with other devices or mixed
reality objects external to mobile device 110 including real-world
object 120 (if it is so enabled) or computer 130.
[0029] Mobile device 110 can also comprise optional sensor 111.
Sensor 111 acquires additional data about mobile device 110 or
real-world object 120 for use in interacting with the mixed
reality. Contemplated sensors include audio sensors,
accelerometers, orientation sensors, GPS (Global Positioning
System) or other physical location sensors, or other sensors beyond
camera 115. Additional sensor 111 provides information that can be
used by software to further identify attributes associated with
real-world object 120 through the location or the orientation of
mobile device 110. Accelerometers provide motion information that
can be used to determine how the mobile device moves relative to
real-world object 120 which can be used to alter the visual display
of virtual objects or can be used as command inputs to control
objects. For example, U.S. Pat. No. 7,271,795 to Bradski describes
a suitable method for using sensor input to alter a visual display
when interacting with a virtual space.
[0030] Mobile device 110 is preferably a hand-held device including
a cell phone, a PDA, a remote control, a hand-held computer, or
other similar portable device. It is also contemplated that mobile
device 110 could include larger, non-hand-held devices. For
example, mobile device 110 could be a vehicle, potentially a police
car, equipped with a camera, a display, or a network connection. A
police car so equipped can aid police officers by automatically
viewing a license plate, for example, and providing the office with
a list of commands relating to the car, the driver, the driver's
record, or other real or virtual objects.
[0031] Real-world object 120 can be any object in the real world
that can be imaged whether it is a machine, a person, a building, a
plant, a set of objects, or other physical items. In embodiments
where real-world object 120 is a machine, it is contemplated that
real-world object 120 also communicates with computer 130 to
support interactions with the mixed reality.
[0032] One should note that real-world object 120 does not
necessarily have to be electronic or mechanical to be commanded or
controlled within a mixed reality. Real-world object 120 could be
co-existent with a virtual object within a virtual world where the
virtual representation of real-world object 120 can be commanded or
controlled. Consider, for example, a virtual game where a real
world fire hydrant corresponds to a virtual fire hydrant in the
virtual game world. Although a real player cannot operate the real
fire hydrant, the player could capture image data of the real fire
hydrant and then command or control the corresponding virtual fire
hydrant as part of the game.
[0033] Real-world object 120 is also contemplated to include
multiple parts that can be controlled individually or collectively
through an image of at least a portion of the object. For example,
a stereo system represents a multi-part object that could include a
CD player, speakers, a tuner, a graphical analyzer, or other parts.
When image data of the stereo is captured, one or more virtual
interfaces, each having a command set, can be superimposed over the
image of the stereo or its individual parts. The act of combining
the virtual interface with the image data from the real-world
essentially forms a mixed reality representation of the stereo.
[0034] Computer 130, external to mobile device 110, runs software
for analyzing image data to recognize attributes associated with
real-world object 120. The preferred embodiment employs techniques
similar to those described in co-owned U.S. Pat. No. 7,016,532
where a remote service identifies network addresses associated with
a target object and returns the network address to a mobile
device.
[0035] The software running on computer 130 communicates with
software on mobile device 110 and, potentially, with software
running on real-world object 120 to support the structure of the
mixed reality environment. It is also contemplated that computer
130 can be incorporated into real-world object 120.
[0036] Computer 130 can be one of any of a full spectrum of
computer resources. At the real-world end of the mixed reality
spectrum, computer 130 can comprise a web server or database that
supplies command or control information to mobile device 110. At
the other end of the mixed reality spectrum where the virtual
aspects dominate the mixed reality, computer 130 can include a
virtual world server or a game server that supply virtual world
environment information for use by mobile device 110.
[0037] Mixed Reality Interface
[0038] In FIG. 1B, mobile device 110 displays a mixed reality
interface on display 113. The mixed reality interface can include
an image of real-world object 120 having command set 125, or
virtual object 150 having command set 155.
[0039] Mobile device 110 uses camera 115 to capture image data of
real-world object 120 and sends at least a portion of the image
data, attributes, or other optional information from sensor 111 to
external computer 130 where attributes associated with real-world
objects 120 are identified.
[0040] As previously mentioned, computer 130 uses the attributes to
identify one or more network addresses or other commands using
techniques similar to those described in U.S. Pat. No. 7,016,532.
The techniques are extended to include any virtual objects within
the mixed reality that are accessible to mobile device 110.
Computer 130 returns information associated with one or more of the
mixed reality objects (either real or virtual) to mobile device 110
including data necessary for rendering virtual object on display
113.
[0041] Consider a case where the local object is a real object.
Attributes from the image data associated with real-world object
120 are identified and used by the software to offer command set
125 to control real-world object 120 which can be physically local
to mobile device 110. As shown in FIG. 1B, real-world object 120
can include a vending machine. One should note that term "offer" is
used euphemistically to mean expose or otherwise provide access to
command set. A user, mobile device 110, or other entity interacts
with the vending machine by interfacing with command set 125
listing one or more vending machine commands. Contemplated commands
include purchasing an item or possibly checking the inventory of
the machine. Other contemplated commands include those that might
require proper authentication including turning the vending machine
on or off. One should appreciate that the commands are not limited
to those described within this document and that all possible mixed
reality objects commands are contemplated.
[0042] Also consider the case where a mixed reality object is a
virtual object. Again, attributes derived from the image data or
optional sensor data are used to by the software to determine if
there are virtual objects local to mobile device 110 within the
mixed reality. The software also uses the attributes or optional
sensor data to offer command set 155. Command set 155 offers one or
more commands to control virtual object 150 which, as shown in FIG.
1B, can include a virtual pet. Example commands for virtual object
150 include turning the virtual object on or off, instructing the
object to move, upgrading the object, or any other command that can
alter the operation of virtual object 150. One should also
appreciate that virtual objects remote to mobile device 110 can
also be controlled.
[0043] In some embodiments, the software on computer 130 runs in
conjunction with software on mobile device 110 to ensure virtual
objects are displayed in proper relation to real objects. For
example, virtual object 150 can be rendered in a manner where it is
partially hidden by real-world object 120 even though the virtual
object does not exist in the real world and is not part of the
original image data.
[0044] One should appreciate that in a mixed reality, virtual
objects and real objects co-exist on equal footing as if they are
both natural to the mixed reality. Therefore, the manner in which
one interfaces to a virtual object of the mixed reality using
mobile device 110 is substantially identical to the manner in which
one interfaces to a real object. Even though the infrastructure for
a mixed reality system might treat real and virtual objects
differently; the user interface of each type of object is
preferably similar so that a user experiences both types of objects
as being natural to the mixed reality.
[0045] Command sets 125 and 155 preferably include one or more
commands. A command set can be displayed using any suitable types
of command prompts including menus, radio buttons, simple list of
one or more commands, drop-down lists, command line prompt, icons
that open to reveal the command set, graphical controls, or other
types of command offerings. Contemplated command input mechanism to
access a command include calling an API, handset motion, stylus,
rocker buttons, voice commands, or other input methods.
[0046] It is also contemplated that a command set can be hidden
from the user, but yet is still accessible through mobile device
110. Consider the example where real-world object 120 is a vending
machine. Command set 125 for the vending machine might not be
visible to the user while mobile device 110 is still aware of
command set 125. A user could access command set 125 by interacting
directly with the image of the vending machine, including
"clicking" on the image of one of the vending machine buttons.
Alternatively, mobile device 110 could access command set 125
automatically using one or more API calls without direct
interaction with the user. Furthermore it is also contemplated that
other remote devices (e.g. computer 130) could also access commend
set 125 to control the mixed reality object.
[0047] One should also appreciate that the command bubbles
displaying command sets 125 and 155 are objects of the mixed
reality. It is, therefore, contemplated that as a user pans, tilts,
or otherwise moves mobile device 110, the command set can also
optionally move within display 113 to stay in proper relation to
object within the mixed reality view.
[0048] In a preferred embodiment, the commands in a command set
comprise one or more network addresses obtained from computer 130.
The network address can include URLs, URIs, remote procedure calls,
email addresses, IP addresses, web service APIs, or other network
callable commands or addresses. When a user selects a command from
the command set, preferably mobile device 110 executes the command
by issuing a call to the corresponding network address. In some
embodiments, the network address points to a function on computer
130, possibly where computer 130 is running as a web server or a
virtual world server. In other embodiments, the network address
points to an accessible function on real-world object 120, or other
mixed reality object accessible by mobile device 120. Although the
preferred embodiment employs web services, it is also contemplated
that other types of command services can also be used including
email, FTP, Telnet, proprietary services, or other services capable
of executing a command. It is also contemplated that a command
associated with a virtual object can be at least partially executed
internal to mobile device 110 as opposed to executing on a remote
computer system.
[0049] Mixed Reality Interactions
[0050] In FIG. 2, method 200 describes how one can interact with a
mixed reality.
[0051] At step 210 a mobile device is preferably used to
electronically capture image data of a real-world object. The
real-world object is considered to have mass and extent and
actually exist in the physical world. A real-world object can
include anything that can be photographed including a person, a
place, or a thing. Image data is contemplated to comprise at least
a still image at step 212 or a video image at step 214. One should
note that a projected image is not considered a real-world object
because is lacks mass, but is rather considered a virtual object.
However, a photograph, a mural, a poster, a magazine advertisement
or other physically rendered image is considered to be a real-world
object. Although a preferred embodiment captures image data of a
real-world object, it is also contemplated that one could capture
image data of a virtual object without significantly departing from
the inventive subject matter.
[0052] At step 220 attributes of the real-world object are
identified from the image data. In some embodiments, the mobile
device comprises software to derive the attribute, however, in
preferred embodiments the image data is transmitted to a remote
computer which identifies the attributes. It is also contemplated
that any combination of software applications, local or remote to
the mobile device, can be used to derive the attributes.
[0053] Attributes include metadata associated with the content of
the image data, and relate to objects in the image. Attributes are
used to identify or to derive a command set associated with various
mixed reality objects accessible by a mobile device. At step 222,
preferred attributes include a name, a class, or an environmental
characteristic associated with a real-world object.
[0054] Names are useful when offering a specific command set for a
target object. For example, if a user captures an image of a
Xerox.RTM. 4150PS printer, then the name "Xerox 4150PS" can be used
to access specific 4150PS command functions and provide them to the
user.
[0055] Classes represent broad categories of objects. To continue
the previous printer example, an appropriate class for the Xerox
printer is "printer". Classifying objects provides for offering
generic commands that are common across many types of objects or
devices. A common set of commands for many appliances include "on"
or "off" commands. In a preferred embodiment, classes are organized
hierarchically as in a classic object oriented programming
methodology.
[0056] Environmental characteristic are attributes that indicate
the environmental condition in which the real-world object exists.
Examples include weather conditions, time of day, geographic
location, or other environmental information. Environmental
characteristics are advantageously used to adjust command sets to
meet the potential needs of a user or alter the behavior of the
mixed reality. For example, on a hot day when interacting with a
vending machine that exists in a mixed reality (e.g. having at
least a portion of the vending machine represented as a virtual
object in the mixed reality), a user could observe that a virtual
entity in the mixed reality, perhaps in a game, also suffers from
the heat and requires a drink. Such an approach can be used for
vendor promotions or other forms of advertising within a mixed
reality.
[0057] A step 230 the attributes are used to interact with software
that is located at least partially external to the mobile device.
The software determines an appropriate command set as a function of
the attributes or other sensor data for one on or more mixed
reality objects that are accessible by the mobile device.
Preferably, software running on a computer external to the mobile
devices uses the attributes to consult a database having command
set information. It is also contemplated that a database of command
set information can be located within or accessed from a real-world
object.
[0058] The mixed reality object for which the command set is
offered can very significantly without departing from the inventive
subject matter. For example, the mixed reality object can include a
virtual object, a portion of the real-world object, or an appliance
(e.g. kitchen appliance, consumer electronics, set-top box, TV, or
other electronic device) as indicated at steps 234, 236, and 238,
respectively. Contemplated appliances include those associated with
kitchens, business offices, consumer electronics, clinics, or other
functional devices.
[0059] Additionally, at step 231 the local mixed reality object
comprises an element of a game where the game element can also be a
real or virtual object. Obtaining a command set associated with the
game elements allows for a user of the mobile device to fully
interact with a mixed reality game.
[0060] In some gaming embodiments, including at step 234, the
real-world object is a player of the game. When a user captures
image data of the player, the user can be offered a set of game
related commands that can affect the player. Contemplated commands
include attack, add player, delete player, or other game related
commands.
[0061] In other embodiments relating to games, as at step 235, the
real-world object can represent a goal object of the game.
Contemplated games having goal objects include virtual tag,
scavenger hunts, geo-caching, hide-and-seek, or other games having
goal objects. Again, such an approach provides for promotions or
other advertising efforts.
[0062] In yet other gaming embodiments, at step 237 the real-world
object can have value within the game system. For example, an image
of an advertisement could result in points added to a player's
score or, perhaps, unlock additional commands that can then be used
within the game or with game objects.
[0063] Preferred embodiments also optionally allow a user to select
a command from the command set at step 250. Contemplated commands
include:
[0064] At step 252 turning a device on or off by selecting an "on"
command or an "off" command
[0065] At step 254 changing a channel or station of a TV, radio, or
other appliance.
[0066] At step 256 inputting data associated with virtually
traveling to a new place where the place can be a real or virtual
location.
[0067] At step 258 playing music.
[0068] At step 257 inputting data associated with a virtual
weapon.
[0069] At step 270, the mobile device can optionally include an
additional sensor other than a camera. The additional sensor data
acquired by the mobile device can also be used to interact with
external software to aid in determining a desired command set. At
step 271 the sensor data optionally provides information associated
with a physical location (e.g. latitude or longitude), preferably
from a GPS receiver. At step 272 the sensor data optionally
provides information associated with orientation or acceleration of
the mobile device preferably from an accelerometer, a gyroscope, a
compass, or other sensor.
[0070] One skilled in the art should now appreciate the numerous
advantages provided by the disclosed subject matter for interacting
with a mixed reality. The disclosed methods provide an easy,
intuitive method for a person to command and control mixed reality
objects. Using a cell phone or PDA offers a ubiquitous interface
with which most people familiar. Users simply capture image data of
an object and then select a command from an offered command set to
control various mixed reality objects or automatically control the
mixed reality objects.
[0071] It should be apparent to those skilled in the art that many
more modifications besides those already described are possible
without departing from the inventive concepts herein. The inventive
subject matter, therefore, is not to be restricted except in the
spirit of the appended claims. Moreover, in interpreting both the
specification and the claims, all terms should be interpreted in
the broadest possible manner consistent with the context. In
particular, the terms "comprises" and "comprising" should be
interpreted as referring to elements, components, or steps in a
non-exclusive manner, indicating that the referenced elements,
components, or steps may be present, or utilized, or combined with
other elements, components, or steps that are not expressly
referenced. Where the specification claims refers to at least one
of something selected from the group consisting of A, B, C . . .
and N, the text should be interpreted as requiring only one element
from the group, not A plus N, or B plus N, etc.
* * * * *