U.S. patent application number 15/372036 was filed with the patent office on 2017-03-30 for multiple actuation handheld device.
This patent application is currently assigned to Immersion Corporation. The applicant listed for this patent is Immersion Corporation. Invention is credited to Danny A. Grant, Robert Heubel.
Application Number | 20170087459 15/372036 |
Document ID | / |
Family ID | 41395581 |
Filed Date | 2017-03-30 |
United States Patent
Application |
20170087459 |
Kind Code |
A1 |
Grant; Danny A. ; et
al. |
March 30, 2017 |
Multiple Actuation Handheld Device
Abstract
A device includes a housing, a processor that is coupled to the
housing, the processor is configured to process a software program
stored in a memory. A touch screen is coupled to the housing and
configured to display graphical objects, wherein a sensor signal
associated with a user's interaction with the touch screen is
provided to the processor. A first actuator is coupled to the touch
screen and positioned within the housing. The first actuator is
configured to output a first haptic effect to the touch screen upon
receiving a first activating signal from the processor. A second
actuator is coupled to the housing and configured to output a
second haptic effect to the housing upon receiving a second
activating signal from the processor. The first activating signal
is associated with a foreground event and the second activating
signal is associated with a background event occurring in the
software program.
Inventors: |
Grant; Danny A.; (Montreal,
CA) ; Heubel; Robert; (San Leandro, CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Immersion Corporation |
San Jose |
CA |
US |
|
|
Assignee: |
Immersion Corporation
|
Family ID: |
41395581 |
Appl. No.: |
15/372036 |
Filed: |
December 7, 2016 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
14623673 |
Feb 17, 2015 |
9545568 |
|
|
15372036 |
|
|
|
|
13475334 |
May 18, 2012 |
8982068 |
|
|
14623673 |
|
|
|
|
12237334 |
Sep 24, 2008 |
8749495 |
|
|
13475334 |
|
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
A63F 13/2145 20140902;
G06F 2203/013 20130101; A63F 13/285 20140902; G06F 3/0416 20130101;
G06F 3/016 20130101; G06F 1/3262 20130101; G06F 3/041 20130101 |
International
Class: |
A63F 13/285 20060101
A63F013/285; A63F 13/2145 20060101 A63F013/2145 |
Claims
1-19. (canceled)
20. A computing device comprising: a housing; a touch screen
coupled to the housing; a sensor configured to detect a contact
with the touch screen and transmit a sensor signal associated with
the contact; a processor; and a memory on which instructions
executable by the processor are stored to cause the processor to:
transmit a display signal to the touch screen, the display signal
being configured to cause the touch screen to display a graphical
object; detect, based on the sensor signal, a physical object
contacting the touch screen at a first location associated with the
graphical object; cause a first actuator to output a first haptic
effect in response to detecting the physical object contacting the
touch screen at the first location; detect, based on the sensor
signal, a movement of the physical object along the touch screen
from the first location to a second location associated with the
graphical object; and cause a second actuator to output a second
haptic effect in response to detecting, during the movement, at
least one of (i) the physical object contacting the touch screen at
the second location, or (ii) an amount of pressure with which the
physical object is contacting the touch screen decreasing.
21. The computing device of claim 20, wherein the memory comprises
instructions executable by the processor to cause the processor to
cause the second actuator to output the second haptic effect in
response to detecting, during the movement, that the physical
object is contacting the touch screen at the second location.
22. The computing device of claim 21, wherein the graphical object
is a virtual guitar comprising a plurality of strings, the first
location is associated with a string of the plurality of strings,
and wherein the memory further comprises instructions executable by
the processor to cause the processor to determine the first haptic
effect according to the string associated with the first
location.
23. The computing device of claim 22, wherein: the second location
is a predetermined distance from the first location; the first
haptic effect is configured to simulate plucking the string of the
virtual guitar; and the second haptic effect is configured to
simulate releasing the string.
24. The computing device of claim 20, wherein the memory comprises
instructions executable by the processor to cause the processor to
cause the second actuator to output the second haptic effect in
response to detecting, during the movement, that the amount of
pressure with which the physical object is contacting the touch
screen is decreasing.
25. The computing device of claim 24, wherein the memory comprises
instructions executable by the processor to cause the processor to
cause the second actuator to output the second haptic effect in
response to detecting, during the movement, that the amount of
pressure with which the physical object is contacting the touch
screen has decreased to an amount below a threshold but above
another threshold.
26. The computing device of claim 20, wherein the first actuator is
a screen actuator coupled to the touch screen and the second
actuator is a housing actuator coupled to the housing, and wherein
the first haptic effect is different from the second haptic
effect.
27. The computing device of claim 20, wherein the memory further
comprises instructions executable by the processor to cause the
processor to cause an actuator to output a third haptic effect
configured to mimic an audio note being output.
28. The computing device of claim 20, wherein the memory further
comprises instructions executable by the processor to cause the
processor to cause an actuator to output a third haptic effect that
is discordant with an audio note being output.
29. The computing device of claim 20, wherein the memory further
comprises instructions executable by the processor to cause the
processor to cause an actuator to output a third haptic effect that
is configured to distract a user from an event occurring in a video
game.
30. A method comprising: displaying, by a computing device, a
graphical object on a touch screen; detecting, by the computing
device, a physical object contacting the touch screen at a first
location associated with the graphical object; causing, by the
computing device, a first actuator to output a first haptic effect
in response to detecting the physical object contacting the touch
screen at the first location; detecting, by the computing device, a
movement of the physical object along the touch screen from the
first location to a second location associated with the graphical
object; and causing, by the computing device, a second actuator to
output a second haptic effect in response to detecting, during the
movement, at least one of (i) the physical object contacting the
touch screen at the second location, or (ii) an amount of pressure
with which the physical object is contacting the touch screen
decreasing.
31. The method of claim 30, wherein the computing device causes the
second actuator to output the second haptic effect in response to
detecting, during the movement, that the physical object contacting
the touch screen at the second location.
32. The method of claim 31, wherein the graphical object is a
virtual guitar comprising a plurality of strings, the first
location is associated with a string of the plurality of strings,
and further comprising determining the first haptic effect
according to the string associated with the first location.
33. The method of claim 33, wherein the computing device causes the
second actuator to output the second haptic effect in response to
detecting, during the movement, that the amount of pressure with
which the physical object is contacting the touch screen has
decreased to an amount below a threshold but above another
threshold.
34. The method of claim 30, further comprising: detecting three or
more inputs in a predetermined pattern; and causing an actuator to
output a haptic effect in response to detecting the three or more
inputs in the predetermined pattern.
35. The method of claim 30, wherein the first actuator is a screen
actuator coupled to the touch screen and the second actuator is a
housing actuator coupled to a housing of the computing device, and
wherein the first haptic effect is different from the second haptic
effect.
36. A non-transitory computer readable medium comprising program
code executable by a processor to cause the processor to: transmit
a display signal to a touch screen, the display signal being
configured to cause the touch screen to display a graphical object;
detect, based on a sensor signal from a sensor, a physical object
contacting the touch screen at a first location associated with the
graphical object; cause a first actuator to output a first haptic
effect in response to detecting the physical object contacting the
touch screen at the first location; detect, based on the sensor
signal from the sensor, a movement of the physical object along the
touch screen from the first location to a second location
associated with the graphical object; and cause a second actuator
to output a second haptic effect in response to detecting, during
the movement, at least one of (i) the physical object contacting
the touch screen at the second location, or (ii) an amount of
pressure with which the physical object is contacting the touch
screen decreasing.
37. The non-transitory computer readable medium of claim 36,
further comprising program code that is executable by the processor
to cause the processor to cause the second actuator to output the
second haptic effect in response to detecting, during the movement,
that the physical object is contacting the touch screen at the
second location.
38. The non-transitory computer readable medium of claim 37,
wherein the graphical object is a virtual guitar comprising a
plurality of strings, the first location is associated with a
string of the plurality of strings, and further comprising program
code that is executable by the processor to cause the processor to
determine the first haptic effect according to the string
associated with the first location.
39. The non-transitory computer readable medium of claim 36,
further comprising program code that is executable by the processor
to cause the processor to cause the second actuator to output the
second haptic effect in response to detecting, during the movement,
that the amount of pressure with which the physical object is
contacting the touch screen has decreased to an amount below a
threshold but above another threshold.
40. The non-transitory computer readable medium of claim 36,
wherein the first actuator is a screen actuator coupled to the
touch screen and the second actuator is a housing actuator coupled
to a housing of the computing device, and wherein the first haptic
effect is different from the second haptic effect.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation of co-pending U.S. patent
application Ser. No. 14/623,673, filed Feb. 17, 2015, entitled
"Multiple Actuation Handheld Device With Housing and Touch Screen
Actuators," which is a continuation of and claims priority to U.S.
patent application Ser. No. 13/475,334, filed May 18, 2012, now
U.S. Pat. No. 8,982,068, issued Mar. 17, 2015, entitled "Multiple
Actuation Handheld Device With First And Second Haptic Actuator,"
which is a continuation of and claims priority to U.S. patent
application Ser. No. 12/237,334, filed Sep. 24, 2008, now U.S. Pat.
No. 8,749,495, issued Jun. 10, 2014, entitled "Multiple Actuation
Handheld Device," the entirety of all of which are hereby
incorporated by reference.
TECHNICAL FIELD
[0002] The present disclosure relates to a multiple actuation
handheld device.
BACKGROUND
[0003] Mobile gaming has become increasingly popular with the
improvements Internet speed as well as more sophisticated mobile
devices. Such portable mobile devices include smart phones (e.g.
Blackberry.RTM., iPhone.RTM.) as well as mobile video game consoles
(e.g. Playstation.RTM. Portable, Nintendo DS Lite.RTM.).
[0004] Additionally, many of the existing mobile devices
incorporate one or more touch screens through which the user
interacts with an avatar or other object while playing a game.
However, none of the existing mobile devices are capable of
outputting haptics through the touch screen as the user is playing
a game on the device.
OVERVIEW
[0005] A device includes a housing, and a processor that is coupled
to the housing. The processor is configured to process a software
program stored in a memory. A touch screen is coupled to the
housing and configured to display graphical objects, wherein a
sensor signal associated with a user's interaction with the touch
screen is provided to the processor. A first actuator is coupled to
the touch screen and is positioned within the housing. The first
actuator is configured to output a first haptic effect to the touch
screen upon receiving a first activating signal from the processor.
A second actuator is coupled to the housing and is configured to
output a second haptic effect to the housing upon receiving a
second activating signal from the processor. The first activating
signal is associated with a foreground event and the second
activating signal is associated with a background event occurring
in the software program.
[0006] A device comprises a housing and a processor coupled to the
housing. The processor is configured to process a software program
stored in a memory. A touch screen is coupled to the housing, and
the touch screen is configured to display graphical objects,
wherein a sensor signal associated with a user's interaction with
the touch screen is provided to the processor. An actuator is
coupled to the touch screen and is positioned within the housing.
The actuator is configured to output a first haptic effect to the
touch screen upon receiving a first activating signal from the
processor which is associated with a foreground event occurring in
the software program. The actuator is configured to output a second
haptic effect to the housing upon receiving a second activating
signal from the processor which is associated with a background
event occurring in the software program.
[0007] A method for operating a mobile device comprises displaying
a graphical environment through a touch screen of a housing of the
mobile device. The method includes sensing a position of a user's
input in contact with the touch screen. The method includes
identifying a haptic event occurring in association with an
interaction within the graphical environment and transmitting an
activating signal to an actuator, wherein the actuator imparts a
haptic effect that corresponds to the haptic event to the touch
screen upon determining that the haptic event is a foreground
event.
[0008] A method for operating a mobile device comprises displaying
a graphical environment through a touch screen of a housing of the
mobile device. The method includes sensing a position of a user's
input in contact with the touch screen and identifying a haptic
event occurring in association with an interaction within the
graphical environment. The method includes transmitting an
activating signal to an actuator, wherein the actuator imparts a
haptic effect that corresponds to the haptic event to the housing
upon determining that the haptic event is a background event.
[0009] In an embodiment, the foreground event occurs as a result of
the user's interaction with the touch screen. In an embodiment, the
first actuator only outputs the first haptic effect when a sensor
coupled to the touch screen indicates the user touching the touch
screen. In an embodiment, the first actuator outputs the first
haptic effect and the second actuator outputs the second haptic
effect at substantially the same time or different times.
BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The accompanying drawings, which are incorporated into and
constitute a part of this specification, illustrate one or more
examples of embodiments and, together with the description of
example embodiments, serve to explain the principles and
implementations of the embodiments.
[0011] In the drawings:
[0012] FIG. 1 illustrates a perspective view of a mobile gaming
device in accordance with the prior art.
[0013] FIG. 2 illustrates a block diagram of a dual actuating touch
screen gaming device in accordance with an embodiment.
[0014] FIG. 3 illustrates a view of a touch screen having virtual
strings displayed thereon in accordance with an embodiment.
[0015] FIG. 4 illustrates a flow chart of the operation of the
device in accordance with an embodiment.
DESCRIPTION OF EXAMPLE EMBODIMENTS
[0016] Example embodiments are described herein in the context of a
multiple actuation handheld mobile device. Those of ordinary skill
in the art will realize that the following description is
illustrative only and is not intended to be in any way limiting.
Other embodiments will readily suggest themselves to such skilled
persons having the benefit of this disclosure. Reference will now
be made in detail to implementations of the example embodiments as
illustrated in the accompanying drawings. The same reference
indicators will be used throughout the drawings and the following
description to refer to the same or like items.
[0017] In the interest of clarity, not all of the routine features
of the implementations described herein are shown and described. It
will, of course, be appreciated that in the development of any such
actual implementation, numerous implementation-specific decisions
must be made in order to achieve the developer's specific goals,
such as compliance with application- and business-related
constraints, and that these specific goals will vary from one
implementation to another and from one developer to another.
Moreover, it will be appreciated that such a development effort
might be complex and time-consuming, but would nevertheless be a
routine undertaking of engineering for those of ordinary skill in
the art having the benefit of this disclosure.
[0018] In accordance with this disclosure, the components, process
steps, and/or data structures described herein may be implemented
using various types of operating systems, computing platforms,
computer programs, and/or general purpose machines. In addition,
those of ordinary skill in the art will recognize that devices of a
less general purpose nature, such as hardwired devices, field
programmable gate arrays (FPGAs), application specific integrated
circuits (ASICs), or the like, may also be used without departing
from the scope and spirit of the inventive concepts disclosed
herein. It is understood that the phrase "an embodiment"
encompasses more than one embodiment and is thus not limited to
only one embodiment. Where a method comprising a series of process
steps is implemented by a computer or a machine and those process
steps can be stored as a series of instructions readable by the
machine, they may be stored on a tangible medium such as a computer
memory device (e.g., ROM (Read Only Memory), PROM (Programmable
Read Only Memory), EEPROM (Electrically Eraseable Programmable Read
Only Memory), FLASH Memory, Jump Drive, and the like), magnetic
storage medium (e.g., tape, magnetic disk drive, and the like),
optical storage medium (e.g., CD-ROM, DVD-ROM, paper card, paper
tape and the like) and other types of program memory.
[0019] FIG. 1 illustrates a perspective view of a mobile gaming
device in accordance with the prior art. As shown in FIG. 1, the
mobile gaming device 10 includes a bottom portion 12 and a top
portion 14 pivotably attached to the bottom portion 12 at the hinge
16. The device 10 includes an interactive touch screen 18 in the
bottom portion 12 with which the user is able to select or interact
with displayed graphical items, objects or characters by physically
touching the screen 18 using a finger, stylus or guitar pick. The
device 10 also includes a display 20 in the top portion 14 which
also displays items, objects, characters or other information to
the user. Although not necessary, the display 20 in the top portion
14 may also be a touch screen in which the user is able to interact
with the displayed items on the display 20 by physically touching
the display 20.
[0020] In addition, the device 10 may include a directional pad 26
and/or selection buttons. A stylus 24 may be used with the touch
screen 18 to interact with the displayed items. Although the stylus
24 has a pen shaped design in FIG. 1, the stylus 24 may have other
designs, such as a guitar pick, as discussed in more detail below.
It should be noted that although the description above and
following discussion are directed to the mobile gaming apparatus in
FIG. 1, it should be noted that the dual actuation system can be
applied in a smart phone or other type of electronic device with
gaming capabilities that are not illustrated herein.
[0021] FIG. 2 illustrates a block diagram of a multiple actuating
handheld device in accordance with an embodiment. As shown in FIG.
2, the device 100 includes a housing 102, one or more touch screens
104, one or more actuators 106 coupled to the touch screen 104, and
one or more processors 110 coupled to the touch screen 104 and the
actuator 106. A memory 114 is preferably coupled to the processor
110, whereby the processor 110 is able to store and retrieve
information from the memory 114. Such information may include, but
is not limited to, haptic effect profiles, game data, software
data, etc.
[0022] In addition, as shown in FIG. 2, the device 100 includes a
sensor 108 coupled to the touch screen 104, whereby the sensor 108
monitors the position and/or movement of the user's finger, stylus
24 or other input means along the touch screen 104. The sensor 108
preferably provides sensor signals to the processor 110 to indicate
the pressure, position and/or movement of the stylus 24, whereby
the processor 110 running the software program updates the display
shown through the touch screen 104 in response thereto. In an
embodiment, the touch screen 104 incorporates the sensor 108
therein as an integral component, and thus the sensor 108 is not a
separate component. However, for purposes of discussion, the sensor
108 is referred to herein as a separate component. Touch screen
technology is well known in the art and is thus not described in
detail herein.
[0023] The actuator 106 (hereinafter referred to as screen
actuator) is configured to output one or more haptic effects to the
touch screen 104 upon receiving an activating signal from the
processor 110. It is contemplated that the actuator 106 may be
configured to output one or more haptic effects to more than one
touch screen 104 on a device (if available). In an embodiment, the
actuator 106 is in direct contact with the touch screen 104,
although it is not necessary as long as the haptic effects output
by the actuator 106 are localized to the touch screen 104 and thus
transmitted therethrough. In other words, such haptic effects from
the screen actuator 104 are preferably localized to the touch
screen 104 and are not significantly felt in the housing. Thus, it
is preferred that the user's hand holding the housing 102 will not
feel any, or at least not a significant amount, of the haptic
effect when the screen actuator 106 is activated. However, the
user's will feel the haptic effect from the actuator 106 when
directly touching the touch screen 104 via the fingers or the
stylus 24 In an embodiment, the device can include a plurality of
actuators for multiple touch screens whereby one or more actuators
in the plurality can output haptic effects to the appropriate touch
screen. For instance, a user playing a device with two or more
touch screens can move a graphical character from one touch screen
to another. One or more actuators coupled to the first touch screen
can output a haptic effect to that touch screen while the character
is displayed thereon. As the user moves the character from the
first touch screen to a second touch screen, one or more actuators
coupled to the second touch screen can output a haptic effect to
the second touch screen, thereby continuing the haptic experience
to the user as the character is moved between the screens. It is
contemplated that the haptic effects output to the first and second
touch screens may be same of different effects and/or at same or
different times.
[0024] In an embodiment, the device 100 preferably includes one or
more actuators 112 coupled to the processor which outputs a haptic
effect upon receiving an activating signal from the processor 110.
The actuator 112 (hereinafter referred to as housing actuator) is
preferably coupled to an interior of the housing 102 such that the
haptic effect output by that actuator 112 is transmitted through
the housing 102 instead of the touch screen 104. In an embodiment,
the housing actuator 112 is part of an external device or
peripheral that is externally mounted to the housing 102. The
haptic effect output by the housing actuator 112 is felt through
the user's hand holding the device 100 rather than the user's hand
interacting with the touch screen 104. Although it is disclosed
that the actuator 112 outputs the haptic effect through the housing
102, it is possible that the haptic effect profile (e.g. magnitude,
duration, and/or frequency) is modulated that the haptic effect is
also felt through the touch screen 104. It should also be noted
that although two actuators are described (e.g. screen actuator 106
and housing actuator 112), it is possible that more than two
actuators be used in the device. In an embodiment, only one
actuator can be used in the device, whereby the actuator is coupled
to the housing 102 and the touch screen 104 and thus can
selectively output haptic effects to the touch screen 104 and/or
the housing 102 when desired.
[0025] The above mentioned actuators can be of various types
including, but not limited to, eccentric rotational mass (ERM)
actuators, linear resonant actuators (LRA), piezoelectric actuator,
voice coil actuator, electro-active polymer (EAP) actuators, memory
shape alloys, pager motors, moving magnet actuators, E-core
actuators etc. Some examples of actuator assemblies capable of
being used with the touch screen are described in U.S. Pat. No.
7,148,875 as well as U.S. patent application Ser. Nos. 10/919,798
filed Aug. 17, 2004, and Ser. No. 11/128,717 filed May 12,
2005.
[0026] The actuators 106 and 112 are capable of outputting the same
or different haptic effects at same or different times. The dual
actuation capability of the device allows for a wider range of
haptic sensations and adds realism to the game being played as will
be discussed below. In particular, the device is capable of
utilizing the screen actuator 106 and the housing actuator 112 to
selectively output haptic effects which provide the user with a
more pleasurable (or immersed) experience during game play.
[0027] The actuators 106 and 112 output their respective haptic
effects in response to a haptic event occurring in the graphical
environment. The haptic event is referred to herein as any
interaction, action, collision, or other event which occurs during
operation of the device which can potentially have a haptic effect
associated with it that is then output to the user in the form of
the haptic effect. For example purposes, a haptic event may occur
when a graphical vehicle the user is controlling experiences wind
turbulence during game play, whereby an example haptic effect
associated with the haptic event is a vibration. Another example is
that a haptic event may occur when a missile collides with the
user's character in the game, whereby an example haptic effect
associated with the haptic event is a jolt or pulse. Haptic events
may also be indirectly associated with the game play, but
nonetheless provides the user with important device information
while the user is playing a game (e.g receiving a text message,
completion of a song download, battery level low, etc.).
[0028] In an embodiment, device 100 preferably takes into account
actions occurring within the video game along with the user's
sensed inputs to determine whether the screen actuator 106 or the
housing actuator 112 will be activated to output haptic effects to
the user. In particular, the device 100 determines which actuator
to activate based on whether haptic event is considered a
foreground event or a background event. Whether the haptic event is
a foreground or background event can be written in the software
code, wherein the processor 110 upon processing the software code,
automatically instructs which actuator to operate. It is also
contemplated that the processor 110 determines whether the haptic
event is a foreground or background event based on whether the
haptic event is a direct result of the user's input into the touch
screen 104. If so, the haptic effect may have a more pronounced
effect if felt in the hand interacting with the touch screen.
[0029] For instance, an interaction between the user's stylus 24
and a graphical object on the touch screen 104, such as dragging a
video game character through sand by moving the stylus 24 across
the touch screen 104, may be considered a foreground event. In
other words, the foreground haptic effects are preferably best
utilized with interactions in which the user is in contact with the
touch screen 104. In operation, the sensor 108 monitors the user
moving the character slowly through sand and provides the processor
such sensing information. In response, the processor 110 in
conjunction with the software program outputs an activating signal
associated with a haptic effect (such as a vibratory textural
haptic effect) to the screen actuator 106, whereby the activating
signal preferably carries information of a haptic effect profile
(e.g. magnitude, frequency, duration). The screen actuator 106 upon
receiving the activating signal outputs the haptic effect to the
touch screen 104, whereby the haptic effect is felt directly by the
user's finger or a hand holding the stylus 24.
[0030] In an embodiment, the device 100 substantially
simultaneously updates the outputted haptic effect based on the
user's input. For instance, in the example, the screen actuator 106
will receive signals from the processor 108 to increase the
frequency and/or vibration in response to the sensor 108 monitoring
the user's stylus 24 moving across the touch screen 104 at an
increasing rate. In this instance, the touch screen 104 may begin
to vibrate faster and/or with greater intensity to magnify the
haptic effect, thereby giving the user a realistic haptic response
during game play.
[0031] In an embodiment, the screen actuator 106 may output an
entirely different haptic effect in response to the sensed movement
of the user's input into the touch screen 104. For example, as
stated above, a vibration may be output by the screen actuator 106
in response to the user moving the character across the screen in
sand. In the example, the screen actuator 106 is instructed to
output a jolt, bump or pop to the touch screen 106 upon the user's
character colliding with a graphical obstacle in the sand. It
should be noted that either or both of the screen and housing
actuators are capable of outputting haptic effects including, but
not limited to, time-based effects such as vibrations and pulses as
well as position-based effects such as textures, detents and
bumps.
[0032] In an embodiment, the device 100 is preferably capable of
outputting haptic effects to the housing. As stated above, the
housing actuator 112 is configured to output haptic effects to the
housing 102 of the device 100, whereby the haptic effects are
preferably felt on the hand holding the device 100. It is
preferred, although not necessary, that the housing actuator 112
output haptic effects for haptic events which are considered
background events. It is preferred that the processor 110
determines whether the haptic event will be a foreground and/or
background event. In an embodiment, the determination of whether a
haptic event is a background and/or foreground event is written in
the software code, whereby the processor 110 automatically selects
which actuator to activate based on the software code. An example
of a background event may be an interaction between the two objects
displayed on the touch screen 104, such as a missile hitting the
graphical character. The processor 110 processes such information
and transmits an activating signal to the housing actuator 112
having the requisite haptic profile (e.g. magnitude, frequency,
duration). The housing actuator 112 upon receiving such signal
outputs the corresponding haptic effect to the housing 102. The
user is then able to feel the haptic effect in the hand that is
holding the device 102, thereby providing the user with a
pleasurable and realistic experience. It is contemplated that the
haptic event be considered a background event when the haptic event
is user-independent, such as pulses when bullets are fired at the
user's character. In contrast, the bullets hitting the character
may be a considered by the processor as a foreground event, thereby
causing the screen actuator 106 to output one or more haptic
effects to the touch screen 104. Haptic events which are based on
background routines in the software or not directly related to the
game being played (e.g. incoming text message, download complete)
may be automatically considered by the processor 110 as a
background event.
[0033] An example is provided in which a graphical gear shifter is
displayed through a touch screen 104, whereby the gear shift is
able to be shifted between two or more positions. In the example,
the displayed gear shifter may be operated by the user by moving
the shifter via a finger or stylus along the touch screen 104. In
the example, the processor 110 is provided sensor information that
the user has shifted the displayed shifter and determines that a
foreground event and a background event have occurred. The
processor 110 preferably updates the gear shifter displayed on the
touch screen to show the new position of the shifter. The processor
110 thereby outputs an activating signal to the screen actuator 106
to output a haptic effect to the touch screen 104 and an activating
signal to the housing actuator 112 to output a haptic effect to the
housing. In response, the actuator 106 outputs the haptic effect to
the touch screen 104, whereby the haptic effect is felt in the
user's contacting finger or stylus. The haptic effect output by the
screen actuator 106 is preferably a jolt or click in response to
the virtual shifter being shifted ("foreground event"). In
addition, the housing actuator 112 outputs a low frequency and/or
magnitude vibration which is felt in the user's hand which is
holding the device 100 ("background event"). In the example, as the
user continues to up-shift as the vehicle goes faster during game
play, the screen actuator 106 outputs a click or jolt and the
housing actuator 112 outputs a higher frequency and/or magnitude
vibration. In contrast, if the user continues were down-shift, the
screen actuator 106 would output a click or jolt whereas the
housing actuator 112 would output a lower frequency and/or
magnitude vibration.
[0034] It should be noted that the above examples are not limiting
and it is contemplated that the processor 110 along with the video
game software processes which events and interactions in the game
are to be designated as foreground events and background events.
However, as stated above, it is preferred, though not necessary,
that the foreground events be designated to actions which are felt
while the user hand or stylus is in contact with the touch screen
104. It is also contemplated that the foreground events be
dedicated to the housing actuator 112 and the background events be
dedicated to the screen actuator 106.
[0035] It is also contemplated that both actuators 106, 112 are
capable of outputting same or different haptic effects at least
partially at the same time in response to an interaction within the
game. For example, the processor 110, as instructed by the
software, may activate the screen actuator 106, which outputs the
haptic effect when the touch screen 104 shows the character
colliding with a stack of barrels. Thereafter, as the barrels fall
to the ground, the processor 110, as instructed by the software,
determines that this is a background haptic event and activates the
housing actuator 112 to output a haptic effect that corresponds to
the event to the housing 102.
[0036] In an embodiment, the processor 110 may determine which
actuator is to output the haptic effect based on whether the sensor
108 indicates that the user's hand or stylus is in contact with the
touch screen 104. Therefore, if the action is such that it could be
either or both of the actuators 106, 112, the processor 110 may
activate just the housing actuator 112 if it is determined that the
user is not contacting the touch screen 104. This process may be
used to conserve battery life of the device 100 if the action is
indeed a foreground event, but it is determined that the user is
not contacting the touch screen 104.
[0037] In another embodiment, the device is configured such that
the foreground and background events are clearly distinguishable
based on the type of game that is played. One example is Guitar
Hero in which the device 100 is used as a mock guitar that allows
the user to to simulate the playing of music, represented on-screen
20 by colored notes that correspond to fret buttons on the
controller. A brief discussion of the Guitar Hero game is provided
for clarity of use with the device 100.
[0038] In a game such as Guitar Hero, the screen actuator 106 is
able to output haptic effects to the touch screen 104 as the user
strums the displayed strings to give the user the feeling of
striking real strings. The touch screen 104 may include one or more
screen actuators 106 coupled thereto which provide the user with
haptic effects isolated to the touch screen 104 to simulate the
feel of playing across the virtual displayed strings. In an
embodiment shown in FIG. 3, the touch screen 104 is shown
displaying four virtual strings 116, 118, 120 and 122, although a
greater or lesser number of displayed strings are contemplated. The
touch screen 104 tracks the position of the user's finger or stylus
as it moves along the surface of the touch screen 104 and provides
positional, velocity and/or acceleration information to the
processor 110. It is contemplated that the screen actautor 106
output a jolt or pop haptic effect once the user's finger comes
into contact with string 116, whereby the haptic effect signifies
that the user has struck one or more strings. The same or different
haptic effect could apply for strings 118, 120 and/or 122. In the
embodiment, the screen actuator 106 outputs a series of jolts or
pops to signify that the user has struck one or more strings,
whereby each jolt or pop represents a string being struck. In
particular, the screen actuator 106 outputs a jolt or pop as the
sensor 108 tracks the user moving the user's finger or stylus
across each displayed string on the touch screen 104. It is
contemplated that direction of the movement of the finger is
tracked such that a first haptic effect is output when the finger
moves across one or more strings from left to right (represented by
arrow 98A) and a second haptic effect is output when the finger
moves across the one or more strings from right to left
(represented by arrow 98B).
[0039] Additionally or alternatively, the device may be configured
to output a first haptic effect when the user's finger is on the
displayed string (as shown at string 116), whereas a second haptic
effect is then output when the finger (shown as 99B) is dragged a
predetermined distance away from the string 116 to simulate the
release of the string 116. In an embodiment a time offset delay
(e,g, 50 ms or other time duration) is preferably added to the
playback of the second haptic effect to separate the two haptic
effects and prevent them from feeling blended. The second haptic
effect can be a jolt or click felt on the touch screen 104 (i.e.
string striking back to its default position) and/or a vibration to
the housing 102 (i.e. vibrational hum felt on the guitar body).
This provides a more distinct feel and creates the experience of
the same type of time lag that occurs when a real guitar string
returns to its original position after being released.
[0040] Additionally or alternatively, the device may utilize
pressure sensor information from the touch screen 104 to produce
the second haptic effect. For instance, once the first haptic
effect is output (the finger has moved across string 116 toward
position 99B), the actuator 106 may wait to output the second
haptic effect until the user's finger has reached position 99B or
has begun to gradually relieve pressure off the touch screen 104
(thus gradually releasing the virtual string). In particular, the
sensor 108 provides the processor 110 with information regarding
the amount of pressure applied to the touch screen 104, whereby the
processor 110 outputs the activating signal upon the pressure data
passing a threshold value. It is preferred, however, that the
second haptic effect occurs while the user's finger is still on the
screen 104 to ensure that the haptic effect is felt by the user. It
should be noted that the first and second haptic effects described
above can be the same as one another or be different from one
another.
[0041] Additionally or alternatively, the housing actuator 112 may
also be activated along with the screen actuator 106, but outputs a
different or varied haptic effect from that output by the screen
actuator 106. In particular, the housing actuator 112 may receive a
signal to instruct it to output a lower frequency vibration to the
housing. The vibration can mimic the audio of the notes being
played or just provide a realistic feeling of a guitar being played
(i.e. a chord being strummed). In an embodiment, the haptic effect
output by the housing actuator 112 may start out with a low
amplitude and increase in magnitude as the user continues to get
the timing right for the game in an effort of simulating the `flow`
experience of playing a guitar. Alternatively if the user's timing
is off, a discordant haptic effect could be output by any or all of
the actuators (screen actuator as well as housing actuator) as an
indication to the user. In an embodiment, the haptic effects output
by the screen and housing actuators 106, 112 may vary depending on
the type of guitar selected by the user in the video game (e.g.
greater vibrations in the body for "hollow body" guitars as opposed
to more subtle vibrations in the body of a "solid body"
guitar.)
[0042] In an embodiment, the screen and/or housing actuators 106,
112 may be instructed by the processor 110 to output haptic
confirmation of button presses or strum events when the user
successfully strikes the notes during play. In other words, haptic
effects would be output by the actuators 106, 112 to provide
confirmation of button presses or strumming effects when the user
successfully depresses the correct button on the controller 200
with the button displayed on the screen 20.
[0043] In an embodiment, the device may be configured to output one
or more haptic effects from either or both actuators 106, 112 to
indicate that the user has successfully depressed in a
predetermined number of correct notes. For example, in the game
there is the concept of "note streaks" when the user successfully
depresses a predetermined number of consecutive correct notes. The
device may output one or more haptic effects from either or both
actuators 106, 112 to reward the player after a note streak
milestone is achieved. A haptic effect could also be output by
either or both actuators 106, 112 that increases in magnitude as
the user continues to hit more consecutive correct notes, whereby
the increased magnitude is felt in the housing 102 or touch screen.
As soon as the user misses a note the effect resets and the user
knows that the effect streak has ended. Alternatively, the device
may be configured to output another haptic effect when the user
depresses one or more incorrect buttons.
[0044] One gaming concept is head-to-head play or "battle mode,"
whereby two players compete against one another to successfully hit
the most number of notes in a given time. A feature in the battle
mode is for one player to launch distraction items that distract
the other player during the competition. For example, in the battle
mode, one distraction item when launched causes the other player's
screen to catch fire, thereby preventing the user from seeing the
notes for a given amount of time and thus inhibiting their ability
to successfully hit notes during that time. The distraction items
can be used between devices 100, whereby launched distraction items
cause the recipient's device 100 to uncontrollably vibrate in the
touch screen 106 and/or housing 102 or output haptic effects which
cause an irregular beat or beats not in sync with the notes
displayed on the screen, or temporarily disable haptic effects
altogether.
[0045] FIG. 4 illustrates a flow chart of the operation of the dual
actuating device in accordance with an embodiment. As shown in FIG.
4, the device 100 runs a software program which allows a
selectively interactive graphical virtual environment to be
displayed on one or more display screens. In particular, the device
displays the graphical environment via the touch screen display as
in 300. A sensor (either separate from or integrated with the touch
screen) continuously monitors inputs by the user as the user
touches the touch screen by a stylus or by using a finger as in
302. Upon a haptic event occurring in the graphical environment,
the processor 110 of the device determines whether a haptic effect
should be output, as in 304. This would depend on the software run
by the processor and whether the software provides for a haptic
effect to be applied for that particular haptic event. If so, the
processor and/or software determines whether the associated haptic
effect should be a foreground event (as in 306) or a background
event (as in 312).
[0046] If the haptic effect is to be considered a foreground event
306, an optional step would be to determine whether the user's
finger or stylus is in contact with the touchscreen (as in 310). As
stated above, determining this could potentially conserve battery
power in the device 100 by only outputting haptic effects to the
touch screen 104 when it is ensured that the haptic effect would be
felt by the user. Alternatively, upon determining that the user is
not in contact with the touch screen 104, the processor may be
configured to instruct the housing actuator 112 to instead output
the haptic effect to the housing. If it is determined that the user
is in contact with the touch screen 104, the activating signal is
sent from the processor to the screen actuator 106 to output the
desired haptic effect to the touch screen 104. In an embodiment, if
it is determined that the user's finger or stylus is not in contact
with the touch screen, it is determined whether the haptic event
would be considered a background event (as in 312). If the haptic
event is not considered a background event, the device 100
preferably does not instruct any of the actuators to output a
haptic effect, and the device 100 continues to sense input by the
user into the touch screen 104 (302). However, if the haptic event
is considered to be a background event, the processor transmits an
activating signal to the housing actuator 112 to output a haptic
effect to the housing 102 (316).
[0047] While embodiments and applications have been shown and
described, it would be apparent to those skilled in the art having
the benefit of this disclosure that many more modifications than
mentioned above are possible without departing from the inventive
concepts disclosed herein. The invention, therefore, is not to be
restricted except in the spirit of the appended claims.
* * * * *