U.S. patent application number 15/895690 was filed with the patent office on 2018-06-21 for automatic fitting of haptic effects.
The applicant listed for this patent is Immersion Corporation. Invention is credited to Juan Manuel CRUZ-HERNANDEZ, Danny GRANT, Christopher J. ULLRICH, Victor Aaron VIEGAS.
Application Number | 20180174408 15/895690 |
Document ID | / |
Family ID | 50112724 |
Filed Date | 2018-06-21 |
United States Patent
Application |
20180174408 |
Kind Code |
A1 |
ULLRICH; Christopher J. ; et
al. |
June 21, 2018 |
AUTOMATIC FITTING OF HAPTIC EFFECTS
Abstract
A system is provided that automatically generates one or more
haptic effects from source data, such as audio source data. The
system fits the one or more haptic effects to the source data by
analyzing the source data and identifying one or more haptic
effects that are the most similar to the source data. The system
matches the identified one or more haptic effects with the source
data. The system subsequently outputs the identified one or more
haptic effects.
Inventors: |
ULLRICH; Christopher J.;
(Ventura, CA) ; GRANT; Danny; (Laval, CA) ;
VIEGAS; Victor Aaron; (Atherton, CA) ;
CRUZ-HERNANDEZ; Juan Manuel; (Montreal, CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Immersion Corporation |
San Jose |
CA |
US |
|
|
Family ID: |
50112724 |
Appl. No.: |
15/895690 |
Filed: |
February 13, 2018 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
15240560 |
Aug 18, 2016 |
9905090 |
|
|
15895690 |
|
|
|
|
14271536 |
May 7, 2014 |
9449043 |
|
|
15240560 |
|
|
|
|
13788487 |
Mar 7, 2013 |
8754758 |
|
|
14271536 |
|
|
|
|
13785166 |
Mar 5, 2013 |
8754757 |
|
|
13788487 |
|
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06F 2203/013 20130101;
G06F 16/683 20190101; A63F 13/285 20140902; A63F 13/424 20140902;
G08B 6/00 20130101; G06F 3/016 20130101; G06F 16/24 20190101; G06F
16/24568 20190101; G06F 2203/014 20130101; G06F 3/165 20130101 |
International
Class: |
G08B 6/00 20060101
G08B006/00; G06F 17/30 20060101 G06F017/30; A63F 13/285 20060101
A63F013/285; G06F 3/01 20060101 G06F003/01; G06F 3/16 20060101
G06F003/16; A63F 13/424 20060101 A63F013/424 |
Claims
1-37. (canceled)
38. A non-transitory computer-readable medium having instructions
stored thereon that, when executed by a processor, cause the
processor to receive source data; to extract a feature of the
source data; to compare the source data with each template of a
plurality of templates by comparing the feature extracted from the
source data with a respective template feature of each template of
the plurality of templates, wherein each template of the plurality
of templates further identifies a respective haptic effect to be
output; to select a template from the plurality of templates based
on the comparison; and to output the respective haptic effect
identified by the template that is selected.
39. The non-transitory computer-readable medium of claim 38,
wherein the source data includes metadata that has feature
information describing the feature of the source data.
40. The non-transitory computer-readable medium of claim 39,
wherein the metadata includes information describing a source from
which the source data is captured or how the source data is
captured.
41. The non-transitory computer-readable medium of claim 40,
wherein the source data is captured by a sensor, and wherein the
metadata describes a position of the sensor.
42. The non-transitory computer-readable medium of claim 40,
wherein the source data is captured by a sensor, and wherein the
metadata describes an orientation of the sensor.
43. The non-transitory computer-readable medium of claim 38,
wherein the feature extracted from the source data is a
characteristic of the source data, the characteristic being at
least one of a duration, frequency, or envelope of the source
data.
44. The non-transitory computer-readable medium of claim 38,
wherein the feature extracted from the source data is a transition
from the source data having a first characteristic value to the
source data having a second characteristic value.
45. The non-transitory computer-readable medium of claim 44,
wherein the feature is a transition from the source data having a
first frequency to the source data having a second frequency.
46. The non-transitory computer-readable medium of claim 38,
wherein the instructions, when executed by the processor, cause the
processor to perform the selecting of the template by selecting a
template with a template feature that is most similar to the
feature extracted from the source data.
47. The non-transitory computer-readable medium of claim 38,
wherein respective template features of the plurality of templates
describe respective data patterns for identifying a plurality of
haptic-related events.
48. The non-transitory computer-readable medium of claim 47,
wherein the source data comprises audio data, and wherein the
plurality of templates comprises sound templates, and wherein
respective template features of the sound templates describe
respective sound patterns for identifying the plurality of
haptic-related events.
49. The non-transitory computer-readable medium of claim 48,
wherein each of the sound patterns identifies at least one of a
gunshot sound, an explosion sound, and a car crash sound.
50. The non-transitory computer-readable medium of claim 38,
wherein the source data comprises video source data, and wherein
the plurality of templates comprise video templates.
51. The non-transitory computer-readable medium of claim 38,
wherein the source data comprises acceleration source data, and
wherein the plurality of templates comprise acceleration
templates.
52. The non-transitory computer-readable medium of claim 38,
wherein the source data comprises multi-modal source data.
53. The non-transitory computer-readable medium of claim 38,
wherein the instructions, when executed by the processor, further
cause the processor to optimize the haptic effect by adjusting a
haptic parameter value of the haptic effect to be more similar to
the feature extracted from the source data than if the haptic
parameter value was not adjusted.
54. A computer-implemented method, comprising: receiving source
data; extracting a feature of the source data; comparing the source
data with each template of a plurality of templates by comparing
the feature extracted from the source data with a respective
template feature of each template of the plurality of templates,
wherein each template of the plurality of templates further
identifies a respective haptic effect to be output; selecting a
template from the plurality of templates based on the comparison;
and outputting the respective haptic effect identified by the
template that is selected.
55. The method of claim 54, wherein the source data includes
metadata describing a source from which the source data is captured
or how the source data is captured.
56. The method of claim 55, wherein the source data is captured by
a sensor, and wherein the metadata describes a position or
orientation of the sensor.
57. A computer-implemented method, comprising: receiving source
data that includes metadata, wherein the metadata includes source
information describing a feature of the source data; extracting the
feature of the source data; comparing the source data with each
template of a plurality of templates by comparing the feature
extracted from the source data with a respective template feature
of each template of the plurality of templates, wherein each
template of the plurality of templates further identifies a
respective haptic effect to be output; selecting a template from
the plurality of templates based on the comparison; and outputting
the respective haptic effect identified by the template that is
selected.
Description
FIELD
[0001] One embodiment is directed generally to haptic effects, and
more particularly, to a device that produces haptic effects in
association with other related output.
BACKGROUND
[0002] Haptics is a tactile and force feedback technology that
takes advantage of a user's sense of touch by applying haptic
feedback effects (i.e., "haptic effects"), such as forces,
vibrations, and motions, to the user. Devices, such as mobile
devices, touchscreen devices, and personal computers, can be
configured to generate haptic effects. In general, calls to
embedded hardware capable of generating haptic effects (such as
actuators) can be programmed within an operating system ("OS") of
the device. These calls specify which haptic effect to play. For
example, when a user interacts with the device using, for example,
a button, touchscreen, lever, joystick, wheel, or some other
control, the OS of the device can send a play command through
control circuitry to the embedded hardware. The embedded hardware
then produces the appropriate haptic effect.
[0003] Devices can be configured to coordinate the output of haptic
effects with the output of other content, such as games or other
media, so that the haptic effects are incorporated into the other
content. For example, in a gaming context, when a game is
developed, an audio effect developer can develop audio effects that
are associated with the game and represent an action occurring
within the game, such as machine gun fire, explosions, or car
crashes. Typically, haptic effects are added to the game late in
the game development process, such as when the game developer is
finishing development of the game application, or when the game
developer ports the finished game application to a new platform.
This generally results in the phenomena where haptic effects are
added after all the audio effects have been developed. Because
haptic effects are typically added so late in the process, it
generally falls on the haptic effect developer, or some other
developer, to make a decision regarding associating a haptic effect
with an audio effect. Further, an audio effect developer typically
does not have input regarding a selection of an appropriate haptic
effect for an audio effect. This can contribute to a degradation of
the quality of haptic effects that are ultimately incorporated into
the content. This quality degradation can be a barrier to
incorporating high-quality haptic effects into such content.
SUMMARY
[0004] One embodiment is a system that automatically fits a haptic
effect. The system receives source data, where the source data
includes one or more characteristics. The system further compares
the source data with one or more haptic primitives, where each
haptic primitive of the one or more haptic primitives includes one
or more haptic parameters. The system further selects one or more
haptic primitives from the one or more haptic primitives based on
the comparison. The system further outputs one or more haptic
effects based on the selected one or more haptic primitives.
[0005] Another embodiment is a system that automatically fits a
haptic effect. The system receives source data, where the source
data includes one or more features. The system further extracts one
or more features from the source data. The system further compares
the one or more extracted features with one or more templates,
where each template includes one or more template features and one
or more haptic effects. The system further selects one or more
templates from the one or more templates based on the comparison.
The system further selects the one or more haptic effects from the
one or more selected templates. The system further outputs the one
or more selected haptic effects.
BRIEF DESCRIPTION OF THE DRAWINGS
[0006] Further embodiments, details, advantages, and modifications
will become apparent from the following detailed description of the
preferred embodiments, which is to be taken in conjunction with the
accompanying drawings.
[0007] FIG. 1 illustrates a block diagram of a system in accordance
with one embodiment of the invention.
[0008] FIG. 2 illustrates a flow diagram of a framework for
generating and playing a haptic effect.
[0009] FIG. 3 illustrates a flow diagram of a selection of a haptic
primitive that is most similar to audio source data, according to
one embodiment of the invention.
[0010] FIG. 4 illustrates a flow diagram for an audio feature
extraction and a selection of a haptic effect based on the
extracted audio feature, according to one embodiment of the
invention.
[0011] FIG. 5 illustrates a flow diagram of the functionality of an
automatic haptic effect fitting module, according to one embodiment
of the invention.
[0012] FIG. 6 illustrates a flow diagram of the functionality of an
automatic haptic effect fitting module, according to another
embodiment of the invention.
DETAILED DESCRIPTION
[0013] One embodiment is a system that can automatically generate
one or more haptic effects given source data, such as audio source
data. In other words, the system can automatically convert received
source data into haptic information, where the received source data
can include data, such as audio data, video data, acceleration
data, or another type of data that can be captured with a sensor.
More specifically, the system can analyze the source data and
identify one or more haptic effects that are the most similar to
the source data. The system can then match the identified one or
more haptic effects with the source data. The system can
subsequently output the identified one or more haptic effects. The
source data can be stored in a storage, where the source data is
retrieved before the source data is automatically converted into
haptic information. Alternatively, the source data can be streamed
before the source data is automatically converted into haptic
information.
[0014] In one embodiment, the system can identify one or more
haptic primitives (described in greater detail below) that are the
most similar to the source data. The system can then select the one
or more haptic primitives and output one or more haptic effects
based on the one or more haptic primitives. The system can
optionally optimize the one or more haptic primitives to be more
similar to the source data. In another embodiment, the system can
identify one or more templates that are the most similar to the
source data. The system can then select the one or more haptic
effects that are associated with the one or more templates, and
output the one or more selected haptic effects. The system can
optionally optimize the one or more selected haptic effects to be
more similar to the source data. The source data can be stored in a
storage, where the source data is retrieved before the source data
is automatically converted into haptic information. Alternatively,
the source data can be streamed before the source data is
automatically converted into haptic information.
[0015] Thus, according to an embodiment, where content, such as a
video game or other type of media, is output, and where the content
includes data, such as audio data, video data, or acceleration
data, the system can automatically add one or more haptic effects
to the content, where each haptic effect "fits" the corresponding
data of the content. Thus, when the existing content is output,
haptic content can automatically be added to the existing content,
where the haptic content matches the existing content.
[0016] FIG. 1 illustrates a block diagram of a system 10 in
accordance with one embodiment of the invention. In one embodiment,
system 10 is part of a device, and system 10 provides an automatic
haptic effect fitting functionality for the device. In another
embodiment, system 10 is separate from the device, and remotely
provides the automatic haptic effect fitting functionality for the
device. Although shown as a single system, the functionality of
system 10 can be implemented as a distributed system. System 10
includes a bus 12 or other communication mechanism for
communicating information, and a processor 22 coupled to bus 12 for
processing information. Processor 22 may be any type of general or
specific purpose processor. System 10 further includes a memory 14
for storing information and instructions to be executed by
processor 22. Memory 14 can be comprised of any combination of
random access memory ("RAM"), read only memory ("ROM"), static
storage such as a magnetic or optical disk, or any other type of
computer-readable medium.
[0017] A computer-readable medium may be any available medium that
can be accessed by processor 22 and may include both a volatile and
nonvolatile medium, a removable and non-removable medium, a
communication medium, and a storage medium. A communication medium
may include computer readable instructions, data structures,
program modules or other data in a modulated data signal such as a
carrier wave or other transport mechanism, and may include any
other form of an information delivery medium known in the art. A
storage medium may include RAM, flash memory, ROM, erasable
programmable read-only memory ("EPROM"), electrically erasable
programmable read-only memory ("EEPROM"), registers, hard disk, a
removable disk, a compact disk read-only memory ("CD-ROM"), or any
other form of a storage medium known in the art.
[0018] In one embodiment, memory 14 stores software modules that
provide functionality when executed by processor 22. The modules
include an operating system 15 that provides operating system
functionality for system 10, as well as the rest of a device in one
embodiment. The modules further include an automatic haptic effect
fitting module 16 that automatically fits a haptic effect, as
disclosed in more detail below. In certain embodiments, automatic
haptic effect fitting module 16 can comprise a plurality of
modules, where each individual module provides specific individual
functionality for automatically fitting a haptic effect. System 10
will typically include one or more additional application modules
18 to include additional functionality, such as the Integrator.TM.
application by Immersion Corporation.
[0019] System 10, in embodiments that transmit and/or receive data
from remote sources, further includes a communication device 20,
such as a network interface card, to provide mobile wireless
network communication, such as infrared, radio, Wi-Fi, or cellular
network communication. In other embodiments, communication device
20 provides a wired network connection, such as an Ethernet
connection or a modem.
[0020] Processor 22 is further coupled via bus 12 to a display 24,
such as a Liquid Crystal Display ("LCD"), for displaying a
graphical representation or user interface to a user. The display
24 may be a touch-sensitive input device, such as a touchscreen,
configured to send and receive signals from processor 22, and may
be a multi-touch touchscreen. Processor 22 may be further coupled
to a keyboard or cursor control 28 that allows a user to interact
with system 10, such as a mouse or a stylus.
[0021] System 10, in one embodiment, further includes an actuator
26. Processor 22 may transmit a haptic signal associated with a
generated haptic effect to actuator 26, which in turn outputs
haptic effects such as vibrotactile haptic effects, electrostatic
friction haptic effects, or deformation haptic effects. Actuator 26
includes an actuator drive circuit. Actuator 26 may be, for
example, an electric motor, an electro-magnetic actuator, a voice
coil, a shape memory alloy, an electro-active polymer, a solenoid,
an eccentric rotating mass motor ("ERM"), a linear resonant
actuator ("LRA"), a piezoelectric actuator, a high bandwidth
actuator, an electroactive polymer ("EAP") actuator, an
electrostatic friction display, or an ultrasonic vibration
generator. In alternate embodiments, system 10 can include one or
more additional actuators, in addition to actuator 26 (not
illustrated in FIG. 1). Actuator 26 is an example of a haptic
output device, where a haptic output device is a device configured
to output haptic effects, such as vibrotactile haptic effects,
electrostatic friction haptic effects, or deformation haptic
effects, in response to a drive signal. In alternate embodiments,
actuator 26 can be replaced by some other type of haptic output
device. Further, in other alternate embodiments, system 10 may not
include actuator 26, and a separate device from system 10 includes
an actuator, or other haptic output device, that generates the
haptic effects, and system 10 sends generated haptic effect signals
to that device through communication device 20.
[0022] System 10 can further be operatively coupled to a database
30, where database 30 can be configured to store data used by
modules 16 and 18. Database 30 can be an operational database, an
analytical database, a data warehouse, a distributed database, an
end-user database, an external database, a navigational database,
an in-memory database, a document-oriented database, a real-time
database, a relational database, an object-oriented database, or
any other database known in the art.
[0023] In one embodiment, system 10 further includes one or more
speakers 32. Processor 22 may transmit an audio signal to speaker
32, which in turn outputs audio effects. Speaker 32 may be, for
example, a dynamic loudspeaker, an electrodynamic loudspeaker, a
piezoelectric loudspeaker, a magnetostrictive loudspeaker, an
electrostatic loudspeaker, a ribbon and planar magnetic
loudspeaker, a bending wave loudspeaker, a flat panel loudspeaker,
a heil air motion transducer, a plasma arc speaker, and a digital
loudspeaker.
[0024] System 10, in one embodiment, further includes a sensor 34.
Sensor 34 can be configured to detect a form of energy, or other
physical property, such as, but not limited to, acceleration, bio
signals, distance, flow, force/pressure/strain/bend, humidity,
linear position, orientation/inclination, radio frequency, rotary
position, rotary velocity, manipulation of a switch, temperature,
vibration, or visible light intensity. Sensor 34 can further be
configured to convert the detected energy, or other physical
property, into an electrical signal, or any signal that represents
virtual sensor information. Sensor 34 can be any device, such as,
but not limited to, an accelerometer, an electrocardiogram, an
electroencephalogram, an electromyograph, an electrooculogram, an
electropalatograph, a galvanic skin response sensor, a capacitive
sensor, a hall effect sensor, an infrared sensor, an ultrasonic
sensor, a pressure sensor, a fiber optic sensor, a flexion sensor
(or bend sensor), a force-sensitive resistor, a load cell, a
LuSense CPS.sup.2 155, a miniature pressure transducer, a piezo
sensor, a strain gage, a hygrometer, a linear position touch
sensor, a linear potentiometer (or slider), a linear variable
differential transformer, a compass, an inclinometer, a magnetic
tag (or radio frequency identification tag), a rotary encoder, a
rotary potentiometer, a gyroscope, an on-off switch, a temperature
sensor (such as a thermometer, thermocouple, resistance temperature
detector, thermistor, or temperature-transducing integrated
circuit), microphone, photometer, altimeter, bio monitor, or a
light-dependent resistor.
[0025] FIG. 2 illustrates a flow diagram of a framework for
generating and outputting a haptic effect, where the haptic effect
is generated based on an audio input, according to one embodiment
of the invention. In one embodiment, the functionality of FIG. 2,
as well as the functionalities of FIGS. 3, 4, 5, and 6, are each
implemented by software stored in memory or another
computer-readable or tangible medium, and executed by a processor.
In other embodiments, each functionality may be performed by
hardware (e.g., through the use of an application specific
integrated circuit ("ASIC"), a programmable gate array ("PGA"), a
field programmable gate array ("FPGA"), etc.), or any combination
of hardware and software. Furthermore, in alternate embodiments,
each functionality may be performed by hardware using analog
components.
[0026] FIG. 2 illustrates audio source data 210. According to the
embodiment, audio source data 210 includes audio data. In certain
embodiments, the audio data included within audio source data 210
can be audio data that is stored in either an audio file or an
audio signal. In an alternate embodiment, audio source data 210 can
be replaced by the audio file or the audio signal. In other
embodiments, the audio data included within audio source data 210
can be audio data that is streamed. Further, in the illustrated
embodiment, audio source data 210 is encoded in a pulse code
modulation ("PCM") format. In certain embodiments, audio source
data 210 can also be encoded in another type of format, such as a
Musical Instrument Digital Interface ("MIDI") format, or a MPEG-2
Audio Layer III ("MP3") format. In addition, in embodiments where
audio source data 210 is encoded in another type of format, audio
source data 210 can include audio data that can be decoded into a
PCM format. Further, in these embodiments, audio source data 210
can also include un-decoded data that can be used in generating a
haptic effect based on audio source data 210, as described below in
greater detail. For example, where audio source data 210 is encoded
in an MP3 format, audio source data 210 can include one or more
Fourier coefficients for the audio data, where the one or more
Fourier coefficients can be directly used to generate a haptic
effect based on audio source data 210. As another example, where
audio source data 210 is encoded in a MIDI format, audio source
data 210 can include a pre-existing set of metadata about the audio
data that can be directly used to generate a haptic effect based on
audio source data 210. Such metadata can include metadata related
to a source of the audio data, such as a position or orientation of
a sensor that captures the audio data. Such metadata can be used by
a matching algorithm to generate a haptic effect based on audio
source data 210, as discussed below in greater detail.
[0027] Further, in alternate embodiments, audio source data 210 can
be replaced by another type of source data that includes another
type of data, such as video source data that includes video data,
acceleration source data that includes acceleration data,
orientation source data that includes orientation data, ambient
light source data that includes ambient light data, or another type
of source data. An example of another type of source data is source
data that includes data that can be captured with a sensor.
Further, in some embodiments, audio source data 210 can be replaced
by source data that includes multi-modal data (i.e. data of two or
more modes, or types, such as audio data and video data).
[0028] At 220, source data 210 is automatically converted to haptic
information, such as haptic data or a haptic stream. According to
the embodiment, the haptic information can include either a single
haptic effect, or a plurality of haptic effects, that can be output
by a device. In certain embodiments, audio source data 210 can be
automatically converted to haptic information by comparing audio
source data 210 to a set of haptic primitives, and selecting at
least one haptic primitive that is or most similar to audio source
data 210, as is further described below in relation to FIG. 3. In
other embodiments, audio source data 210 can be automatically
converted to haptic information by comparing audio source data 210
to a set of templates, selecting at least one template that is most
similar to audio source data 210, and selecting at least one haptic
effect that is associated with the at least one selected template,
as is further described below in relation to FIG. 4.
[0029] At 230, the haptic information generated at 220 can be
encoded. The haptic information can be encoded according to any
haptic encoding technique known to one of ordinary skill in the
art. For example, the haptic information can be encoded using a
haptic effect signal. The haptic effect signal can subsequently be
persisted on a disk, memory, or other computer-readable storage
medium. As another example, the haptic information can be encoded
using a haptic effect file. The haptic effect file can have one of
many different formats. In certain embodiments, the haptic effect
file can have an extensible markup language ("XML") format, such as
an Immersion Vibration Source ("IVS") haptic effect file. In
certain other embodiments, the haptic effect file can have a binary
format, such as an Immersion Vibration Target ("IVT") haptic effect
file. The haptic information encoded at 230 can be further
compressed and/or included in one or more asset archives used in a
computer application.
[0030] At 240, the haptic information encoded at 230 can be
decoded. The haptic information can be decoded according to any
haptic decoding technique known to one of ordinary skill in the
art. By decoding the haptic information, the haptic information can
be converted from an encoded format, such as a haptic effect signal
or a haptic effect file, into a format where the haptic information
can be interpreted, and a single haptic effect, or a plurality of
haptic effects, can be output based on the haptic information.
[0031] At 250, the haptic information decoded at 240 can be output.
The haptic information can be output according to any haptic output
technique known to one of ordinary skill in the art. For example,
the haptic information can be output in the form of a single haptic
effect, or a plurality of haptic effects, that can be output by a
mobile device, a game pad, or a wearable haptic device. Further,
the haptic effect(s) can produce any type of haptic feedback, such
as vibrations, deformations, electrostatic sensations, or
kinesthetic sensations. A single haptic effect generator, such as
an actuator, can be used to output the haptic effect(s), or
multiple haptic effect generators can be used. Thus, in this
embodiment, any haptic effect that is generated from audio source
data 210 can be encoded and persisted for later usage. However, in
alternate embodiments, a single haptic effect, or a plurality of
haptic effects, can be generated from audio source data 210 and
output in real-time.
[0032] FIG. 3 illustrates a flow diagram of a selection of a haptic
primitive that is most similar to audio source data, according to
one embodiment of the invention. FIG. 3 illustrates audio source
data 310. According to the embodiment, audio source data 310
includes audio data. In certain embodiments, the audio data
included within audio source data 310 can be audio data that is
stored in either an audio file or an audio signal. In other
embodiments, the audio data included within audio source data 310
can be audio data that is streamed. Further, in the illustrated
embodiment, audio source data 310 is encoded in a PCM format. In
certain embodiments, audio source data 310 can also be encoded in
another type of format, such as a MIDI format, or an MP3 format.
Further, in alternate embodiments, audio source data 310 can be
replaced by another type of source data that includes another type
of data, such as video source data that includes video data,
acceleration source data that includes acceleration data,
orientation source data that includes orientation data, ambient
light source data that includes ambient light data, or another type
of source data that includes another type of data. An example of
another type of source data is source data that includes data that
can be captured with a sensor. Further, in some embodiments, audio
source data 310 can be replaced by source data that includes
multi-modal data.
[0033] Further, audio source data 310 can include characteristics.
A characteristic of audio source data 310 is a physical
characteristic of audio source data 310, such as a physical
characteristic of audio data stored in an audio signal, where audio
source data 310 includes data stored in an audio signal, or a
physical characteristic of audio data stored in an audio file,
where audio source data 310 includes data stored in an audio file.
A specified subset of characteristics of audio source data 310 can
be classified as significant characteristics. Examples of
significant characteristics of audio source data 310 can include:
amplitude, frequency, duration, envelope, density, magnitude, and
strength. A characteristic can include a numeric value, where the
numeric value can define a characteristic of audio source data
310.
[0034] FIG. 3 further illustrates a haptic primitive set 320, where
haptic primitive set 320 includes haptic primitives. A haptic
primitive is a definition of a haptic effect that can be generated.
For example, a haptic primitive can include a definition for a
periodic haptic effect that is 50 milliseconds ("ms") in duration,
has a 10 ms attack portion, has a 5 ms sustain portion, and has an
8 ms decay portion. In certain embodiments, haptic primitive set
320 can include a single haptic primitive. As part of the
definition of the haptic effect, a haptic primitive can further
optionally include haptic parameters, where a haptic parameter is a
parameter that can define a haptic signal used to generate a haptic
effect, and thus, can also define the haptic effect to be
generated. In certain embodiments, a haptic primitive can include a
single haptic parameter. More specifically, a haptic parameter is a
quantity of a haptic effect quality, such as magnitude, frequency,
duration, amplitude, strength, envelope, density, or any other kind
of quantifiable haptic parameter. According to the embodiment, a
haptic effect can be defined, at least in part, by the one or more
haptic parameters, where the one or more haptic parameters can
define characteristics of the haptic effect. A haptic parameter can
include a numeric value, where the numeric value can define a
characteristic of the haptic signal, and thus, can also define a
characteristic of the haptic effect generated by the haptic signal.
Thus, each haptic primitive of haptic primitive set 320 can include
one or more numeric values, where the one or more numerical values
can parameterize a haptic effect. Examples of haptic parameters can
include: an amplitude haptic parameter, a frequency haptic
parameter, a duration haptic parameter, an envelope haptic
parameter, a density haptic parameter, a magnitude haptic
parameter, and a strength haptic parameter.
[0035] According to the embodiment, an amplitude haptic parameter
can define an amplitude of a haptic signal used to generate a
haptic effect, and thus, can define an amplitude of the haptic
effect. A frequency haptic parameter can define a frequency of a
haptic signal used to generate a haptic effect, and thus, can
define a frequency of the haptic effect. A duration haptic
parameter can define a duration of a haptic signal used to generate
a haptic effect, and thus, can define a duration of the haptic
effect. An envelope haptic parameter can define an envelope of a
haptic signal used to generate a haptic effect, and thus, can
define an envelope of the haptic effect. A density haptic parameter
can define a density of a haptic signal used to generate a haptic
effect, and thus, can define a density of the haptic effect. A
magnitude haptic parameter can define a magnitude of a haptic
signal used to generate a haptic effect, and thus, can define a
magnitude of the haptic effect. A strength haptic parameter can
define a strength of a haptic signal used to generate a haptic
effect, and thus, can define a strength of the haptic effect.
[0036] FIG. 3 illustrates a matching algorithm 330 that receives
audio source data 310 and haptic primitive set 320 as input.
According to the embodiment, matching algorithm 330 is an algorithm
that can compare audio source data 310 with each haptic primitive
included within haptic primitive set 320, and can select one or
more haptic primitives of haptic primitive set 320 that are most
similar to audio source data 310. In certain embodiments, matching
algorithm 330 can also optionally optimize the selected one or more
haptic primitives of haptic primitive set 320 so that they are more
similar to audio source data 310, as is further described below in
greater detail.
[0037] According to an embodiment, matching algorithm 330 can first
identify significant characteristics (or a single significant
characteristic) of audio source data 310. Further, for each haptic
primitive of haptic primitive set 320, matching algorithm 330 can
identify haptic parameters (or a single haptic parameter). Matching
algorithm 330 can further compare the haptic parameters of each
haptic primitive to the significant characteristics of audio source
data 310. By comparing the haptic parameters to the significant
characteristics, matching algorithm 330 can determine how similar
the haptic parameters are to the significant characteristics. In
comparing the haptic parameters of each haptic primitive with the
significant characteristics of audio source data 310, matching
algorithm 330 can identify a haptic parameter for each haptic
primitive that corresponds to the significant characteristic of
audio source data 310, and can further compare the identified
significant characteristic with the corresponding identified haptic
parameter. For example, matching algorithm 330 can identify an
amplitude characteristic of audio source data 310, and can further
identify an amplitude haptic parameter of a haptic primitive of
haptic primitive set 320. Matching algorithm 330 can then further
compare the identified amplitude characteristic and the
corresponding identified amplitude haptic parameter.
[0038] In comparing a significant characteristic with a
corresponding haptic parameter, matching algorithm 330 can compare
a value of the significant characteristic with a value of the
corresponding haptic parameter. By comparing a value of the haptic
parameter to a value of the corresponding significant
characteristic, matching algorithm 330 can determine how similar
the value of the haptic parameter is to the value of the
significant characteristic. Matching algorithm 330 can then select
the haptic primitives (or the single haptic primitive) from haptic
primitive set 320 in which the values of haptic parameters are most
similar to the values of the significant characteristics of audio
source data 310.
[0039] Matching algorithm 330 can use any comparison metric to
determine which haptic primitive is (or which haptic primitives
are) the most similar to audio source data 310. More specifically,
matching algorithm 330 can use any comparison metric in order to
determine which haptic parameters for a specific haptic primitive
are most similar to the significant characteristics of audio source
data 310. As an example, matching algorithm 330 can, for each
haptic primitive of haptic primitive set 320, determine a number of
haptic parameters whose values are identical to values of their
corresponding significant characteristics of audio source data 310,
and can select a haptic primitive that includes a highest number of
haptic parameters whose values are identical to values of their
corresponding significant characteristics of audio source data 310.
As another example, matching algorithm 330 can, for each haptic
primitive of haptic primitive set 320, calculate a deviation
between a value of each haptic parameter and a value of its
corresponding characteristic of audio source data 310, calculate a
mean or total deviation for the haptic primitive, and can further
select a haptic primitive with the lowest mean or total deviation.
These are only example comparison metrics, and, in alternate
embodiments, matching algorithm 330 can use alternate comparison
metrics to determine which haptic primitive is (or which haptic
primitives are) most similar to audio source data 310. Further,
matching algorithm 330 can use an optimization algorithm to
maximize or minimize a comparison metric across all haptic
primitives of haptic primitive set 320 (including each haptic
parameter of each haptic primitive). Examples of such optimization
algorithms are readily appreciated by one of ordinary skill in the
relevant art.
[0040] In one embodiment, as previously described, matching
algorithm 330 can optionally optimize the selected haptic
primitives (or the selected single haptic primitive) of haptic
primitive set 320. According to the embodiment, by optimizing each
selected haptic primitive, matching algorithm 330 can adjust a
value of a single haptic parameter (or a plurality of haptic
parameters) of each selected haptic primitive to be more similar to
a value of a corresponding characteristic of audio source data 310.
The adjusting of the value(s) can be an upward adjustment or a
downward adjustment. This can produce a refinement of the haptic
parameter(s) of each selected haptic primitive of haptic primitive
set 320. The refinement of the haptic parameter(s) can, thus,
refine each selected haptic primitive of haptic primitive set 320,
so that each selected haptic primitive is more similar to audio
source data 310.
[0041] FIG. 3 illustrates an ordered haptic primitive set 340,
where ordered haptic primitive set 340 is output that is produced
by matching algorithm 330. According to the embodiment, ordered
haptic primitive set 340 includes a set of haptic primitives (i.e.,
haptic primitive 1, haptic primitive 2, . . . haptic primitive M),
where the haptic primitives are ordered based on how similar the
haptic primitives are to audio source data 310. In certain
embodiments, ordered haptic primitive set 340 includes all the
haptic primitives included within haptic primitive set 320. In
other alternate embodiments, ordered haptic primitive set 340 only
includes the haptic primitives included with haptic primitive set
320 that matching algorithm 330 identifies as being most similar to
audio source data 310 (i.e., the haptic primitives in which values
of the haptic parameters of the haptic primitive are most similar
to values of the characteristics of audio source data 310). In
certain embodiments, matching algorithm 330 can assign a score to
each haptic primitive of ordered haptic primitive set 340, where
the score identifies how similar each haptic primitive is to audio
source data 310. Thus, each haptic primitive of ordered haptic
primitive set 340 can be selected from haptic primitive set 320 and
ordered based on its assigned score. One or more haptic effects can
subsequently be generated from each haptic primitive of ordered
haptic primitive set 340. The one or more haptic effects can
subsequently be transmitted, stored, or broadcast. Subsequently,
the one or more haptic effects can be output. In alternate
embodiments, the one or more haptic effects can be output after
they are generated.
[0042] An example of selecting a haptic primitive that is most
similar to audio source data is now described. In the example,
audio source data 310 can include data stored within an audio file
associated with a firing of a shotgun in a video game, where the
data included in audio source data 310 is stored in a waveform
audio file format ("WAV"). Matching algorithm 330 can process audio
source data 310 and determine characteristics of audio source data
310, such as duration, frequency, content, and/or envelope. Given
these parameters, matching algorithm 330 can assess a set of haptic
primitives stored within haptic primitive set 320 to determine a
best match to audio source data 310. In this example where audio
source data 310 includes audio data associated with a gunshot audio
effect, the duration can be short, and a magnitude envelope could
feature two distinct parts (e.g., an impulse-like loud noise
followed by a longer envelope for an echo of the gunshot), where
the two distinct parts could include distinctive frequency content.
In embodiments where the audio is best matched with two or more
haptic primitives that are played sequentially, or with some
overlap, matching algorithm 330 can segment audio source data 310
and iterate on the segments of audio source data 310 in order to
identify a good match.
[0043] According to the example, a first section of the available
haptic primitives of haptic primitive set 320 can be based on a
duration of audio source data 310. A haptic primitive (and thus, a
haptic effect) can be selected. Once the haptic primitive is
selected, the haptic primitive (and thus, the haptic effect) can be
further optimized by having the haptic parameters of the haptic
primitive be automatically adjusted. For example, if audio source
data 310 has a duration of 345 ms, and the haptic primitive of
haptic primitive set 320 that is the most similar to audio source
data 310 has a duration haptic parameter with a value of 300 ms,
the duration haptic parameter can be adjusted to have a value of
345 ms, so that the haptic effect generated based on the haptic
primitive is more similar to audio source data 310.
[0044] Further, in accordance with the example, matching algorithm
330 can use an envelope shape and frequency content of audio source
data 310 to select a haptic primitive from haptic primitive set 320
that more closely matches audio source data 310. More specifically,
matching algorithm 330 can select a haptic primitive with an
envelope haptic parameter and a frequency haptic parameter, so that
the haptic effect generated from the haptic primitive has an
envelope and frequency similar to audio source data 310 (i.e., an
impulse-like loud noise followed by a longer envelope for an echo
of the gunshot, with associated frequency content). In an alternate
embodiment, matching algorithm 330 can compound or combine two
haptic primitives (and thus, two haptic effects) to more closely
match audio source data 310. Thus, in an alternate example,
matching algorithm 330 can select a first haptic primitive (and
thus, a first haptic effect) to match the impulse-like loud noise,
and can further select a second haptic primitive (and thus, a
second haptic effect) to match the trailing echo. Matching
algorithm 330 can generate a third haptic primitive that combines
the first haptic primitive and the second haptic primitive (and
thus, can generate a third haptic effect that combines the first
haptic effect and the second haptic effect). In combining the first
haptic primitive and the second haptic primitive as part of the
third haptic primitive, matching algorithm 330 can further place
the first haptic primitive and the second haptic primitive in a
proper temporal sequence with a correct spacing between the two
haptic primitives.
[0045] Another example of selecting a haptic primitive that is most
similar to audio source data is now described. In the example,
audio source data 310 can include audio data associated with the
audio of an explosion. Further, haptic primitive set 320 can
include ten haptic primitives, where each haptic primitive can be
used to generate an explosion haptic effect. Matching algorithm 330
can compare audio source data 310 with each haptic primitive of
haptic primitive set 320, and select the haptic primitive with a
duration haptic parameter that is most similar to a duration of
audio source data 310. Alternatively, matching algorithm 330 can
select the haptic primitive with an amplitude haptic parameter that
is most similar to an amplitude of audio source data 310. Matching
algorithm 330 can then optionally optimize the selected haptic
primitive by adjusting the haptic parameters of the selected haptic
primitive to be more similar to the characteristics of audio source
data 310. Thus, matching algorithm 330 can select the explosion
haptic effect that most closely fits audio source data 310.
Further, the optimized haptic primitive (i.e., the haptic primitive
in which one or more haptic parameters are adjusted) can be
included within haptic primitive set 320 to further improve the
performance of matching algorithm 330 for subsequent matching.
[0046] FIG. 4 illustrates a flow diagram for an audio feature
extraction and a selection of a haptic effect based on the
extracted audio feature, according to one embodiment of the
invention. According to the embodiment, a matching algorithm can
extract a single feature (or a plurality of features) from audio
source data (or other type of stream), and compare the extracted
feature(s) to pre-computed templates that are deemed relevant, and
where haptic effects are associated with each template.
[0047] FIG. 4 illustrates audio source data 410 that is received.
According to the embodiment, audio source data 410 includes audio
data. In certain embodiments, the audio data included within audio
source data 410 can be audio data that is stored in either an audio
file or an audio signal. In other embodiments, the audio data
included within audio source data 410 can be audio data that is
streamed. Further, in the illustrated embodiment, audio source data
410 is encoded in a PCM format. In certain embodiments, audio
source data 410 can also be encoded in another type of format, such
as a MIDI format, or an MP3 format. Further, in alternate
embodiments, audio source data 410 can be replaced by another type
of source data that includes another type of data, such as video
source data that includes video data, acceleration source data that
includes acceleration data, orientation source data that includes
orientation data, ambient light source data that includes ambient
light data, or another type of source data that includes another
type of data. As an example of acceleration data, a controller of a
device can automatically assign a haptic effect to an on-screen or
free-space gesture, such as rocking the device back and forth. When
the user performs the action, acceleration data associated with the
action can be sent to the controller. The controller can process
the acceleration data and determine that the user made several
abrupt motions. The control can then select haptic effects that are
most similar to the abrupt motions, such as high-frequency haptic
effects. An example of another type of source data is source data
that includes data that can be captured with a sensor. Further, in
some embodiments, audio source data can be replaced by source data
that includes multi-modal data.
[0048] Further, audio source data 410 can include features. An
example of a feature is a characteristic, which is previously
described above. Another example of a feature is a transition from
a first characteristic to a second characteristic, such as a
transaction from a high frequency to a low frequency. A feature can
include a numeric value, where the numeric value can define a
feature of audio source data 410.
[0049] FIG. 4 further illustrates sound template set 420. According
to the embodiment, sound template set 420 includes sound templates.
In certain embodiments, sound template set 420 can include a single
sound template. A sound template can include template features.
Similar to a feature, a template feature can be a characteristic,
or alternatively, can be a transition from a first characteristic
to a second characteristic. As described below in greater detail, a
sound template can also be associated with corresponding haptic
effects. Additionally, in certain embodiments, a sound template can
be a portion of audio source data, and can include audio data. The
portion of the audio source data can be a pattern that consistently
occurs within audio source data. Examples of sound templates
include a gunshot sound, a car crash sound, an explosion sound, a
collision sound, a punch sound, an arrow sound, or another type of
percussive sound. According to an embodiment, a sound template can
be selected as candidates for conversion to haptic effects. In one
embodiment, the sound templates of sound template set 420 can be
stored in a database, or library, of templates for use at a time
when audio source data, such as audio source data 410, is
processed. In certain embodiments, in order to create the database,
or library, where the sound templates of sound template set 420 are
stored, the sound templates can be processed using one of the
following techniques in order to extract template features that
later can be compared to features extracted from audio source data
410: (1) a spectrogram (i.e., time vs. frequency plots of a time
domain signal); (2) a cepstrum (i.e., a fast Fourier transform
("FFT") of a logarithm of a spectrum of a signal); (3) stochastic
and probability models (e.g., Hidden Markov Models ("HMM")); or (4)
authoring haptic parameters based on audio source data (such as
audio source data 410). Audio files that are derived from a
synthesis engine, sensor, or compositing tool (e.g., Protools) also
can have feature information available as metadata. The techniques
can be used similarly as they are used for speech recognition.
However, according to the embodiments, the techniques can be used
to recognize haptic-related events rather than speech.
Specifically, HMM can be used to identify the sound templates of
sound template set 420 within audio source data 410. Another
algorithm that can be used is a neural network algorithm.
[0050] Further, in alternate embodiments, sound template set 420
can be replaced by another type of template set that includes
templates of another type. One example of another type of template
set is a video template set that includes video templates, where a
video template can be a portion of video source data, and can
include video data. Another example is an acceleration template set
that includes acceleration templates, where an acceleration
template can be a portion of acceleration source data, and can
include acceleration data. Another example is an orientation
template set that includes orientation templates, where an
orientation template can be a portion of orientation source data,
and can include orientation data. Another example is an ambient
light template set that includes ambient light templates, where an
ambient light template can be a portion of ambient light source
data, and can include ambient light data. Another example is a
template set that includes sensor templates, where a sensor
template can include data that can be captured with a sensor.
Further, in some embodiments, a template set can include templates
of a plurality of types.
[0051] FIG. 4 further illustrates haptic effect set 430. According
to the embodiment, haptic effect set 430 includes haptic effects.
In certain embodiments, haptic effect set 430 can include a single
haptic effect. Each sound template of sound template set 420 can
have haptic effects of haptic effect set 430 associated with the
sound template. In certain embodiments, each haptic effect of
haptic effect set 430 can be assigned to sound templates of haptic
effect set 430 within a database that stores sound template set
420. Thus, each sound template of sound template set 420 has haptic
effects from haptic effect set 430 assigned to it. The haptic
effects of haptic effect set 430 can be generated using an
automatic audio-to-haptic conversion, such as the automatic
audio-to-haptic conversion previously described in relation to FIG.
3. In other embodiments, the haptic effects of haptic effect set
430 can be individually designed by a haptic effect designer. In
certain embodiments, the haptic effects of haptic effect set 430
can be part of the database that contains the sound templates of
sound template set 420. In other embodiments, the haptic effects of
haptic effect set 430 can be stored in a separate database along
with indices that correspond to the corresponding sound templates
of sound template set 420.
[0052] At 440, audio source data 410 is received, where audio
source data 410 is selected for conversion to a single haptic
effect, or a plurality of haptic effects. Audio source data 410 is
processed, and one or more algorithms are applied in order to
extract features from audio source data 410. The extracted features
from audio source data 410 are compared to the sound templates of
sound template set 420. More specifically, for each sound template
of sound template set 420, the extracted features from audio source
data 410 are compared to template features of that sound template.
By comparing the extracted features from audio source data 410 with
the template features of the sound templates of sound template set
420, sound templates (or a single sound template) can be identified
as having template features that are the most similar to the
extracted features from audio source data 410. In other words,
sound templates (or a single sound template) can be identified as
being the most similar to audio source data 410. These identified
sound templates can subsequently be selected from sound template
set 420. Once the identified sound templates have been selected,
haptic effects associated with each selected template can also be
selected from haptic effect set 430. In one embodiment, the
selected haptic effects can optionally be optimized. According to
the embodiment, by optimizing each selected haptic effect, a value
of a single haptic parameter (or a plurality of haptic parameters)
of each selected haptic effect can be adjusted to be more similar
to a value of a corresponding feature of audio source data 410. The
adjusting of the value(s) can be an upward adjustment or a downward
adjustment. This can produce a refinement of the haptic
parameter(s) of each selected haptic effect of haptic effect set
430. The refinement of the haptic parameter(s) can, thus, refine
each selected haptic effect of haptic effect set 430, so that each
selected haptic effect is more similar to audio source data 430.
Once the haptic effects have been selected, the haptic effects can
be used to generate haptic information 450. Haptic information 450
is a collection of the selected haptic effects that can
subsequently be output. In one embodiment, haptic information 450
can take the form of haptic data, a haptic track, or a haptic
stream. In certain embodiments, haptic information 450 can be
encoded according to any haptic encoding technique that is known to
one of ordinary skill in the art. Haptic information 450 can
subsequently be transmitted, stored, or broadcast. Subsequently,
haptic information 450 can be output. In alternate embodiments,
haptic information 450 can be output after it is generated.
[0053] Thus, by storing haptic effects, associating the haptic
effects with a sound template, and comparing audio source data with
the sound template, the association between the haptic effect and
the sound template can be reused to further enhance or optimize the
automatic conversion of audio data into a haptic effect.
[0054] FIG. 5 illustrates a flow diagram of the functionality of an
automatic haptic effect fitting module (such as automatic haptic
effect fitting module 16 of FIG. 1), according to one embodiment of
the invention. The flow begins and proceeds to 510. At 510, source
data is received, where the source data includes one or more
characteristics. In certain embodiments, the source data includes
audio source data. In some of these embodiments, the audio source
data is encoded in a PCM format. In embodiments where the source
data includes audio source data, the audio source data includes
audio data, where the audio data can be stored in an audio file or
an audio signal. In other embodiments, the source data includes
video source data. In yet other embodiments, the source data
includes acceleration source data. In yet other embodiments, the
source data includes orientation source data. In yet other
embodiments, the source data includes ambient light source data. In
yet other embodiments, the source data includes multi-modal source
data. Further, in certain embodiments, the one or more
characteristics of the source data include at least one of: an
amplitude, a frequency, a duration, an envelope, a density, a
magnitude, or a strength. The flow proceeds to 520.
[0055] At 520, the source data is compared with one or more haptic
primitives, where each haptic primitive includes one or more haptic
parameters. In certain embodiments, the comparing further includes
comparing the one or more characteristics of the source data with
the one or more haptic parameters of each haptic primitive. In
certain embodiments, the one or more haptic parameters include at
least one of: an amplitude haptic parameter, a frequency haptic
parameter, a duration haptic parameter, an envelope haptic
parameter, a density haptic parameter, a magnitude haptic
parameter, or a strength haptic parameter. The flow proceeds to
530.
[0056] At 530, one or more haptic primitives are selected based on
the comparison performed at 520. In certain embodiments, the
selecting includes selecting the one or more haptic primitives in
which values of the one or more haptic parameters are most similar
to values of the one or more characteristics of the source data.
The flow proceeds to 540.
[0057] At 540, the selected one or more haptic primitives are
optimized. In certain embodiments, the selected one or more haptic
primitives are optimized to be more similar to the source data. In
some of these embodiments, the optimizing includes adjusting a
value of at least one haptic parameter of each selected haptic
primitive to be more similar to a value of a corresponding
characteristic of the source data. In certain embodiments, 540 is
omitted. The flow proceeds to 550.
[0058] At 550, one or more haptic effects are output based on the
selected one or more primitives. In certain embodiments, before the
one or more haptic effects are output, the one or more haptic
effects can be encoded and/or stored in a storage, such as file.
Subsequently, in these embodiments, the one or more haptic effects
can be retrieved and/or decoded, before they are output. The flow
then ends.
[0059] FIG. 6 illustrates a flow diagram of the functionality of an
automatic haptic effect fitting module (such as automatic haptic
effect fitting module 16 of FIG. 1), according to another
embodiment of the invention. The flow begins and proceeds to 610.
At 610, source data is received, where the source data includes one
or more features. In certain embodiments, the source data includes
audio source data. In some of these embodiments, the audio source
data is encoded in a PCM format. In embodiments where the source
data includes audio source data, the audio source data includes
audio data, where the audio data can be stored in an audio file or
an audio signal. In other embodiments, the source data includes
video source data. In yet other embodiments, the source data
includes acceleration source data. In yet other embodiments, the
source data includes orientation source data. In yet other
embodiments, the source data includes ambient light source data. In
yet other embodiments, the source data includes multi-modal source
data. The flow proceeds to 620.
[0060] At 620, one or more features are extracted from the source
data. The flow proceeds to 630.
[0061] At 630, the one or more extracted features are compared with
one or more templates, where each template includes one or more
template features and one or more haptic effects. In certain
embodiments, the one or more templates include one or more sound
templates. In alternate embodiments, the one or more templates
include one or more video templates. In other alternate
embodiments, the one or more templates include one or more
acceleration templates. In certain embodiments, the comparing
includes comparing each extracted feature of the source data with
each template feature of the one or more templates. The flow
proceeds to 640.
[0062] At 640, one or more templates are selected from the one or
more templates based on the comparison performed at 630. In certain
embodiments, the selecting includes selecting the one or more
templates in which the one or more template features are most
similar to the one or more extracted features of the source data.
The flow proceeds to 650.
[0063] At 650, one or more haptic effects are selected from the one
or more selected templates. The flow proceeds to 650.
[0064] At 660, the one or more selected haptic effects are output.
In certain embodiments, before the one or more selected haptic
effects are output, the one or more selected haptic effects can be
encoded and/or stored in a storage, such as file. Subsequently, in
these embodiments, the one or more selected haptic effects can be
retrieved and/or decoded, before they are output. The flow then
ends.
[0065] Thus, according to an embodiment, a system is provided that
can analyze a source data, such as audio source data, and can
identify one or more haptic effects that are the most similar to
the source data. The system can then match the identified one or
more haptic effects with the source data. The system can
subsequently output the identified one or more haptic effects.
Through this technique, the system can utilize source data, such as
audio source data, before the source data has been mixed. By
utilizing source data earlier in the process, before the source
data has been mixed, the system can increase an appropriateness and
quality of a haptic effect that can be added to the overall
content. Further, the system can be very valuable for haptic effect
design in games and other content that have effects, such as audio
effects, and haptic effects playing simultaneously. The system can
provide an easy way for content producers to identify appropriate
haptic content and customize the haptic content for their game or
experience.
[0066] The features, structures, or characteristics of the
invention described throughout this specification may be combined
in any suitable manner in one or more embodiments. For example, the
usage of "one embodiment," "some embodiments," "certain
embodiment," "certain embodiments," or other similar language,
throughout this specification refers to the fact that a particular
feature, structure, or characteristic described in connection with
the embodiment may be included in at least one embodiment of the
present invention. Thus, appearances of the phrases "one
embodiment," "some embodiments," "a certain embodiment," "certain
embodiments," or other similar language, throughout this
specification do not necessarily all refer to the same group of
embodiments, and the described features, structures, or
characteristics may be combined in any suitable manner in one or
more embodiments.
[0067] One having ordinary skill in the art will readily understand
that the invention as discussed above may be practiced with steps
in a different order, and/or with elements in configurations which
are different than those which are disclosed. Therefore, although
the invention has been described based upon these preferred
embodiments, it would be apparent to those of skill in the art that
certain modifications, variations, and alternative constructions
would be apparent, while remaining within the spirit and scope of
the invention. In order to determine the metes and bounds of the
invention, therefore, reference should be made to the appended
claims.
* * * * *