U.S. patent application number 14/322566 was filed with the patent office on 2014-11-06 for method and system for providing a graphical representation on a second screen of social messages related to content on a first screen.
The applicant listed for this patent is THOMSON LICENSING. Invention is credited to Mark Leroy Walker.
Application Number | 20140327677 14/322566 |
Document ID | / |
Family ID | 47472156 |
Filed Date | 2014-11-06 |
United States Patent
Application |
20140327677 |
Kind Code |
A1 |
Walker; Mark Leroy |
November 6, 2014 |
METHOD AND SYSTEM FOR PROVIDING A GRAPHICAL REPRESENTATION ON A
SECOND SCREEN OF SOCIAL MESSAGES RELATED TO CONTENT ON A FIRST
SCREEN
Abstract
The present disclosure is directed towards a method and system
method for providing a graphical representation of social messages
on a second screen relating to content displayed on a first screen.
The disclosure involves monitoring social media for one or more
messages relating to content being displayed on a first screen,
processing the one or more message relating to content being
displayed on the first screen to associate messages with a time
segment of the content being displayed on the first screen, and
providing a graphical representation of the one or more social
messages, associated with one or more time segments of the content
being displayed on the first screen, on the second screen.
Inventors: |
Walker; Mark Leroy;
(Castaic, CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
THOMSON LICENSING |
Issy de Moulineaux |
|
FR |
|
|
Family ID: |
47472156 |
Appl. No.: |
14/322566 |
Filed: |
December 27, 2012 |
PCT Filed: |
December 27, 2012 |
PCT NO: |
PCT/US2012/071811 |
371 Date: |
July 2, 2014 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
61584134 |
Jan 6, 2012 |
|
|
|
Current U.S.
Class: |
345/440 |
Current CPC
Class: |
G06T 11/206 20130101;
G09G 2370/06 20130101; G06T 2215/16 20130101; H04L 43/045 20130101;
H04N 21/4307 20130101; H04N 21/8133 20130101; G09G 2370/16
20130101; G11B 27/32 20130101; H04L 51/32 20130101; H04L 65/403
20130101; H04N 21/8547 20130101; H04N 21/42204 20130101; H04L 43/10
20130101; G09G 5/12 20130101; H04N 21/4122 20130101; H04N 21/442
20130101; H04N 21/4788 20130101; H04N 21/42224 20130101; H04N
21/42203 20130101; H04N 21/4126 20130101; G06F 16/95 20190101; H04N
21/845 20130101; G06Q 30/0255 20130101; H04N 9/87 20130101; G06F
3/1423 20130101 |
Class at
Publication: |
345/440 |
International
Class: |
H04L 12/26 20060101
H04L012/26; G06T 11/20 20060101 G06T011/20 |
Claims
1. A method of providing a graphical representation of social
messages on a second screen relating to content displayed on a
first screen: monitoring social media for one or more messages
relating to content being displayed on a first screen; processing
the one or more message relating to content being displayed on
first screen device to associate message with a time segment of the
content being displayed on the first screen; and providing a
graphical representation of the one or more social messages
associated with one or more time segments of the content being
displayed on the first screen on the second screen.
2. The method of claim 1, wherein the social media being monitored
is Twitter.
3. The method of claim 1, wherein the graphical representation
comprises a series of bins, wherein each bin is associated with a
time segment of the content being displayed on the first screen and
the one or more messages are associated with the bins.
4. The method of claim 3, wherein selecting a bin caused the
associated one or more messages to be displayed.
5. The method of claim 1, wherein the step of processing the one or
more messages comprises: determining if a timestamp is associated
with the one or more social messages; and associating the one more
messages with a time segment of the content being displayed on the
first screen.
6. The method of claim 5, wherein the timestamp is a hashtag
provided as part of the one or more messages.
7. The method of claim 1, wherein the graphical representation
indicates the number of messages associated with a time segment of
the content being displayed on the first screen.
8. The method of claim 1, wherein the step of monitoring is
performed remotely.
9. The method of claim 1, wherein the graphical representation
provides the ability to control the content being displayed on the
first screen.
10. A second screen device comprising: a screen configured to
display content; storage for storing data; and a processor
configured to monitor social media for one or more messages
relating to content being displayed on a first screen, process the
one or more messages relating to content being displayed on first
screen device to associate message with a time segment of the
content being displayed on the first screen, and provide a
graphical representation of the one or more social messages
associated with one or more time segments of the content being
displayed on the first screen for display on the screen of the
second screen device.
11. The second screen device of claim 10, further comprising a
wireless network interface for wireless communication.
12. The second screen device of claim 10, wherein the processor is
further configured to provide access to social media.
13. The second screen device of claim 10, wherein the second screen
device comprises a touch screen device.
14. The second screen device of claim 10, wherein the graphical
representation comprises a series of bins, wherein each bin is
associated with a time segment of the content being displayed on
the first screen and the one or more messages are associated with
the bins.
15. A machine readable medium containing instructions that when
executed perform the steps comprising: monitoring social media for
one or more messages relating to content being displayed on a first
screen; processing the one or more message relating to content
being displayed on first screen device to associate message with a
time segment of the content being displayed on the first screen;
and providing a graphical representation of the one or more social
messages associated with one or more time segments of the content
being displayed on the first screen on the second screen.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit of U.S. Provisional
Application Ser. No. 61/584,134 filed Jan. 6, 2012 which is
incorporated by reference herein in its entirety.
[0002] This application is also related to the applications
entitled: "METHODS AND SYSTEMS FOR SYNCHONIZING CONTENT ON A SECOND
SCREEN", "METHOD AND SYSTEM FOR SYNCHING SOCIAL MESSAGES WITH
CONTENT TIMELINE", "METHOD AND SYSTEM FOR PROVIDING A DISPLAY OF
SOCIAL MESSAGES ON A SECOND SCREEN WHICH IS SYNCHED WITH CONTENT ON
A FIRST SCREEN", "ALTERNATE VIEW PLAYBACK ON A SECOND SCREEN", and
"METHOD ANN SYSTEM FOR PROVIDING DYNAMIC ADVERTISING ON SECOND
SCREEN BASED ON SOCIAL MESSAGES" which are have been filed
concurrently and are incorporated by reference herein in their
entirety.
BACKGROUND
[0003] 1. Technical Field
[0004] The present invention generally relates to providing
additional content related to displayed content.
[0005] 2. Description of Related Art
[0006] Social messaging is becoming a ubiquitous feature across
various software applications. One problem this presents is that
the sheer quantity of messages can become overwhelming especially
for popular topics. At some point the individual messages get lost
in the crowd and the quantity of messages becomes the interesting
social aspect. This disclosure describes a way to visualize the
volume of social activity over time which can then be used to
identify interesting points in time for media content and also be
used to navigate to those points in time.
[0007] Many second-screen applications provide a social message
"feed" that simply streams social messages as they happen. There is
little organization of this information beyond displaying the most
recent at the top of the list or perhaps the notion of a "promoted"
message that advertisers use to keep their message at the top of
the stack. Messages are quickly replaced by new messages sometimes
faster than a user can scan them. Once the messages have been
buried in the stack their effective relevance to time is
diminished.
SUMMARY
[0008] The presented graphical representation referred to here as a
Social Heatmap organizes social information to correspond with the
timeline of a of a media item such as content being displayed on a
first screen device.
[0009] In accordance with one embodiment, a method for providing a
graphical representation of social messages on a second screen
relating to content displayed on a first screen is provided. The
method involves monitoring social media for one or more messages
relating to content being displayed on a first screen, processing
the one or more message relating to content being displayed on
first screen device to associate message with a time segment of the
content being displayed on the first screen, and providing a
graphical representation of the one or more social messages
associated with one or more time segments of the content being
displayed on the first screen on the second screen.
[0010] In accordance with another embodiment, a second screen
device capable of displaying a graphical representation of social
messages associated with content being displayed on a first screen.
The second screen device includes a screen, storage, and a
processor. The screen is configured to display content. The storage
is for storing data. The processor is configured to monitor social
media for one or more messages relating to content being displayed
on a first screen; process the one or more message relating to
content being displayed on first screen device to associate message
with a time segment of the content being displayed on the first
screen, and provide a graphical representation of the one or more
social messages associated with one or more time segments of the
content being displayed on the first screen for display on the
screen of the second screen device
BRIEF DESCRIPTION OF THE DRAWINGS
[0011] FIG. 1 is a system diagram outlining the delivery of video
and audio content to the home in accordance with one
embodiment.
[0012] FIG. 2 is system diagram showing further detail of a
representative set top box receiver.
[0013] FIG. 3 is a diagram depicting a touch panel control device
in accordance with one embodiment.
[0014] FIG. 4 is a diagram depicting some exemplary user
interactions for use with a touch panel control device in
accordance with one embodiment.
[0015] FIG. 5 is system diagram depicting one embodiment of a
system for implementing techniques of the present invention in
accordance with one embodiment.
[0016] FIG. 6 is a flow diagram depicting an exemplary process in
accordance with one embodiment.
[0017] FIG. 7 is a diagram depicting an exemplary methodology of
synching between devices in accordance with one embodiment.
[0018] FIG. 8 is a diagram depicting an exemplary methodology of
synching between devices in accordance with one embodiment.
[0019] FIGS. 9A-9F are exemplary skeletal screen views depicting
features in accordance with one embodiment when used in passive
mode.
[0020] FIGS. 10A-10D are exemplary skeletal screen views depicting
features in accordance with one embodiment when used in active
mode.
[0021] FIGS. 11A-11C are exemplary skeletal views depicting a
social media sharing feature in accordance with one embodiment.
[0022] FIGS. 12A and 12B are exemplary skeletal views depicting a
content selection features in accordance with one embodiment.
[0023] FIGS. 13A-13E are exemplary skeletal views depicting
additional features in accordance with one embodiment.
[0024] FIGS. 14A-14L are exemplary skinned screen views depicting
how certain features could appear to a user.
[0025] FIG. 15 is a exemplary skeletal view depicting social media
features in accordance with one embodiment.
[0026] FIG. 16 is a flow diagram depicting the functionality of
social media features in accordance with one embodiment.
[0027] FIG. 17 is an exemplary skinned screen view depicting social
media features in accordance with one embodiment.
[0028] FIG. 18 is a flow diagram providing a general methodology of
providing content on a second screen based on social messages
regarding content being displayed on a first screen in accordance
with one embodiment.
[0029] FIG. 19 is an exemplary view of a social quote event in
accordance with one embodiment.
[0030] FIG. 20 is an exemplary view of dynamic advertizing on a
second screen based on social messages in accordance with one
embodiment.
[0031] FIG. 21 is an exemplary view of how a social message can be
associated with content in accordance with one embodiment.
[0032] FIG. 22 is flow diagram providing a methodology for
generating a social message associated with content in accordance
with one embodiment.
[0033] FIG. 23 is flow diagram providing a methodology for
processing a social message associated with content in accordance
with one embodiment.
[0034] FIG. 24 is flow diagram providing a methodology for
providing a graphical representation of social messages associated
with time periods of content in accordance with one embodiment.
[0035] FIG. 25 is an exemplary representation of bins that make up
a graphical representation of social messages associated with time
periods of content in accordance with one embodiment.
[0036] FIG. 26 is an exemplary representation of the association of
messages to the bins that make up a graphical representation of
social messages associated with time periods of content in
accordance with one embodiment.
[0037] FIG. 27 is a exemplary skeletal view depicting a graphical
representation of social messages associated with time periods of
content in accordance with one embodiment.
[0038] FIG. 28 is an exemplary view of the operation of a graphical
representation of social messages associated with time periods of
content in accordance with one embodiment.
[0039] FIG. 29 is another exemplary view of the operation of a
graphical representation of social messages associated with time
periods of content in accordance with another embodiment.
[0040] FIG. 30 is another exemplary view of the operation of a
graphical representation of social messages associated with time
periods of content in accordance with another embodiment.
[0041] FIG. 31 is another exemplary view of the operation of a
graphical representation of social messages associated with time
periods of content in accordance with another embodiment.
[0042] FIG. 32 is another exemplary view of the operation of a
graphical representation of social messages associated with time
periods of content in accordance with another embodiment.
[0043] FIG. 33 is a exemplary skinned screen view depicting a
graphical representation of social messages associated with time
periods of content in accordance with one embodiment.
DETAILED DESCRIPTION
[0044] Turning now to FIG. 1, a block diagram of an embodiment of a
system 100 for delivering content to a home or end user is shown.
The content originates from a content source 102, such as a movie
studio or production house. The content may be supplied in at least
one of two forms. One form may be a broadcast form of content. The
broadcast content is provided to the broadcast affiliate manager
104, which is typically a national broadcast service, such as the
American Broadcasting Company (ABC), National Broadcasting Company
(NBC), Columbia Broadcasting System (CBS), etc. The broadcast
affiliate manager may collect and store the content, and may
schedule delivery of the content over a deliver network, shown as
delivery network 1 (106). Delivery network 1 (106) may include
satellite link transmission from a national center to one or more
regional or local centers. Delivery network 1 (106) may also
include local content delivery using local delivery systems such as
over the air broadcast, satellite broadcast, or cable broadcast.
The locally delivered content is provided to a receiving device 108
in a user's home, where the content will subsequently be searched
by the user. It is to be appreciated that the receiving device 108
can take many forms and may be embodied as a set top box/digital
video recorder (DVR), a gateway, a modem, etc. Further, the
receiving device 108 may act as entry point, or gateway, for a home
network system that includes additional devices configured as
either client or peer devices in the home network.
[0045] A second form of content is referred to as special or
additional content. Special or additional content may include
content delivered as premium viewing, pay-per-view, or other
content otherwise not provided to the broadcast affiliate manager,
e.g., movies, video games or other video elements. In many cases,
the special content may be content requested by the user. The
special content may be delivered to a content manager 110. The
content manager 110 may be a service provider, such as an Internet
website, affiliated, for instance, with a content provider,
broadcast service, or delivery network service. The content manager
110 may also incorporate Internet content into the delivery system.
The content manager 110 may deliver the content to the user's
receiving device 108 over a separate delivery network, delivery
network 2 (112). Delivery network 2 (112) may include high-speed
broadband Internet type communications systems. It is important to
note that the content from the broadcast affiliate manager 104 may
also be delivered using all or parts of delivery network 2 (112)
and content from the content manager 110 may be delivered using all
or parts of delivery network 1 (106). In addition, the user may
also obtain content directly from the Internet via delivery network
2 (112) without necessarily having the content managed by the
content manager 110.
[0046] Several adaptations for utilizing the separately delivered
additional content may be possible. In one possible approach, the
additional content is provided as an augmentation to the broadcast
content, providing alternative displays, purchase and merchandising
options, enhancement material, etc. In another embodiment, the
additional content may completely replace some programming content
provided as broadcast content. Finally, the additional content may
be completely separate from the broadcast content, and may simply
be a media alternative that the user may choose to utilize. For
instance, the additional content may be a library of movies that
are not yet available as broadcast content.
[0047] The receiving device 108 may receive different types of
content from one or both of delivery network 1 and delivery network
2. The receiving device 108 processes the content, and provides a
separation of the content based on user preferences and commands.
The receiving device 108 may also include a storage device, such as
a hard drive or optical disk drive, for recording and playing back
audio and video content. Further details of the operation of the
receiving device 108 and features associated with playing back
stored content will be described below in relation to FIG. 2. The
processed content is provided to a display device 114. The display
device 114 may be a conventional 2-D type display or may
alternatively be an advanced 3-D display.
[0048] The receiving device 108 may also be interfaced to a second
screen such as a touch screen control device 116. The touch screen
control device 116 may be adapted to provide user control for the
receiving device 108 and/or the display device 114. The touch
screen device 116 may also be capable of displaying video content.
The video content may be graphics entries, such as user interface
entries, or may be a portion of the video content that is delivered
to the display device 114. The touch screen control device 116 may
interface to receiving device 108 using any well known signal
transmission system, such as infra-red (IR) or radio frequency (RF)
communications and may include standard protocols such as infra-red
data association (IRDA) standard, Wi-Fi, Bluetooth and the like, or
any other proprietary protocols. In some embodiments, the touch
screen control device 116 can be interfaced directly with delivery
networks 1 and 2. Operations of touch screen control device 116
will be described in further detail below.
[0049] In the example of FIG. 1, the system 100 also includes a
back end server 118 and a usage database 120. The back end server
118 includes a personalization engine that analyzes the usage
habits of a user and makes recommendations based on those usage
habits. The usage database 120 is where the usage habits for a user
are stored. In some cases, the usage database 120 may be part of
the back end server 118 a. In the present example, the back end
server 118 (as well as the usage database 120) is connected to the
system the system 100 and accessed through the delivery network 2
(112).
[0050] Turning now to FIG. 2, a block diagram of an embodiment of a
receiving device 200 is shown. Receiving device 200 may operate
similar to the receiving device described in FIG. 1 and may be
included as part of a gateway device, modem, set-top box, or other
similar communications device. The device 200 shown may also be
incorporated into other systems including an audio device or a
display device. In either case, several components necessary for
complete operation of the system are not shown in the interest of
conciseness, as they are well known to those skilled in the
art.
[0051] In the device 200 shown in FIG. 2, the content is received
by an input signal receiver 202. The input signal receiver 202 may
be one of several known receiver circuits used for receiving,
demodulation, and decoding signals provided over one of the several
possible networks including over the air, cable, satellite,
Ethernet, fiber and phone line networks. The desired input signal
may be selected and retrieved by the input signal receiver 202
based on user input provided through a control interface or touch
panel interface 222. Touch panel interface 222 may include an
interface for a touch screen device. Touch panel interface 222 may
also be adapted to interface to a cellular phone, a tablet, a
mouse, a high end remote or the like.
[0052] The decoded output signal is provided to an input stream
processor 204. The input stream processor 204 performs the final
signal selection and processing, and includes separation of video
content from audio content for the content stream. The audio
content is provided to an audio processor 206 for conversion from
the received format, such as compressed digital signal, to an
analog waveform signal. The analog waveform signal is provided to
an audio interface 208 and further to the display device or audio
amplifier. Alternatively, the audio interface 208 may provide a
digital signal to an audio output device or display device using a
High-Definition Multimedia Interface (HDMI) cable or alternate
audio interface such as via a Sony/Philips Digital Interconnect
Format (SPDIF). The audio interface may also include amplifiers for
driving one more sets of speakers. The audio processor 206 also
performs any necessary conversion for the storage of the audio
signals.
[0053] The video output from the input stream processor 204 is
provided to a video processor 210. The video signal may be one of
several formats. The video processor 210 provides, as necessary a
conversion of the video content, based on the input signal format.
The video processor 210 also performs any necessary conversion for
the storage of the video signals.
[0054] A storage device 212 stores audio and video content received
at the input. The storage device 212 allows later retrieval and
playback of the content under the control of a controller 214 and
also based on commands, e.g., navigation instructions such as
fast-forward (FF) and rewind (Rew), received from a user interface
216 and/or touch panel interface 222. The storage device 212 may be
a hard disk drive, one or more large capacity integrated electronic
memories, such as static RAM (SRAM), or dynamic RAM (DRAM), or may
be an interchangeable optical disk storage system such as a compact
disk (CD) drive or digital video disk (DVD) drive.
[0055] The converted video signal, from the video processor 210,
either originating from the input or from the storage device 212,
is provided to the display interface 218. The display interface 218
further provides the display signal to a display device of the type
described above. The display interface 218 may be an analog signal
interface such as red-green-blue (RGB) or may be a digital
interface such as HDMI. It is to be appreciated that the display
interface 218 will generate the various screens for presenting the
search results in a three dimensional grid as will be described in
more detail below.
[0056] The controller 214 is interconnected via a bus to several of
the components of the device 200, including the input stream
processor 202, audio processor 206, video processor 210, storage
device 212, and a user interface 216. The controller 214 manages
the conversion process for converting the input stream signal into
a signal for storage on the storage device or for display. The
controller 214 also manages the retrieval and playback of stored
content. Furthermore, as will be described below, the controller
214 performs searching of content and the creation and adjusting of
the grid display representing the content, either stored or to be
delivered via the delivery networks, described above.
[0057] The controller 214 is further coupled to control memory 220
(e.g., volatile or nonvolatile memory, including RAM, SRAM, DRAM,
ROM, programmable ROM (PROM), flash memory, electronically
programmable ROM (EPROM), electronically erasable programmable ROM
(EEPROM), etc.) for storing information and instruction code for
controller 214. Control memory 220 may store instructions for
controller 214. Control memory may also store a database of
elements, such as graphic elements containing content. The database
may be stored as a pattern of graphic elements. Alternatively, the
memory may store the graphic elements in identified or grouped
memory locations and use an access or location table to identify
the memory locations for the various portions of information
related to the graphic elements. Additional details related to the
storage of the graphic elements will be described below. Further,
the implementation of the control memory 220 may include several
possible embodiments, such as a single memory device or,
alternatively, more than one memory circuit communicatively
connected or coupled together to form a shared or common memory.
Still further, the memory may be included with other circuitry,
such as portions of bus communications circuitry, in a larger
circuit.
[0058] The user interface process of the present disclosure employs
an input device that can be used to express functions, such as fast
forward, rewind, etc. To allow for this, a touch panel device 300
may be interfaced via the user interface 216 and/or touch panel
interface 222 of the receiving device 200, as shown in FIG. 3. The
touch panel device 300 allows operation of the receiving device or
set top box based on hand movements, or gestures, and actions
translated through the panel into commands for the set top box or
other control device. In one embodiment, the touch panel 300 may
simply serve as a navigational tool to navigate the grid display.
In other embodiments, the touch panel 300 will additionally serve
as the display device allowing the user to more directly interact
with the navigation through the grid display of content. The touch
panel device may be included as part of a remote control device
containing more conventional control functions such as actuator or
activator buttons. The touch panel 300 can also includes at least
one camera element. In some embodiments, the touch panel 300 may
also include a microphone.
[0059] Turning now to FIG. 4, the use of a gesture sensing
controller or touch screen, such as shown, provides for a number of
types of user interaction. The inputs from the controller are used
to define gestures and the gestures, in turn, define specific
contextual commands. The configuration of the sensors may permit
defining movement of a user's fingers on a touch screen or may even
permit defining the movement of the controller itself in either one
dimension or two dimensions. Two-dimensional motion, such as a
diagonal, and a combination of yaw, pitch and roll can be used to
define any three-dimensional motion, such as a swing. A number of
gestures are illustrated in FIG. 4. Gestures are interpreted in
context and are identified by defined movements made by the
user.
[0060] Bumping 420 is defined by a two-stroke drawing indicating
pointing in one direction, either up, down, left or right. The
bumping gesture is associated with specific commands in context.
For example, in a TimeShifting mode, a left-bump gesture 420
indicates rewinding, and a right-bump gesture indicates
fast-forwarding. In other contexts, a bump gesture 420 is
interpreted to increment a particular value in the direction
designated by the bump. Checking 440 is defined as in drawing a
checkmark. It is similar to a downward bump gesture 420. Checking
is identified in context to designate a reminder, user tag or to
select an item or element. Circling 440 is defined as drawing a
circle in either direction. It is possible that both directions
could be distinguished. However, to avoid confusion, a circle is
identified as a single command regardless of direction. Dragging
450 is defined as an angular movement of the controller (a change
in pitch and/or yaw) while pressing a button (virtual or physical)
on the tablet 300 (i.e., a "trigger drag"). The dragging gesture
450 may be used for navigation, speed, distance, time-shifting,
rewinding, and forwarding. Dragging 450 can be used to move a
cursor, a virtual cursor, or a change of state, such as
highlighting outlining or selecting on the display. Dragging 450
can be in any direction and is generally used to navigate in two
dimensions. However, in certain interfaces, it is preferred to
modify the response to the dragging command. For example, in some
interfaces, operation in one dimension or direction is favored with
respect to other dimensions or directions depending upon the
position of the virtual cursor or the direction of movement.
Nodding 460 is defined by two fast trigger-drag up-and-down
vertical movements. Nodding 460 is used to indicate "Yes" or
"Accept." X-ing 470 is defined as in drawing the letter "X." X-ing
470 is used for "Delete" or "Block" commands. Wagging 480 is
defined by two trigger-drag fast back-and-forth horizontal
movements. The wagging gesture 480 is used to indicate "No" or
"Cancel."
[0061] Depending on the complexity of the sensor system, only
simple one dimensional motions or gestures may be allowed. For
instance, a simple right or left movement on the sensor as shown
here may produce a fast forward or rewind function. In addition,
multiple sensors could be included and placed at different
locations on the touch screen. For instance, a horizontal sensor
for left and right movement may be placed in one spot and used for
volume up/down, while a vertical sensor for up and down movement
may be place in a different spot and used for channel up/down. In
this way specific gesture mappings may be used.
[0062] The system and methodology can be implemented in any number
of ways depending on the hardware and the content involved.
Examples of such deployment include DVD, Blu-Ray disc (BD);
streaming video or video on demand (VOD), and broadcast (satellite,
cable, over the air). Each of these deployments would have
different architectures but one could standardize the triggers for
each of these events (the additional content) that represents what
would be queued by the application running on the second screen.
For example, event A and event B would be triggered by a synching
mechanism associated with any of these sources of a video. When the
tablet encounters "event A", the program running on the second
screen device (e.g. tablet) will enact "event A". Similarly, if
"event B" is encountered, the program running on the second screen
device would do "event B".
[0063] FIG. 5 depicts a generic system 500 on which such
methodology could be implemented. Here the system 500 includes a
first screen device 510, a second screen device 520, a playback
device 530, a network 540 and server 550. Each of these elements
will be discussed in more detail below.
[0064] The first screen device 510 is a display device, such as
display device 114 described above in relation to FIG. 1, for
displaying content such as television programs, movies, and
websites. Examples of such first screen display devices include,
but are not limited to, a television, monitor, projector, or the
like. The first screen device 510 is connected to the playback
device 530 which can provide the primary content to the first
screen device 510 for display. Examples of such communication
include, but are not limited to HDMI, VGA, Display port, USB,
component, composite, radio frequency (RF), and infrared (IR), and
the like. In certain embodiments, the first screen display device
510 may be connected to the network 540, in either a wired or
wireless (WiFi) manner, providing additional connection to the
second screen device 520 and server 550. In some embodiments, the
first display device 510 may include the functionality of the
playback device 530. In still other embodiments, the first screen
display device 510 may be in non-networked communication 560 with
the second screen device 520. Examples of such non-networked
communication 560 include, but are not limited to, RF, IR,
Blue-Tooth, and other audio communication techniques and
protocols.
[0065] The second screen device 520 is device capable of displaying
additional content related to the primary content being displayed
on the first screen device 510. The second screen device may be a
touch screen control device 116 or touch screen device 300 as
described above. Examples of second screen devices include, but are
not limited to, a smart phone, tablet, laptop, personal media
player (e.g. ipod), or the like. The second screen device 520 is in
communication with playback device 530 using either network 540,
non-networked communication 560, or both. The second screen device
550 is also in communication with the server 550 via the network
540 for requesting and receiving additional content related to the
primary content being displayed on the first screen device 510. In
some embodiments, the second screen device 520 may be in networked
or non-networked communication 560 with the first screen device
510. Examples of such non-networked communication 560 include, but
are not limited to, RF, IR, Blue-Tooth (BT), audio communication
techniques and protocols, or the like.
[0066] The playback device 530 is device capable of providing
primary content for display on the first screen device 510.
Examples of such playback display devices include, but are not
limited to, a DVD player, Blue-Ray Disc (BD) player, game console,
receiver device (cable or satellite), Digital Video Recorder (DVR),
streaming device, personal computer, or the like. The playback
device 530 is connected to the first screen device 510 for
providing the primary content to the first screen device 510 for
display. Examples of such connections include, but are not limited
to HDMI, VGA, Display port, USB, component, composite, radio
frequency (RF), and infrared (IR), and the like. The playback
device 530 is also connected to the network 540, in either a wired
or wireless (WiFi) manner, providing connection to the second
screen device 520 and server 550. In some embodiments, the
functionality of the playback device 530 may be included in the
first screen display device 510. In still other embodiments, the
playback device 530 may be in non-networked communication 560 with
the second screen device 520. Examples of such non-networked
communication 560 include, but are not limited to, RF, IR,
Blue-Tooth (BT), and other audio communication techniques and
protocols.
[0067] The network 540 can be a wired or wireless communication
network implemented using Ethernet, MoCA, and wireless protocols or
a combination thereof. Examples of such a network include, but are
not limited to, delivery network 1 (106) and delivery network 2
(112) discussed above.
[0068] The server 550 is a content server configured to provide
additional content to the second screen device 520. In certain
embodiments, the server may also provide the primary content for
display on the first screen device 510. The service is connected to
the network 540 and can communicate with any of the devices that
are also connected. Examples of such a server include, but are not
limited to, content source 102, broadcast affiliate manager 104,
content manager 110, and the back end server described above.
[0069] FIG. 6 depicts a flow diagram 600 for a methodology for
displaying additional content related to primary content being
viewed is disclosed. The method includes the following steps:
Displaying primary content on a first screen device 510 (step 610).
Providing, in association with the display of the primary content
on the first screen, a synching mechanism to synch additional
content (step 620). Displaying, on a second screen device 520,
additional content related to the primary content on the first
screen 510 that is synched to the content displayed on the first
screen device according to the synching mechanism (step 630). In
certain embodiments, the method also includes the steps of
receiving commands from the second screen device 520 to control the
display of primary content on the first screen device 510 (step
640) and controlling the display of the primary content on the
first screen device 510 based on the commands received from the
second screen device 520 (step 650). Each of these steps will be
described in more detail below.
[0070] The step of displaying primary content (step 610), such as a
movie or television show, is performed on the first screen device
510. This involves the primary content being provided to the first
screen display 510. The primary content can be provided by the
playback device 530 or be received directly from a content provider
at the first screen display device 510. The primary content is then
shown or otherwise displayed on the first screen device 510. The
display of the primary content also includes the control of the
content being displayed. This can include the traditional playback
commands of play, stop, pause, rewind, and fast forward as well as
the navigation of on screen menus to select the content and other
playback options. In certain embodiments, the display on the first
screen device 510 (step 620) further includes displaying an
indicator of the type of additional content being displayed on the
second screen device 520.
[0071] The provided synching mechanism (step 620) can be
implemented in a number of ways. In certain embodiments the
synching mechanism is performed by an application running on the
second screen device 520, the playback mechanism 530, the first
screen device 510 or any combination thereof. At its most basic,
the second screen device 520 is configured (via an application) to
detect synching signals, cues, or other type of indicators that
directs the second screen device 520 to update the display of
additional content to coincide with the primary content being
displayed on the first screen 510. The synching signals, cues or
other type of indicators, can be provided as part of the primary
content or can be generated by the playback device 530 of first
screen device 510 (via an application) in accordance with the
chapter, scene, time-code, subject matter, or content being
displayed. The synching signals, cues or other type of indicators
can be transmitted to the second screen device 520 using the
network, in either a wired or wireless (WiFi) manner, or using
non-networked communication 560 such as audio signals. Examples of
some of the implementations are given below. Other possible
implementations will be apparent given the benefit of this
disclosure.
[0072] The step of displaying the additional content, such as
supplemental materials, video clips, websites, and the like (step
630) is performed on the second screen device 520. The additional
content can be stored locally on the second screen device 520 or be
provided by the server 550, playback device 530, or first screen
device 510. The display of the additional content is synched to the
primary content being displayed on the first screen device 510
according to the synching mechanism. For example, when the second
screen device 520 detects a synching signal, cue or other type of
indicator, the second screen device 520 updates the display of the
additional content accordingly. In some embodiments, this further
involves contacting and requesting the additional content from the
server 550, playback device 530, or first screen device 510 and
subsequently downloading and displaying the additional content. In
some embodiments, the additional content to be displayed can be
selected, modified, or omitted based on the user using the
system.
[0073] In certain embodiments, the display on the second screen
device 520 (step 630) further includes displaying the status of the
display of the primary content on the first screen device 510 such
as whether the display of the primary content on the first screen
device 510 has been paused. In certain other embodiments, the
display on the second screen device 520 (step 630) further includes
displaying the status of the synch between the additional content
on the second screen device 520 and the primary content on the
first screen device 510.
[0074] In certain embodiments, the second screen device 520 is
capable of transmitting as well as receiving. The optional steps
640 and 650 address this capability. In step 640 commands are
received from the second screen device 520. Ideally, these commands
are received at the device controlling the playback of the primary
content on the first screen device 510. In certain embodiment, the
playback device 530 is the device receiving the commands. The
commands can be sent via the network 540 or non-networked
communication 560. Once received, the commands can control the
display of the primary content (step 650). Examples of such control
include, but are not limited to, play, stop, pause, rewind,
fast-forward, as well as chapter, scene, and selection. These
commands can also be used to synch the primary content displayed on
the first screen device 510 with the additional content being
displayed on the second screen device 520.
[0075] FIG. 7 provides a high level overview of one example of
system 700 with a synching mechanism implemented using a
non-networked communication 560. In this system 700, the
non-networked communication synching mechanism is audio
watermarking 710. In this example, audio watermarking 710 involves
inserting a high-frequency signal, cue, or other indicator into the
audio signal of the primary content being displayed on the first
screen device 510. The audio watermark is inaudible to humans but
can be detected by a microphone in the second screen device 520.
When the second screen device 520 detects an audio watermark, the
displayed additional content is updated to synch with the primary
content being displayed on the first screen device 510 based on the
detected watermark. The audio watermarks can be incorporated into
the primary content at the source of the content or inserted
locally by the playback device 520 or first screen device 510.
[0076] FIG. 8 provides a high-level overview of one example of a
system 800 with a synching mechanism implemented using the network
540. In this system 800 the synching mechanism is wireless
communication (WiFi) 810 between a playback device 530 (a Blu-Ray
Disc player) and the second screen device 520 (an iOS device
running an application). In the example of FIG. 7, the features and
protocols of a BD-Live enabled device are used. There are two main
components of this protocol: connection and communication. Both are
described below. For simplicity the second screen iOS application
will be referred to as the "iPad" and the BD-Live enabled device
will be referred to as the "disc".
[0077] Connection occurs when an iOS enabled device 520 first
launches the second screen application and attempts to connect to a
BD-Live enabled device 530 on the same Wi-Fi network 540. [0078] 1.
Disc is inserted into BD player [0079] 2. Disc enters UDP
`listening` loop [0080] 3. iPad launches second screen application
[0081] 4. iPad performs UDP broadcast of authentication token
[0082] 5. Disc receives authentication token and authenticates
[0083] 6. Disc retrieves IP from tokens sender (iPad's IP) [0084]
7. Disc responds to authentication with its IP and PORT [0085] 8.
iPad confirms IP and PORT [0086] 9. iPad closes UDP socket
communication [0087] 10. iPad establishes direct TCP socket
communication with disc based on IP and PORT provided.
[0088] Communication occurs after a connection has been established
between the second screen iOS application and a BD-Live enabled
device. [0089] 1. iPad and Disc are aware of each other's IP's as
well as what PORT communication should occur using [0090] 2. TCP
socket communication is maintained for the duration of the
applications lifecycle.
[0091] One advantage of such a wireless communication as seen in
this example is that it is bi-directional allowing the second
screen device to transmit as well as receive commands. This allows
for two way synching as well as control of playback from the second
screen device 520.
[0092] In certain embodiments, the application of the second screen
device 520 could be specific to a specific program or movie on a
specific system (e.g. BD). In other embodiments, the second screen
application could be generic to a studio with available plug-ins to
configure the application to a particular program or movie. In
still other embodiments the second screen application could be
universal to system (BD, VOD, broadcast), content, or both. Other
possible implementations and configurations will be apparent to one
skilled in the art given the benefit of this disclosure.
[0093] The system can be operated in with a passive approach or an
interactive approach. In the passive approach icons displayed on
first screen device 510 prompt the user to look at the second
screen device 520 for an additional content event being displayed
that is related to the primary content displayed on the first
screen device 510. The icon preferably indicates what type of
additional content event is available on the second screen device
520 (e.g., a shopping cart icon indicates a purchase event, an "I"
icon indicates an information event, a stickman icon indicates a
character information event, etc.) FIG. 9A-F depicts some of the
aspects that may be displayed to the user in passive mode.
[0094] FIGS. 9A-F depict skeletal examples of what may be displayed
on the screen 900 of the second screen device to a user when using
an application in passive mode that provides additional content on
the second screen device 520 that is synched with the primary
content on the first screen device 510.
[0095] FIG. 9A depicts a splash screen that may be displayed to the
user when the application is launched. It includes the product logo
and indication of the primary content 902. Here new content screens
transition in from right in a conveyer-belt like manner as indicate
by arrow 904.
[0096] FIG. 9B depicts a pop-up message 906 that is displayed to a
user when no playback device 530 is detected by second screen
device 520.
[0097] The screen 900 of FIG. 9C shows a synch button/icon 908,
chapter timeline 910, active chapter indicator 912, chapter-event
indicator 914, chapter number indicator 916, event timeline 918,
chapter background 920, event card 922, and timeline view icons
924. The synch button 908 provides a mechanism to synch the content
between the first and second screen devices 510, 520. The synch
button 908 may also indicate the status of the synch between the
content on the first and second screen devices 510, 520. The
chapter timeline 910 indicates the chapters of the primary content.
The movie title leader is in the background of the chapter timeline
910 and indicates the primary content. As the primary content
progresses the chapters move along chapter timeline in a
conveyer-belt life fashion with the active chapter indicator 912
indicating the current chapter in the primary content via highlight
and center position of the chapter timeline 912. The chapter-event
indicator 914 indicates that events displayed in the event timeline
918 are part of the active chapter shown in the chapter timeline
910. The event timeline 918 displays event cards 922 indicating
events that correspond to what is transpiring in the current
chapter of the primary content. For each chapter, the first
displayed event card 922 indicates the chapter that the following
events occur in. As the primary content progresses the event cards
922 move along event timeline 918 in a conveyer-belt like fashion
with the current event in the center position of the event timeline
918. Each chapter may be provided with a unique background 920 for
the events of that particular chapter. The timeline view
icon/button 924 indicates that the viewer is in timeline view
showing the chapter timeline 910 and event timeline 918 as well as
provides a mechanism to access the timeline view.
[0098] The screens 900 of FIGS. 9D and 9E depict how event cards
922 progress across the event timeline 918. Here the synch
button/icon 908 indicates that the timeline view of the additional
content is in synch with the primary content on the first screen
device 510. In FIG. 9D, the current triggered event card 926 is
shown in the center position of the event timeline 918 and
represents the first triggered event. To the left of the current
triggered event card 926 in the event timeline 918 is the previous
event card 928, in this case the card indicating the chapter. To
the right of the current triggered event card 926 in the event
timeline 918 is the next event card 930, in this case the card
indicating the next scheduled event. Since, in FIG. 9D, this is the
current triggered event card 926 is for the first triggered event,
the chapter indicator 916 indicates that it is chapter 1. The
current triggered event card 926 includes the additional content
932 related to the primary content. The current triggered event
card 926 also provides an indicator 934 as to what type of
additional content is displayed. In certain embodiments this
indicator matches an indicator shown on the first screen display
510. The current event card 926 also includes buttons/icons for
synching 936 and sharing 938. The synch button/icon 936 provides a
mechanism that causes the primary content displayed on the first
screen device 520 to be synched with the current event. The share
button/icon 938 provides a mechanism to share the additional
content of the event with a social network. The elements of the
screen 900 of FIG. 9E are similar to the elements of FIG. 9D except
that the current triggered event card 926 is for an event that
happens later in the timeline as indicated by the chapter indicator
916 which indicates the current chapter is chapter 3.
[0099] FIG. 9F depicts examples of other possible functionality
that may be provided as part of display on the second screen device
920. Here the chapter timeline 910 is provided with a collapse
icon/button 940 which provides a mechanism to toggle the chapter
timeline between visible 940a and hidden 940b. Likewise the synch
button/icon 908 can toggle between status indicating whether synch
is currently active 908a and status indicating synch has been lost
and re-synching is available 908b. In some embodiments a volume
button icon 942 is provided. The volume button/icon 942 provides a
mechanism to turn the sound of the first screen display "OFF" or
"ON". The volume button 942 may also indicate the status of whether
the volume is "ON" indicating muting is available 942a, or "OFF"
indicating sound is available 942b. In some other embodiments a
play/pause button/icon 944 is provided. The play/pause button 944
provides a mechanism to pause or resume playback of content on the
first screen display 510. The pause/play button may also indicate
the status of whether the playback can be paused 944a, or "or
resumed 944b.
[0100] In the interactive approach, the user selects an additional
content event on the second screen device 520 and what is displayed
on the primary screen device 510 is synched to the selected event.
As indicated previously, the events of additional content are
synched to the primary content. If the user swipes the movie
timeline or the events, the events become out of synch with the
movie being shown on the main screen. To re-synch touches the synch
button on the tablet. The timeline or events are the synched back
to what is being displayed on the main screen. Likewise, a user can
select a trivia event or map event, touch the synch button, and the
scene in the movie related to the selected trivia or map event will
be played on the main screen. Examples of this can be seen in FIG.
10A-D.
[0101] FIG. 10A depicts how a user may interact with the chapter
timeline 910 and event timeline 918 on the screen 900. Here icons
1000 and 1002 represent how the user can touch the screen to scroll
left or right in the chapter or event timelines 910, 918.
[0102] FIG. 10B depicts one embodiment of the screen 900 when a
user interacts with the chapter timeline 910. In this example the
synch button/icon 908 indicates that the additional content on the
second screen display 520 is out of synch with the primary content
on the first screen display 510. Icon 1000 represents the user
scrolling through the chapter timeline 910. The current chapter
remains highlighted 912 until the transition to the new chapter is
completed. When navigating through the chapter timeline 910 a
chapter position indicator 1004 is provided that indicates what
chapter of the available chapters is selected. The chapter
indicator 916 also indicates the selected chapter and updates when
the transition to the new chapter is complete. In this example,
while the user is navigating through the chapter timeline 910, the
event timeline 918 is dimmed. In certain embodiments, the user may
jump directly to a particular chapter by selecting the chapter from
the timeline 910.
[0103] FIG. 10C depicts one embodiment of the screen 900 when a
user interacts with the event timeline 918. Icon 1002 represents
the user scrolling through the event timeline 918. Here, the
timeline 918 is being transitioned from current triggered event
card 926 to the next event card 930. When navigating through the
event timeline 918 an event position indicator 1004 is provided
that indicates what event of the available events is selected.
[0104] FIG. 10D depicts one embodiment of the screen 900 when a
user interacting with the event timeline 918 causes a transition
from one chapter to another. Icon 1002 represents the user
scrolling through the event timeline 910 causing a chapter change.
Here, the timeline 918 is being transitioned a new event card 922
indicating a new set of events related to a new chapter. When
navigating through the event timeline 918 causes a transition to a
new chapter the event position indicator 1004 is centered until the
new series of events begins.
[0105] FIGS. 11A-C and 12A-B indicate some of the other interactive
activities that can be accessed via the event cards 922. FIGS.
11A-C depict the social media sharing feature. FIGS. 12A-B depict
the chapter selection as well as selection and playback of
additional media files.
[0106] FIG. 11A-C shows various pop-up fields on the display 900
when the sharing feature is active via the share button/icon 937.
FIG. 11A shows the field 1100 displayed when the user has logged
into their social network (in this case Facebook). Area 1102
indicates the event being shared and area 1104 indicates the
comments the user is going to share about the event. Button 1106
provides the mechanism to submit the event and comments to be
shared. FIG. 11B shows the field 1100 displayed when the user has
not yet signed in to the social network. In this example button
1108 is provided to sign into Facebook and button 1110 is provided
to sign into twitter. Options to sign into other social networks
may also be provided. FIG. 11C shows a onscreen Qwerty keyboard
1112 that may be used to enter comments into area 1104 for user's
comments. In certain embodiments, this may be a default keyboard
provided by the second screen device 520.
[0107] FIG. 12A-B shows the selection of chapters as well media
content for playback by the user. In the example of 12A, if the
user single taps 1200 the currently playing chapter shown in the
chapter timeline 912 the playback on the first screen device 510 is
paused. If the user double taps 1202 the currently playing chapter
shown in the chapter timeline, playback of on the first screen
device will jump to the beginning of the chapter and the events
timeline 918 will be set to the first event of that chapter. In
some embodiments, the event cards 922 may include media files 1204
such as video or audio clips. If the media file is an audio clip,
then selection of the audio clip results in playback on the current
screen 900. If the media file is a video clip, then selection of
the video clip results in the launching of a full-screen media
player 1206 as seen in FIG. 12B. In this example the media player
includes on-screen controls 1208. To return to the previous screen,
the user only needs to tap the non-video surface 1210 of the media
player.
[0108] FIG. 13A-E depicts some other possible features regarding
the additional content. These include a map view 1300, family tree
1310, and settings 1320. FIG. 13A depicts the menu bars for these
options. In this example each of these menu bars are provided with
first screen device controls 1330 including pause/resume and
mute/un-mute. FIG. 13B depicts the map view display 1300. The map
view display 1300 includes a map 1302 including marked locations
1304 and information about the locations 1306. Icons are also
provided to select other maps 1308. FIG. 13C depicts the family
tree view 1310. The family tree view shows the family tree with
fields 1312 indicating the relationship between the family members.
In this example the button/icon 1314 at the bottom indicates what
view is currently being shown (i.e. the family tree view). If a
field 1312 is selected, a pop-up field 1316 is displayed, as shown
in FIG. 13D, providing information about the person in the field
1312. FIG. 13e depicts the settings view 1320. In view 1320 the
user is provided with controls for adjusting the preferences for
the audio and video 1322, events 1324, and social network sharing
1326.
[0109] FIGS. 14A-L depict skinned examples of what may be displayed
on the screen 900 of the second screen device to a user when using
an application that provides additional content on the second
screen device 520 that is synched with the primary content on the
first screen device 510. FIG. 14A is a skinned version of the
splash screen as shown and described in relation to FIG. 9A. FIGS.
14B-F depict skinned versions of the timeline view as seen and
described in relation to FIGS. 9C-F and 10A-D. FIG. 14G depicts a
skinned version of a screen display wherein all the available video
clips that are part of the additional content are displayed for the
user. FIG. 14H depicts a skinned version of a screen display
wherein all the available audio clips that are part of the
additional content are displayed for the user. FIG. 14I depicts a
skinned version of the maps view as shown and described in relation
to FIGS. 13B. FIGS. 14J and 14K depict skinned version of the
family tree view as shown and described in relation to FIGS. 13C
and 13D respectively. FIG. 14L depicts a skinned version of the
settings view as shown and described in relation to FIG. 13E.
[0110] The events and features shown in the figures are just some
examples of possible events. In certain embodiments, a user may be
able to configure or otherwise select what events they wish to be
shown (e.g., don't show me purchase events). In other embodiments
the user may be able to select or bookmark events for viewing at a
later time. In still other embodiments certain events may
unavailable or locked out depending on the version of the program
being viewed (i.e. purchased vs. rented or BD vs. VOD vs.
Broadcast). In other embodiments, the events available can be
personalized for a user based on previous viewing habits (i.e. in
system such as TIVO where a user's viewing habits are tracked or
using the personalization engine 118 of FIG. 1).
[0111] Other possible configurations include shopping features. For
example, a store front could be provided and accessible from the
second screen to for purchasing movie merchandise. In another
embodiment points or awards could be provided to a user for
watching, reviewing, or recommending a program or film. For
example, the more movies watched or shared with friends, the more
points awarded. The points can then be used for prizes or discounts
on related goods.
[0112] Similarly, achievements can also be awarded. These
achievements could be pushed to a social networking site. Example
achievements could include:
[0113] Watching certain scenes--Achievement
[0114] Watching certain discs in a series--Achievement
[0115] Watching certain discs by a particular studio or
actor--Achievement
[0116] In still other implementations a Wild feature could be
implemented. A running Wiki could let a user and other users of a
disc comment on certain scene. For example, tracking metadata could
be created which is pushed to a web based wild. Such metadata could
include:
[0117] Chapter Information
[0118] Time Codes
[0119] Thumbnails of Scenes
[0120] Actor/Director Information
[0121] This pushed information can be used to create a running Wild
which lets others comment on the movie. These comments could then
be reintegrated into the second screen application as events which
can be accessed.
[0122] Additional features and screens are also possible. For
example, in some embodiments activity on one or more social
networks that is related to content displayed on the first screen
can be monitored and used to provide additional content on the
second screen.
[0123] As discussed in regard to FIG. 11A-C. The application on the
second screen device 520 can support social media such as Facebook
and Twitter. Additional examples of this can be seen in FIGS.
15-17.
[0124] FIG. 15 depict a wireframe screenshot 1500 showing the panel
1510 displayed over the background, in this case a "greyed-out" or
"dimmed" timeline view 1520, when the user has not yet signed in to
the social network. In this example panel 1510 is for signing into
twitter. The panel 1510 provides an area 1512 to provide a username
or email address as well as an area 1512 to provide a password.
Button 1516 authorized the application to access twitter. Button
1518 declines the sign-in. A similar panel or field can be provided
to sign into Facebook, or other social media networks.
[0125] FIG. 16 depict a flow diagram 1600 of the screens displayed
to a user based on whether they are signed into a social network.
At junction 160 it is determined is the user it signed in or
otherwise authorized on the social network. If the user has not
provided authorization (1612) screen 1500 with panel 1510 of FIG.
15 is displayed to the user prompting them to sign in. If the user
has previously provided authorization (1614) or signs in using
screen 1500 (1618) screen 1620 with message panel 1622 is
displayed. From the message panel 1622 the user may cancel or send
a message using buttons 1624 and 1626 respectively. The text of the
message can be entered using an onscreen keyboard 1628. Once a
message is sent or canceled out of (1630) or if authorization is
never provided (1632) junction 1634 is arrived at wherein the
panels are dismissed and the user is returned to the previous
screen view (1640).
[0126] FIG. 17 depicts a skinned version screen 1700 of the screen
1620 of FIG. 16. A message panel 1710 is provided overtop the
skinned background 1720. The message panel 1710 includes a text
area 1712 as well as cancel 1714 and send 1716 buttons. Text can be
entered into the text area 1712 using an onscreen keyboard
1730.
[0127] In addition to providing the ability to post social messages
from the application on the second screen device 520, the user's
experience can further be enhance being able to track comments on
social media relating to content being viewed on the first screen
device 510 and providing additional content on the second screen
device 520 that is synched to the primary content on the first
screen device 510 based on the tracked comments relating to the
primary content.
[0128] FIG. 18 shows a flow diagram 1800 depicting one possible
methodology for providing such functionality on a second screen
device 520. At a basic level, the method includes three steps.
First, social media activity regarding content being displayed on
the first screen device 510 is monitored (step 1810). The monitored
social media activity is then processed (step 1820). Finally,
additional content based on the social media activity is provided
on the second screen device 520 which is synched to the primary
content on the first screen device 510 (step 1830). Each of these
steps will be discussed in more detail below in reference to
specific exemplary implementation that rely on social media
activity.
Example
Social Quote Event
[0129] Popular broadcast media can potentially generate an
overwhelming amount of related social messages. Second screen
applications designed to support particular broadcasts have been
developed to extend the branded experience. These second screen
applications may include a relevant social feed by filtering
messages by a hashtag or other keyword. Even with this filtering in
place the message count can quickly become unwieldy for popular
events (e.g. Superbowl 46 where the Tweets per second peaked at
10,245). What is disclosed herein is a mechanism to identify and
surface relevant social messages for the user to see without being
inundated with large numbers of messages to scan.
[0130] Current practice in many social messaging applications is to
simply present the messages as they are received resulting in an
almost continuous scrolling of messages as they are rendered in the
user interface. Other implementations may throttle requests for
messages to reduce load on the back-end servers but the number of
results for each request may be large and would be difficult for a
user to scan the list before the next request is fulfilled.
[0131] This disclosure offers an approach that curates social
messages that are then offered to the user with timing appropriate
to a second screen application that may not be entirely focused on
the social messaging aspect. This is valuable when it is desirable
for second-screen applications that want to integrate social
messaging with other aspects of the media experience like bonus
content, trivia and advertising.
[0132] In the context of an application on a second-screen device
520 where multiple items are displayed to the user in a timed
fashion in sync with the media being displayed on a first screen
device 510, curated social information can be interleaved with
other "timeline events". For example an actor trivia "timeline
event" may be presented in conjunction with a character's
appearance on screen. A "Social Quote Event" may be timed for
display just after an intense action scene. This latter event
displays only the high-level/relevant social messages based on
targeting or frequency. This provides the user a sense of what is
being communicated in the social network while not requiring the
user to scan through hundreds or even thousands of messages. An
example of this can be seen in FIG. 19.
[0133] FIG. 19 depicts a wireframe screenshot 1900 showing a
"Social Quote" event panel 1910 as part of the timeline of other
events 1920. The panel 1910 provides a text area 1912 displaying
the relevant curated social quotes. In certain embodiments one or
more additional buttons are provided. In this example Button 1914
provides the ability to post or in this case re-tweet a given
quote. Other possible features and implementation will be apparent
to one skilled in the art given the benefit of this disclosure.
[0134] The process of providing, on a second screen device 520,
social media comments relevant to content being displayed on a
first screen device 510 follows the general methodology set forth
in FIG. 18.
[0135] First social media activity is monitored (step 1810). This
involves looking for keywords, hashtags, or the like that include
the name of the content being displayed on the first screen device
510, the name of the actors, the name of the director, or other
related information using techniques and methodologies that are
well known in the art. Monitoring of social media can be performed
on the second screen device 520, a server 530, provided by a third
party service, or a combination of thereof.
[0136] The step of processing (step 1820) is where the data is
curated to only provide the most relevant comments or messages. To
achieve the curated "Social Quote" the following heuristics may be
applied: [0137] If the social message is targeted to the current
user it should be deemed important for display. As an example this
is like an "at" message (@user) in Twitter. Other targeting or
addressing schemes can be employed as well (e.g. post on a wall in
Facebook) [0138] Many social messages are simply re-sending
original messages. Twitter provides a mechanism of "re-tweets" that
adds metadata to a message to identify how many times it has been
passed along. In this case the re-tweet count provides a metric of
"importance" which is then used to select these messages for
display. [0139] if the message contains multiple keywords it can be
given greater "importance". [0140] if a user may have specified
user preferences for which the messages can be compared to.
[0141] In certain embodiments is also conceivable that social
messages can be "sponsored" so that they automatically gain
relative importance above the crowd of other messages. For example
social media comments that were posted from a second screen
application may be given priority. A further refinement to this
would be along the lines of targeted advertising--the importance of
sponsored messages can be weighted against user-specific criteria
to achieve its relative ranking amongst other messages. This
processing or curating can be performed on the second screen device
520, a server 530, provided by a third party service, or a
combination of thereof.
[0142] One or a few social messages thus ranked can be displayed to
the user to provide targeted and other "significant" messaging
without having to resort to scanning ever-flowing lists of social
messages (step 1830).
[0143] In the case of the timeline-based application on a second
screen device 520, "Social Quote" events can be distributed
throughout the timeline such that regular exposure to social
networks is purposefully interleaved with other event types for a
thoughtfully designed experience.
Example
Dynamic Advertising
[0144] A key to effective advertising is to deliver relevant offers
at the right time. Current techniques involves extensive data
collection on user behavior (e.g. Google tracking your searches,
application use, viewing habits, etc.) which is then used to select
advertisements that best suit the user's profile. A great deal of
specificity can be derived using public and private data to develop
these personalization profiles.
[0145] Providing a focused context for collecting data has the
potential for improving advertisement delivery even further. Social
messaging may indeed be used as a source for personalization but
may be handled independently from other activities that the user
may be engaged in simultaneously. For example a user may tweet
about a particular movie and a back-end system in the cloud can
record the interest in that movie. This data could then be used the
next time a user visits a website that takes advantage of this data
to deliver advertisements. While this may potentially result in a
higher advertisement relevance to the user during the subsequent
visit the context of the original social message is lost.
Second-screen applications offer a known environment from which
social messages can be monitored to enhance advertising
personalization.
[0146] An application on a second screen device 520 can designed to
display scheduled "events" in sync with primary video playback.
Events can represent trivia, social content, voting, bonus
material, advertisements, etc that are timed for display at
relevant points in the primary video playback.
[0147] The context that the second-screen application (and device)
provides is rich, to include: [0148] Media information. This
establishes an interest in a particular media item (broadcast TV,
Video, etc) [0149] Specific Media time information. The application
knows where you are within a piece of media and any specific
metadata associate with that time. [0150] Location information.
Second-screen devices 520 are typically equipment with mechanisms
for identifying your current location
[0151] The disclosed embodiments interleave "events" to support an
overall experience. Some of these event types afford a social
interaction element such as voting or can offer social messaging
anytime throughout the presentation. Subsequent "events" can be
reserved for advertising and can react to previous activity to
include social messaging. This can be accomplished by identifying
keywords within the social message itself and providing that to an
advertising service along with other relevant information such as
location, metadata associated with the current media time, etc.
[0152] The application consolidates all these variables and offers
advertisements within the context of the overall experience.
[0153] For Example a James Bond film could be playing on a first
screen device and a location such as Hong Kong is displayed on
screen. The second-screen app displays a supporting media item such
as trivia about the location. The user could send a social message
(tweets) to describe her desire to go on a vacation. The word
"vacation" is parsed by the application in preparation for later
advertising display. Later in the second-screen timeline an
advertisement "event" focusing on a vacation offer is
displayed.
[0154] This process also follows the general methodology 1800 of
FIG. 18. The monitoring (step 1810) takes place when it is detected
that the user has sent a social message including the keyword
"vacation." In this example, the monitoring is performed on the
second screen device 520, but is could also be performed at the
server 530, by a third party service, or a combination thereof.
[0155] In the processing step (step 1820), the specific
advertisement is selected from an ad service by submitting the
keyword "vacation". Additional information such as the specifically
known location in the movie can be derived from metadata associate
to the movie at the particular time of the tweet. Additionally the
advertisement can recommend a local travel agent using the user's
location information. The processing can be performed on the second
screen device 520, but is could also be performed at the server
530, by a third party service, or a combination thereof.
[0156] Finally, the advertisement can be displayed on the second
screen device (Step 1830). An example of this can be seen in FIG.
20.
[0157] FIG. 20 depicts a wireframe screenshot 2000 showing a
dynamic advertising event panel 2010 as part of the timeline of
other events 2020. The panel 2010 provides a text area 2012
displaying the relevant advertising offers. Other possible features
and implementation will be apparent to one skilled in the art given
the benefit of this disclosure.
[0158] The advertisement thus becomes a natural part of the
"conversation" all within the context of the media consumption
experience.
[0159] While the addition of social messaging to a second screen
application as discussed in the examples above enhances the
experience for the user, further functionality and enhancement of
the experience can be achieved if the social messaging can be
time-stamped or otherwise synched with the playback of content on
the first screen device 510.
Example
Synching Social Messages
[0160] Social messages are typically ephemeral and relevant only to
the present moment in time. A common convention for associating a
message to a particular event or topic is to use hashtags. These
hashtags provide a means to filter for social messages of interest.
Filtering for these hashtags provides a "real-time" view of
messages on the topic. This disclosure takes this notion a step
further and describes a mechanism for associating message to a
specific point in time relative to the start of a media content
(such as video). Social messages employing these techniques can be
associated to specific points in time within the particular
content. For example, social messages can be associated with an
opening scene that sparks social commentary and later for other
arbitrary points in the media timeline. This example becomes even
more relevant when a piece of media is replayed at a later
time.
[0161] In one embodiment, hashtags are used to add a timestamp. The
current practice of using hashtags is well known. There is
currently no widely established mechanism for describing time
offset information within a social message or as additional
metadata. The present disclosure provides a mechanism that can be
implemented without custom extensions to the social messaging
protocol. The encoded time offset is sent as part of the message
itself. It is possible to obviate this approach by providing
specific metadata that is not typically displayed as part of the
social message itself.
[0162] FIG. 21 depicts exemplary social messages 2100 that
including a hashtag identifying the media 2110 is provided (this is
common use today) with the addition of another hashtagged string of
characters encoding the time offset 2120. In this example, the
hashtag #ijkcs (2110) is used to identify Indiana Jones and the
Kingdom of the Crystal Skull. The hashtag #1054675(2120) is a
checksummed encoding of the frame offset.
[0163] FIG. 22 depicts an exemplary flowchart of a method for
creating time-stamped social messages. The method begins at block
2210. It is then determined that a user desires to create a social
message (block 2220). In this embodiment, this occurs when the user
selects a comment button provided as part of the application
running on the second screen device 520. A timestamp of the
position of the playback is then created (block 2230). In this
example, this involves generating a checksum. To create the
checksum the application receives positioning information directly
if the media is being played back within the application or from
other methods that provide position information from external
sources. This numeric data is then checksummed. The user can then
enter the text for the social message and request it be sent (block
2240). The timestamp is then added to the message and the message
is sent or otherwise committed to the social network (block 2250).
In this example, this involves appending the checksum to the social
message text. The checksum is used to ensure integrity of the data
and allow a consuming application to ignore bad position data that
might be maliciously or inadvertently created. This ends the method
(block 2260)
[0164] FIG. 23 depicts an exemplary flowchart of a method for
consuming or otherwise reading and decoding a time-stamped social
message. Once begun (block 2310) the method includes receiving a
social message pertaining to the particular content in question, in
this case, the content being displayed on the first screen device
510 (block 2320). In this example this involves the selection of a
message based on the hashtag representing the video of interest
(basic search on hashtag). The position information (timestamp) can
then extracted (block 2330). In this example, this involves the
application looking for an additional hashtag that immediately
follows and processed as an encoded position. If the position
information is found (block 2340), The position information is
separated into a predetermined position and checksum. The position
is confirmed against the checksum (block 2350). If the position
information is valid, the position can then be used to associate
the social message to a particular time within the video and the
message information can be displayed synched with the display on
the first screen device 510 (block 2360). If the position
information cannot be confirmed, the message can be discarded or
displayed without being synched with the display on the first
screen device 510 (block 2370). This ends the method (block
2380).
[0165] Assuming not all social commentary will be generated using
an application employing the techniques of the present disclosure,
the following approach may be used to modify message so they can be
consumed by applications sensitive to the position information.
[0166] A separate service (e.g. in the cloud) can monitor and
record real-time feeds of social messages of interest (say for a
particular television broadcast). This service could use known
broadcast schedules and correlate the expected time of the
television event with the time the messages are seen in real-time.
The service then records the social message with the appended
position information. The resulting repository of time-stamped
message is then accessed in lieu of a direct connection to the
social network. In certain embodiment origin information for the
social messages can be used to correlate to specific regions with
specific broadcast times.
Example
Heat Map
[0167] As discuss previously, social messaging is becoming a
ubiquitous feature across various software applications. One
problem this presents is that the sheer quantity of messages can
become overwhelming especially for popular topics. At some point
the individual messages get lost in the crowd and the quantity of
messages becomes the interesting social aspect. This disclosure
describes a way to visualize the volume of social activity over
time which can then be used to identify interesting points in time
for media content and also be used to navigate to those points in
time.
[0168] Many second-screen applications provide a social message
"feed" that simply streams social messages as they happen. There is
little organization of this information beyond displaying the most
recent at the top of the list or perhaps the notion of a "promoted"
message that advertisers use to keep their message at the top of
the stack. Messages are quickly replaced by new messages sometimes
faster than a user can scan them. Once the messages have been
buried in the stack their effective relevance to time is
diminished.
[0169] The presented graphical representation referred to here as a
Social Heatmap organizes social information to correspond with the
timeline of a of a media item such as content being displayed on a
first screen device 510. In the case of live broadcast the social
message timestamp can be used to make the correlation. In the case
of recorded content, messages need to be stamped with the time
relative to the start of the media. This can be done within the
application since the relative time of the media playback can be
acquired via syncing mechanisms. The sync time is added to the
social message which is then made available via a social network to
others using an application that is aware of the time information.
In this way each social message has a relative media timestamp
which aids in visual placement on the screen when displaying the
information on a second screen device 520.
[0170] FIG. 24 depicts an exemplary flowchart of a method for
consuming or otherwise reading and decoding a time-stamped social
message. Once begun (block 2410) the method includes receiving a
social message pertaining to the particular content in question, in
this case, the content being displayed on the first screen device
510 (block 2440). This can involve the selection of a message based
on the hashtag representing the video of interest (basic search on
hashtag) as discussed above or it could be based on keywords in the
messages themselves. Timestamp information associated with the
message is then looked for and messages without associated
timestamp information are discarded (block 2430). In certain
embodiments this involves the application looking for an additional
hashtag that immediately follows and processed as an encoded
position. Based on the associated timestamp information, the
messages can be allocated to bins (block 2450). This process is
discussed in more detail below. A graphical representation of the
social messages associated with specific time periods of the
primary content on the first screen device 510 can then be
displayed on the second screen device 520 while the primary content
is being displayed on the first screen device 510. In certain
embodiments the graphical representation of the social messages
associated with specific time periods also graphically represent
the intensity or frequency (i.e. the number) of social messages
associated the specific periods of time and as such is referred to
as a heatmap. The user can then use the heatmap to navigate the
through the content on the first screen and the associated social
messages (block 2460). Such navigation is discussed in more detail
below. This ends the method (block 2370).
[0171] Bins are mechanism used for grouping messages with discrete
sections of time. The number of messages in each bin can
corresponds to the activity level for that section of time. It has
been found that the screen width of each bin should be wide enough
to afford navigation (e.g. using a touch device) but small enough
to provide navigation resolution. An example of this can be seen in
FIGS. 25 and 26.
[0172] In the example of FIG. 25, a tablet application allocates
1000 pixels of width for the heatmap 2500. These 1000 pixels
represent the entirety of the media playback time, say 50 minutes.
This allows approximately 20 pixels of screen width per minute of
content. If we use a bin width of 20 pixels then our navigation
resolution will be to the nearest minute and the sensitivity for
navigation will require a move of 20 pixels from bin to bin. FIG.
26 depicts an exemplary plot of sample count for values allocated
to bins for rendering. The number of messages in a bin may be
graphically represented using color, plots, or other indicators.
The actual sensitivity that should be used will depend on the input
device, user demographics and other factors.
[0173] FIG. 27 depicts an exemplary wireframe screenshot 2700
including a social heatmap or timeline. In this embodiment event
panels 2710 displayed as part of a timeline view on the second
screen device 520 which a user can scroll through as described
previously. A Comment button 2720 is provided to allow the user to
send social messages regarding content being displayed on the first
510 or second 520 screen devices. Social messages generated using
such a function can include content and timestamp information for
processing and inclusion in the heatmap 2750. Sync indicator button
2730 allows the user to synchronize the events of the timeline view
with the content being displayed on the first screen device 510.
Button 2730 can also indicate the status of the synchronization. At
the bottom of the screenshot 2700 there is also a playback position
indicator 2740 and the social heatmap 2750. The playback position
indicator graphically displayed the current position in the
playback of the primary content on the first screen device 510. In
certain embodiments, a user can adjust the indicator to change the
current playback position in the primary content being displayed on
the first screen device 510. The social heatmap 2750 has bins
representing social messages associated with time periods in the
playback of the primary content. Selecting bins, for example by
sliding or "scrubbing" along the social heatmap 2750 causes the
associated social messages to be displayed. Examples of this can be
seen in FIGS. 28-32.
[0174] FIG. 28 depicts the screenshot 2700 of FIG. 27 with a
pop-over panel 2800 that is displays the social messages associated
with a bin 2810 in the social heatmap 2750 when a specific bin is
selected by a user as represented by icon 2820. FIG. 29 depicts one
embodiment of how messages can graphically be transition between
(as indicated by arrow 2900) in the popover panel as a user scrolls
or scrubs along the heatmap 2750 (as indicated by icon 2820). FIG.
30 indicates one possible embodiment of a panel 3000 that can be
displayed if there are no social messages associated with a bin
2810. FIG. 32 depict how multiple messages can be scrolled through
within a panel 3100. In this example, messages can be scrolled
through vertically as indicated by arrows 3110. In certain
embodiments a scroll bar indicator 3120 is provided to indicate
that there are multiple messages to be scrolled through. FIG. 32
depict another embodiment wherein selecting individual messages
within the panel 3200 provides the user with additional
functionality. In this example, selecting a message provides
additional buttons that allow the user to resend ("re-tweet") a
message 3210 or got the specific instance in the playback of the
content on the first screen device 510 for with the message is
associated.
[0175] FIG. 33 depicts a skinned version screen 3300 of the screen
2700 of FIG. 27. In this example color and peaks are used to
graphically indicate the intensity of social messages along the
heatmap 3310.
[0176] The present description illustrates the principles of the
present disclosure. It will thus be appreciated that those skilled
in the art will be able to devise various arrangements that,
although not explicitly described or shown herein, embody the
principles of the disclosure and are included within its spirit and
scope.
[0177] All examples and conditional language recited herein are
intended for informational purposes to aid the reader in
understanding the principles of the disclosure and the concepts
contributed by the inventor to furthering the art, and are to be
construed as being without limitation to such specifically recited
examples and conditions.
[0178] Moreover, all statements herein reciting principles,
aspects, and embodiments of the disclosure, as well as specific
examples thereof, are intended to encompass both structural and
functional equivalents thereof. Additionally, it is intended that
such equivalents include both currently known equivalents as well
as equivalents developed in the future, i.e., any elements
developed that perform the same function, regardless of
structure.
[0179] Thus, for example, it will be appreciated by those skilled
in the art that the block diagrams presented herewith represent
conceptual views of illustrative circuitry embodying the principles
of the disclosure. Similarly, it will be appreciated that any flow
charts, flow diagrams, state transition diagrams, pseudocode, and
the like represent various processes which may be substantially
represented in computer readable media and so executed by a
computer or processor, whether or not such computer or processor is
explicitly shown.
[0180] The functions of the various elements shown in the figures
may be provided through the use of dedicated hardware as well as
hardware capable of executing software in association with
appropriate software. When provided by a processor, the functions
may be provided by a single dedicated processor, by a single shared
processor, or by a plurality of individual processors, some of
which may be shared. Moreover, explicit use of the term "processor"
or "controller" should not be construed to refer exclusively to
hardware capable of executing software, and may implicitly include,
without limitation, digital signal processor ("DSP") hardware, read
only memory ("ROM") for storing software, random access memory
("RAM"), and nonvolatile storage.
[0181] Other hardware, conventional and/or custom, may also be
included. Similarly, any switches shown in the figures are
conceptual only. Their function may be carried out through the
operation of program logic, through dedicated logic, through the
interaction of program control and dedicated logic, or even
manually, the particular technique being selectable by the
implementer as more specifically understood from the context.
[0182] Although embodiments which incorporate the teachings of the
present disclosure have been shown and described in detail herein,
those skilled in the art can readily devise many other varied
embodiments that still incorporate these teachings. Having
described preferred embodiments for a method and system for
providing media recommendations (which are intended to be
illustrative and not limiting), it is noted that modifications and
variations can be made by persons skilled in the art in light of
the above teachings.
[0183] While the example set forth above has focused on an
electronic device, it should be understood that the present
invention can also be embedded in a computer program product, which
comprises all the features enabling the implementation of the
methods described herein, and which, when loaded in a computer
system, is able to carry out these methods. Computer program or
application in the present context means any expression, in any
language, code or notation, of a set of instructions intended to
cause a system having an information processing capability to
perform a particular function either directly or after either or
both of the following a) conversion to another language, code or
notation; b) reproduction in a different material form.
[0184] Additionally, the description above is intended by way of
example only and is not intended to limit the present invention in
any way, except as set forth in the following claims.
* * * * *