U.S. patent application number 14/036366 was filed with the patent office on 2014-04-03 for enhancements to unified communications and messaging systems.
This patent application is currently assigned to SOFTWARE HOT-HOUSE LTD.. The applicant listed for this patent is SOFTWARE HOT-HOUSE LTD.. Invention is credited to Christopher Douglas Blair.
Application Number | 20140096034 14/036366 |
Document ID | / |
Family ID | 40262321 |
Filed Date | 2014-04-03 |
United States Patent
Application |
20140096034 |
Kind Code |
A1 |
Blair; Christopher Douglas |
April 3, 2014 |
ENHANCEMENTS TO UNIFIED COMMUNICATIONS AND MESSAGING SYSTEMS
Abstract
The invention relates to a system and method of handling a
plurality of communications between parties. Novel approaches to
the presentation, storage and creation of both real-time and "store
and forward" interactions are presented. These make it easier for a
user of the system to read and understand what is being
communicated, by whom and to whom. This extends beyond the
presentation of a single message to sets of related messages. By
combining recording of real-time interactions with the presentation
and handling of inherently "store and forward" mechanisms, users
find it easier to mix and match communication methods optimally for
the task at hand.
Inventors: |
Blair; Christopher Douglas;
(South Chailey, GB) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
SOFTWARE HOT-HOUSE LTD. |
South Chailey |
|
GB |
|
|
Assignee: |
SOFTWARE HOT-HOUSE LTD.
South Chailey
GB
|
Family ID: |
40262321 |
Appl. No.: |
14/036366 |
Filed: |
September 25, 2013 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
13002858 |
Jan 6, 2011 |
8555178 |
|
|
PCT/EP2009/052670 |
Mar 6, 2009 |
|
|
|
14036366 |
|
|
|
|
Current U.S.
Class: |
715/752 |
Current CPC
Class: |
H04L 51/36 20130101;
H04M 1/72552 20130101; H04L 51/04 20130101; H04M 2250/60 20130101;
G06Q 10/107 20130101 |
Class at
Publication: |
715/752 |
International
Class: |
H04L 12/58 20060101
H04L012/58 |
Foreign Application Data
Date |
Code |
Application Number |
Mar 6, 2008 |
GB |
0804164.2 |
Claims
1. A computer-implemented method, comprising: identifying address
data for an interaction, the address data defining a plurality of
participant addresses for a plurality of participants in the
interaction; identifying a plurality of alternate communication
media, each alternate communication medium being associated with a
plurality of participants in the interaction whether or not the
alternate communication medium has already been used within the
interaction; displaying a representation of the interaction, the
representation including a plurality of communication icons, each
communication icon corresponding to a specific alternate
communication medium through which at least a subset of the
participant addresses may be contacted; and displaying an
indication of which of the participant addresses may be contacted
via each such alternate communication medium.
2. The computer-implemented method of claim 1, wherein a display
preference is one of a preference defined by an initiator, a cost
preference based on a cost of using each alternate communication
medium, a speed of response preference, and a coverage preference
based on a number of the participants that can be reached over each
alternate communication medium.
3. The computer-implemented method of claim 1, wherein the
interaction is a constituent interaction in an interaction thread,
and further comprising: automatically recording an alternate
communication established over a corresponding alternate
communication medium; identifying the recorded communication as a
subsequent constituent interaction in the interaction thread; and
displaying the interaction and the recorded communication as two
separate constituent interactions in the interaction thread.
4. The computer-implement method of claim 3, wherein the alternate
communication is one of a telephone communication or an instant
messaging communication.
5. A computing device, comprising: a display subsystem including a
display device; a processing subsystem in data communication with
the display subsystem; a communication subsystem operable to
communicate over a plurality of communication media; a data store
in data communication with the processing subsystem and storing
instructions executable by the processing subsystem that upon such
execution cause the computing device to perform operations
comprising: identifying address data for an interaction, the
address data defining a plurality of participant addresses for a
plurality of participants in the interaction; identifying alternate
communication media, each alternate communication medium being
associated with at least two participants in the interaction
whether or not the medium has already been used within the
interaction; displaying a representation of the interaction, the
representation including a plurality of communication icons, each
communication icon corresponding to an alternate communication
medium through which at least a subset of the participant addresses
may be contacted; and displaying an indication of which of the
participant addresses may be contacted via each such alternate
communication medium.
6. The computing device of claim 5, wherein the computing device is
a mobile communication device.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] The instant application is a continuation of U.S. patent
application Ser. No. 13/002,858, titled "ENHANCEMENTS TO UNIFIED
COMMUNICATIONS AND MESSAGING SYSTEMS", filed on Jan. 6, 2011, which
is the national stage entry of and claims priority to PCT
application serial number PCT/EP2009/052670, titled "ENHANCEMENTS
TO UNIFIED COMMUNICATIONS AND MESSAGING SYSTEMS", filed on Mar. 6,
2009, which claims priority to GB patent application serial number
0804164.2, titled "ENHANCEMENTS TO UNIFIED COMMUNICATIONS AND
MESSAGING SYSTEMS" filed on Mar. 6, 2008, the entire specifications
of which are incorporated herein by reference.
BACKGROUND OF THE INVENTION
[0002] 1. Field of the Invention
[0003] The invention relates to a system and method of handling a
plurality of communications between parties. Novel approaches to
the presentation, storage and creation of both real-time and "store
and forward" interactions are presented.
[0004] 2. Discussion of the State of the Art
[0005] "Unified Communications" (UC) Systems are well known. These
integrate a variety of disparate communication methods--including,
but not limited to, telephony, cellular text messaging ("SMS"),
web-chat (or "instant messaging") and email--so that users need not
access each one separately. Features of such systems typically
include (a) forwarding of calls from one device, number or media to
another, (b) a common message store, (c) access to a plurality of
such channels from a single device or application.
[0006] A common approach to the presentation of messages of a
variety of types has been to use an application--such as
Microsoft.RTM. Outlook--that was designed originally for email.
Voice recordings and video recordings are typically transported and
stored as "attachments" using standards such as MIME (Multipurpose
Internet Mail Extensions). The way in which a series of messages is
presented to the user has changed little from the days when people
checked their email once every few days and would converse with a
relatively small number of people at a time.
[0007] Today, many workers carry email enabled telephones such as
the RIM BlackBerry.RTM.. This encourages much more rapid and
frequent responses from a wider set of individuals than before.
However, most messaging applications continue to show a simple list
of messages with a relatively small number of columns. The user can
typically click on a column header to sort the list according to
that column. While this allows messages to be seen chronologically
or by subject it is not good at helping the user to grasp quickly
what has been "said" by whom, to whom and when.
[0008] Furthermore, the common practice of including the text of a
received email within the reply to that email leads to very long
messages--much of which is duplicating one or more other messages
already received. This makes it harder for the recipient to
identify the new or changed information within each exchange of
messages.
[0009] The list of addresses on an email is often shown in a very
limited space--making it impossible to see the entire list without
scrolling through it, a few at a time. This makes it difficult to
appreciate who has seen the message and who has been added to or
dropped from the list of recipients.
[0010] When messages include audio, this is often shown simply as
an attachment--above, below or to one side rather than in the
logical flow of the email. This is therefore difficult to place in
context. It is also difficult to appreciate, at a glance, whether
the attachment represents a few seconds or many hours of
material.
[0011] An object of this invention is to provide a system and
method better suited to the sending, assimilating and responding to
the rapid-fire, many-way, multimedia interactions that are
increasingly common in business interactions.
SUMMARY OF THE INVENTION
[0012] Accordingly, this invention provides a system and method for
presenting messages, whether sent or received, in such a way that
the reader can more quickly identify the course of the discussion
to date and can respond more easily and effectively.
[0013] Throughout this specification, the exchange of a plurality
of messages that relate to a particular topic is referred to as a
"thread". Threads may "split" or "diverge" into "sub-threads"--for
example when a user replies to a subset of the original recipients
and that group continue to exchange messages. Threads may later
"merge"--for example when someone then forwards a response to all
of the original recipients.
[0014] Each such "thread" consists of one or more constituent
interactions--which may be "messages" or genuine live interactions
in the case of real-time (e.g. telephony) or near real-time (e.g.
instant messaging) media.
[0015] Those involved in such a thread are referred to as the
"participants". An individual (or automated address representing an
endpoint) may be the sender of, recipient of or merely "copied"
("cc") or "blind copied" ("bcc") on a particular message.
[0016] The present invention includes:
[0017] a) Means for presenting the list of participants in the
thread more efficiently.
[0018] b) Means for reducing duplication of information
presented
[0019] c) Means for presenting the chronology and/or involvement of
different participants in a "thread" and constituent
"sub-threads".
[0020] d) Means for encouraging the use of the most appropriate
available medium for contacting said participants.
[0021] e) Means for recording real-time or near real-time
interactions as these are undertaken.
[0022] f) Means for viewing, editing and/or annotating said
recordings.
[0023] g) Means for sending said recordings to some or all of the
participants.
[0024] h) Means for recording "voicemail" that allow said voicemail
to be sent to multiple participants.
[0025] An embodiment of the present invention may incorporate any
subset of the above features.
[0026] The order in which method steps are listed is immaterial in
most cases and does not imply a strict ordering of steps within the
invention other than where such order is inherent in the steps.
Where the figures describe specific mechanisms within the
invention, these too may be performed in any order. Note
specifically that the display or presentation of information may
occur before completion of all of the steps. Other, (typically the
slower), steps may be performed in parallel with or after the
initial display and the display subsequently updated with the
richer content that becomes available as those steps complete.
Examples of this would be accessing a corporate directory or
looking up icons to use instead of domain names.
BRIEF DESCRIPTION OF THE DRAWING FIGURES
[0027] Examples of the invention are described with reference to
the following drawings:
[0028] FIG. 1-Traditional email client presentation of a received
email.
[0029] FIG. 2-Enhanced presentation of the email of FIG. 1.
[0030] FIG. 3-Exemplary Method by which the presentation of FIG. 2
may be derived.
[0031] FIG. 4-Traditional presentation of an email after several
responses
[0032] FIG. 5-Enhanced presentation of the email of FIG. 4.
[0033] FIG. 6-Raw Message content of Message in FIGS. 4 and 5.
[0034] FIG. 7-Exemplary Method by which emails may be related
and/or parsed
[0035] FIG. 8-Exemplary Method by which the presentation of FIG. 5
may be derived.
[0036] FIG. 9-Traditional email client presentation of a list of
received emails
[0037] FIG. 10-Enhanced presentation of the emails of FIG. 8
[0038] FIG. 11-Exemplary Method by which the presentation of FIG.
10 may be derived.
[0039] FIG. 12-Alternative presentation to that of FIG. 10
[0040] FIG. 13-Exemplary presentation of a split thread
[0041] FIG. 14-Presentation of thread designed to encourage use of
other media
[0042] FIG. 15-Presentation and annotation of an active telephone
call
[0043] FIG. 16-Presentation and annotation of lengthy telephone
call
[0044] FIG. 17-Presentation and annotation of "chopped" telephone
call
[0045] FIG. 18-Presentation and annotation of "chopped" instant
messaging session
[0046] FIG. 19-Presentation and annotation of an incoming telephone
call
[0047] FIG. 20-Presentation and annotation of an active video
call
[0048] FIG. 21-Exemplary System Design
[0049] FIG. 22 shows an exemplary user interface element, according
to an embodiment of the invention.
DETAILED DESCRIPTION
Participants in an Email Thread
[0050] Consider first, the presentation of the list of participants
associated with a received message and one's own position within
this list. The message will typically have come "From" a single
individual or source and be sent "To" one or more recipient
addresses. It may also have been copied for information ("cc:") to
one or more addresses. It may also have been "blind-copied"
("bcc:") to one or more addresses. The "blind copy" list, by
definition, is not normally visible to other recipients though it
is advantageous to be aware that one has only been blind-copied on
a message.
[0051] FIG. 1 shows a simple E-mail as printed by a typical
application. Notice the long list of "To:" (1) and "cc:" (2)
addresses. As printed, the full extent of the "To:" list (1) is not
visible unless the user clicks on the "+" symbol to expand it. On a
device with limited display area, such as a handheld computer or
RIM Blackberry.RTM. the list would normally be in a scrollable area
with only a small proportion visible at one time. Otherwise, the
body (3) of the message would not be immediately visible--forcing
the user to scroll down to see it. Other elements displayed
typically include the Subject (4), Sender (5), date and/or time
(6)
[0052] FIG. 2 shows the same email displayed using an example
embodiment of the invention. Features to note include:
[0053] a) Unnecessary labels are replaced by graphical conventions
and devices including but not limited to
[0054] i. Font style, size, weight and/or color. (Bold indicating
"From" (7), normal text indicating "To" (8) and italics indicating
"cc" (9) in this instance).
[0055] ii. Enclosing boxes (such as (10) which groups all
participants together), table cells or other shapes. Such
boundaries may optionally be configurable e.g. by dragging the
lines, corners or resizing handles.
[0056] iii. Specific positioning (e.g. bottom right for date/time
(11), top left for Subject (12)) further avoid the need for
additional textual labels. Again, these may optionally be
configurable as system-wide and/or individual preferences. Ideally,
users can drag and drop and resize elements to suit their way of
working and the space available to them.
[0057] iv. Shape, texture, fill gradient, 3-dimensional effects
etc
[0058] v. Graphical icons and/or popup (e.g. Alt-tag) text
explaining them.
[0059] Note: The above, non-exhaustive list of means for
distinguishing and displaying fields within the display to suit
one's or one's company's means of working is applicable to all
features and display elements described below but is not repeated
unnecessarily--with only the means used in the particular exemplary
figure being discussed explicitly.
[0060] b) Improved grouping and reduced text for participants is
achieved by
[0061] i. Grouping of addresses e.g. by domain (13), (14), (15)
[0062] ii. Use of company logos and/or colors where available
instead of domain names e.g. (16)
[0063] iii. Use of abbreviations (17) where these have been
configured by the user and/or the Participant--typically for those
with whom they commonly converse.
[0064] iv. The user's own address defaults to an abbreviation of
"ME". Upper case is preferred as it is more obvious when italicized
than the mixed case "Me". Optionally, this address may be
specifically highlighted (e.g. unique color) to help the user spot
it rapidly within a list of names. Similarly it may be brought to
the front of a list to make it more visible even though
alphabetical ordering would not place it first.
[0065] v. Optionally, where more participants are listed inside a
single domain than can be shown within the allocated or desired
area, a reduced set of participants may be shown with an expansion
or scroll mechanism (such as the "+" symbol of FIG. 1) inside or
next to each domain grouping.
[0066] vi. Where recipients are from many domains--such that there
is not room to show all of these in the allocated or desired area
(10), one or more domain groupings may be reduced to just the
company logo. Holding the cursor over the logo may show the list of
recipients within that domain. Clicking on a logo may expand that
domain grouping to show the names within it.
[0067] vii. Where integration to a corporate directory or other
means of determining organizational hierarchies is available, the
names within the address areas may be linked so as to show
reporting hierarchies. For example, a line with an arrow may be
drawn pointing from an individual to their manager. In addition,
information extracted from said directories may be presented when
the user selects an address.
[0068] viii. Where participants are in different time-zones and
this can be determined (e.g. permanent location from corporate
directory; current location from instant messaging "presence"
information) names may be grouped together according to timezone
and/or an indication of this may be given e.g. by a small clock
icon showing the time difference ("+5", "-2") superimposed or on a
pop-up label. The latter may also or alternatively show the current
time at their location.
[0069] c) Optional display of the time of receipt:
[0070] i. This may show relative time (difference between now and
the time of receipt) and/or absolute time of receipt (11).
[0071] ii. The background color in this example may be red, amber,
green or grey and is used to indicate adherence to agreed levels of
service (respectively, "out of tolerance"; "close to tolerance";
"well within tolerance" and "don't care")--the thresholds for which
may be triggered by sender, subject, priority, the user's personal
goals or other attributes.
[0072] iii. Holding the cursor over this area (11) or otherwise
selecting it will show the alternative display (relative time if
absolute is showing and vice versa).
[0073] iv. Clicking on the field will toggle it to show said
alternative display.
[0074] v. Right-clicking or otherwise selecting this area (11) will
provide shortcuts to appropriate functions. For example: "Remind me
tomorrow", "Send Apology for delay", "Escalate" etc.
[0075] d) Additional information not present in FIG. 1 can be added
without using additional space. For example:
[0076] i. Red subject text (18) indicates "urgent"
[0077] ii. Amber background (19) indicates an unknown external
company
[0078] iii. Green background (20) indicates a known, friendly
partner company.
[0079] iv. The reader's own domain will typically be shown in that
company's signature color (13).
[0080] v. Specific colors, shapes, or styles may be allocated to
specific domains.
[0081] e) Optional removal of whitespace including but not limited
to
[0082] i. Wholly blank lines such as (21) and (22) on FIG. 1
[0083] ii. Automatic reuse of partially blank lines (in this
example, at top right by the recipient details (10) and bottom
right by the date/time (11))
[0084] iii. Conversion of fixed pitch fonts to more efficient
proportionally spaced ones. Preferably, the presence of tab
characters or multiples spaces may be taken as an indication of
attempts at manual formatting--for which fixed pitch is important
and hence the application may leave the fixed pitch font
untouched.
[0085] iv. Conversion to a standard font size. Optionally, the user
may specify a preferred font size and type and whether all messages
should be converted to this; none or only those containing a single
font.
[0086] f) Optionally, a control (163) may be present that, when
clicked, returns the message to its original form--allowing the
user to see the impact of any of the parsing and optimization that
the application has performed. Should the message already be shown
without such optimization, this control may be disabled or
invisible. Alternatively, it may operate as a toggle. For example a
user that prefers to see messages in their original state initially
may occasionally click on this tool to condense a particularly long
message or one with many blank lines.
[0087] g) Optional optimization of field width (such as is
performed by browsers when displaying html tables. Column width
adjusts automatically to suit the contents of the fields--such as
the subject (12) and Address fields (18), (10).
[0088] h) Many users are unaware of and do not experiment with
features such as right-click. Hence the system may use means of
encouraging users to do so. For example, an icon may be used to
highlight a "tip" such as "click here to set an abbreviation for
this frequently used name".
[0089] Alternatively, a status bar, panel, scrolling ticker or
other display area may show such tips. The overall effect of these
enhancements is to reduce the amount of text on screen while
maintaining or even increasing the amount of information provided.
In this example, the full address list; message priority; company
relationship and service level warning are shown--none of which
were present in FIG. 1. This helps the user to read and assimilate
the message more quickly and better understand how and when to
respond yet with less need for scrolling or other interaction with
the display than in the standard display of FIG. 1.
[0090] Preferably, the configuration, style, position and content
of each of these fields are user definable. Preferences will
typically be set to a company-wide default and individuals given
some degree of freedom over their personal layout and style
preferences. This approach is implicit anywhere that "preferences"
are discussed below.
[0091] On initial installation, or on request, the application may
examine existing public, corporate and/or personal
directories/address books and any existing message stores that the
user configures. By looking at previous messages, the application
can identify those addresses with whom the user interacts--and how
frequently. By grouping these into domains and ranking according to
frequency/volume of interactions, the system can identify those
addresses that it is worth the user specifying abbreviations for.
By using the directory information and/or greetings (e.g. "Dear
Tim") and/or sign-offs/electronic signatures, the application can
show a list of the relatively few addresses that make up the bulk
of the recipients. A pareto (80/20) rule typically applies with 80%
or so of messages involving around 20% of the addresses.
Preferably, the system identifies the greeting used for each
recipient from existing mail messages sent from this user to said
recipient (e.g. "Dear Tim" versus "Dear Miss Brown") to determine
whether first name or surname is used. A list of probably
abbreviations is therefore built. Preferably, further processing of
this list identifies any duplicates within the list and assigns the
simpler name to the more frequently used address. The other(s) are
then distinguished using the first initial or initials of the
surname.
[0092] Preferably, the user is presented with a suggested list of
abbreviations--for example, in a grid view such as that of FIG. 22.
This figure shows a simplified version of the form used for
confirming and/or altering address abbreviations. This may be shown
at installation time or under a menu item, toolbar button or
similar command. This is also useful should the user need to check
who a particular abbreviation has been assigned to. Standard grid
features, such as sortable columns, scroll-bars etc. have not been
shown but would normally be present.
[0093] The first column (151) shows the full email address of the
individual. A further column (161) shows the number of messages
exchanged with this individual--or, optionally their ranking when
addresses are ordered with respect to the number of messages
exchanged. The central columns (152), (164), (153), (154), (155)
show automatically derived options for abbreviations. The final
column (156) allows the user to enter their own alternative--such
as a nickname (156). One cell in each row is selected (e.g. (160),
(159)) with the initial selections preferably being set according
to an analysis method such as that outlined above. The user may
change any of the default settings with a single click on their
preferred cell (or, in the case of a user entered alternative, a
click followed by entry of the required abbreviation). Note that
these same options may be provided on a right-click menu item when
pointing at an individual address in any of the displays shown in
other Figures. This makes it very easy for the user to refine the
abbreviations they use as new addresses start to appear or as
previously unusual contacts become regular correspondents.
[0094] Preferably, the system highlights and avoids the use of
duplicate abbreviations. Duplicated entries are clearly marked
(e.g. with "equals" sign" (158) as a warning that using this
abbreviation could be ambiguous or, if one of them is already in
use, a red "equals" sign (165)) so the user knows to avoid this
one. If the user does choose such an abbreviation for one of the
potentially many addresses to which it could refer, this too is
clearly highlighted (e.g. the "equals" sign in green (157)).
Alternatively, ambiguous options may be disabled or greyed out but
as the address list evolves over time, it is inevitable that a
selected abbreviation may become ambiguous later hence the
application would normally only block choices that conflicted with
an existing abbreviation at the time the choice was made.
Abbreviations are therefore unique for a given user's contact list
at all times. Should a new contact be added to the list of
addresses, the system will ensure that a unique abbreviation is
used--with the fallback position being to use the full address as
the "abbreviation".
[0095] Preferably, however, conflicts may be resolved taking into
account the domain name. For example, where two or more addresses
with forename "John" are found, one being in the same domain as the
user and the other(s) external, the former would automatically be
assigned "John" by default and the other(s) would use part or all
of the surname in addition to "John" or "J" +surname.
[0096] System-wide and/or personal preferences may be configured to
indicate the preferred order in which abbreviations are selected
(e.g. Familiar, then Initials, then Formal)
[0097] FIG. 3 shows an exemplary method by which the sender's and
recipients' addresses have been analyzed and processed for display
as shown in FIG. 2. Said method is described below:
[0098] a) The total set of known participants is extracted from the
message (174). These addresses are parsed (either side of the "@"
sign) into name and domain. Addresses are stored locally (173) and
sorted, grouped and/or indexed according to the domain from which
they come. Each participant's role is noted--To, From, cc, bcc
etc.
[0099] b) For each domain in which there is one or more
participants, the way in which said domain should be displayed is
determined (175). One or more directories and/or preference
settings may identify the domain as "internal", "partner",
"customer" or other category--each of which corresponds to a
display style and higher level grouping of domains. A logo is next
identified for the domain--ideally an existing, cached one (170)
otherwise one is retrieved from a directory (169), preference
settings (172) or, usually as a last resort, these may be obtained
from the well-known method of retrieving the file--favicon.ico"
from the domain's website (171). Where an icon cannot be
determined, the textual name of the domain may be used or an icon
of, for example, a question mark. The "alt" tag associated with the
icon is normally set equal to the name of the domain and may also
include the aforementioned "category". Optionally, the icon may
also be a hot-link to the website of the domain. These added
features allow the user to place his cursor over an unfamiliar icon
and see the name or to click on it and open a browser window onto
their website. Domains may then be ordered according to how they
are used within the message and/or to their relationship to the
reader's domain. For example, the first (typically top, left)
domain to be shown may be that of the sender. The next (if
different) may be that of the recipient. Then those including at
least one "to" address followed by those only including "cc"
addresses.
[0100] c) As the address is to be placed inside an area associated
with a particular domain, there is no need to show the domain name
against each address from that domain. Only the individual's name
need be shown. Even this may be shortened for those individuals
with whom the user interacts frequently. The preferred abbreviation
for each name is determined (176) by reference to one or more
directories (169) and/or preference settings (172). However,
because the list of individuals with whom one interacts is
constantly changing, it is often necessary to adjust said
abbreviations. Hence the abbreviation (if any, the full name if
not) is itself made a hot link. Clicking on or hovering over the
name will bring up a number of options including the full name and
domain; options to change the abbreviation to alternative,
automatically derived abbreviations (e.g. forename only, initials,
forename and first letter of surname etc.). It may also allow
manual entry of a preferred abbreviation. Further items available
on this pop-up list include details of and means for connecting to
this user via any or all communication mechanisms that user is
known or expected to support (as determined from previous
interactions, directory information and/or preference settings).
Further entries may include but are not limited to: current state
or "presence information"; job function; reporting hierarchy. The
latter may, for example, be used to position the names of two or
more individuals such that reporting lines can be drawn between
them. This makes it very clear, for example, that someone has
addressed a message to an individual and also copied their
manager.
[0101] d) The content of the message is extracted (177). Note that
this and subsequent steps may be done in parallel with the above
preparation of address fields. The content may be in plain text,
HTML, Rich Text Format (RTF) or other file format for which an
appropriate renderer may be needed.
[0102] e) If preference settings dictate, blank lines may be
removed (178); tabs may be reduced to one or more spaces.
[0103] f) If preference settings dictate, the font style and size
may be changed (179).
[0104] g) The message text can now be rendered within the available
width and any whitespace at each corner may be determined
(180)--for example by finding the largest rectangle that can be
drawn from each corner without obscuring any message text.
[0105] h) The display space is then allocated (181) as follows. The
textual addressing information to be displayed within each domain's
containing area is determined. Rather than waste space with labels
such as "From", "To" and "cc" each address may be highlighted
according to which of these categories it is in. For example, those
the message was "To" could be bold while "cc" could be italic or
normal font. Text color, background color, font style, font size
and various other graphical attributes can also be used to
distinguish between these categories. Similarly, an indication may
be given of whether the recipients have received and/or read the
message--as can be determined, for example, by the use an automated
interpretation of read receipts. The space required for each domain
display can then be calculated. This has to be combined with that
needed for the message content, subject, date/time stamp and any
other controls to be included. If preferences or display limits
dictate that there is insufficient room to display all required
fields, then an iterative process of reducing the size of address
areas by placing one or more names under a linked control and/or
applying scroll bars to one or more regions of the screen is used
to determine an optimal layout. At a mini mum, it would show the
icon for each domain involved--so that a reader will be aware, for
example, that one of their customers was sent the message. Failing
to appreciate the audience of a message can be very dangerous. If
there is then space available to show more information, those names
with abbreviations are shown next--as these are likely to be
shorter and also more relevant to the reader. By holding the cursor
over, or otherwise selecting, an area showing addresses from a
domain, the full list of addresses in that domain may be shown.
[0106] i) Fields are then displayed (182) within their bounding
areas--noting that reuse of whitespace in corners may result in a
"layered" view where some fields are overlaid on the underlying
rectangle within which the message text is displayed. Note also
that where relative time is chosen, a timer may be used to refresh
this changing value--typically once a minute.
Exchange of Information Within a Thread
[0107] The above example considered only the information presented
in the "body" of a single message. Each time a message is responded
to, the response frequently includes the whole of the previously
received message--albeit typically with address information added
and sometimes with formatting or indenting changes to differentiate
it from the new response text. Each email application differs in
how it does this and most give the user several options--such as
whether their response starts above or below the original
message.
[0108] FIG. 4 shows an example of how a typical email client would
show the message of FIGS. 1 and 2 after three subsequent replies.
In this case one user (Tim Headman) has configured his email client
to place his response after the original text (22) and (26) while
the other user (John Doe) has chosen to place his response before
the original message text (23).
[0109] Note the following deficiencies of the display of FIG.
4:
[0110] a) It is difficult to follow the flow of the
replies--particularly when a mixture of "before the original" and
after the original" preferences is used. In this case (26) is the
response to (23) which is the response to (22) which is the
response to the original message (21).
[0111] b) Where a respondent types something (27) within a previous
reply and/or the original message (21), this is difficult to
identify--particularly on longer emails where the temptation is to
view only the top or bottom of the text where responses are likely
to be.
[0112] c) Although they have in this case, not all users include
the previous message within their Response--yet this may be
important for the recipient to see.
[0113] d) Only part of the "cc" list is shown initially. The user
must click on the "+"symbol (25) next to it to expand it and see
the full list.
[0114] e) The original message's addressing (FIG. 1) made it clear
that Mary White and Jack Black were being copied the message--it
was not directed "To" them. Hence one could infer that they were
being advised of the desire to have a sales meeting whereas the
other recipients were actually being invited to it. This detail has
been lost from the display of FIG. 4.
[0115] f) It is not possible to infer from this display whether
anyone else has seen some or all of the information within it. For
example, an additional recipient could have been included on
response (23) but was not included on this, latest, response (26).
There could therefore be someone in the organization that is aware
of the thread of discussion but is not aware of the subsequent
outcome and may mistakenly believe that response (23) was the final
word on the matter.
[0116] g) The times of the intermediate messages have been
lost.
[0117] h) Other, related messages are not referenced and can only
be identified by looking at the list of incoming messages. These
include:
[0118] i. Out of Office Replies
[0119] ii. Failure to deliver notification
[0120] iii. "Read" Receipts
[0121] iv. "Received" Receipts
[0122] Note: The above "related messages" are collectively referred
to hereafter as "supervisory messages". They are typically
automatically generated, to a rigid format and convey a very
specific meaning that is usually straightforward for an application
to determine.
[0123] FIG. 5 shows the same message as formatted by an example of
the invention. Note the following advantages that FIG. 5 has over
that of FIG. 4:
[0124] a) The entire thread is shown--including the original
message (28); intermediate replies (29) and (30) and the newly
received response (31):
[0125] i. All constituent messages are shown--regardless of whether
each was included in the most recent response.
[0126] ii. All constituent messages are shown in strict
chronological order--regardless of individuals' preferences for
showing response text before or after the previous text. The user
may choose to have the original message at the top or the bottom of
the list (the latter being shown in FIG. 4).
[0127] iii. The boundary of each constituent message is clearly
delineated (32)
[0128] iv. Each constituent message can be expanded to show in full
(or to a maximum preferred size, within which a scrollable area
allows larger messages to be shown a part at a time) or collapsed
to a single line summary as has been done here for an intermediate
reply (29).
[0129] v. The user's preferences determine how many (if any)
previous constituent messages are expanded by default. In this
example, 1 is expanded (30).
[0130] vi. Optionally, a separate user preference setting
determines whether the original message will be expanded by default
or not. In this example it is expanded.
[0131] vii. The user may click on controls (33), (34) to expand or
collapse a message respectively. In so doing, the control will
toggle between (33) and (34) or vice versa.
[0132] viii. Those messages that have already been read (29) or
were written by this user (28), (30) are distinguished from those
that this user has not yet read. In this example, (31) has text in
a stronger color than the others which are greyed out.
[0133] ix. In a more complex exchange between more than two
parties, intermediate messages may not have been read by this user.
In this case, these would automatically be "expanded" even if the
user preference is for such previous replies to be compressed.
[0134] b) Where a respondent types something (27) inside the lines
of a previous reply and/or the original message (21),
[0135] i. This is highlighted--in this example, by a different font
color, background color, boxing (35) and connector (36).
[0136] ii. Should such an "insert" be made into a constituent
message that would not otherwise be shown "expanded" (e.g. (29))
then its presence will automatically cause that message to be
"expanded".
[0137] iii. Where the whole text of said expanded message is not
wholly visible (e.g. scroll bars are needed) the scrollable area
will be moved automatically so as to show the start of the inserted
text.
[0138] iv. Where multiple inserts have been made into a constituent
message, additional connectors may be shown (e.g. at bottom right
of the surrounding box (35)). Clicking on said connector will
scroll the text to the next insert.
[0139] v. Said subsequent inserts may also show a connector back to
the previous insert. Clicking on this will scroll the content to
said previous insert.
[0140] c) The same techniques described for FIG. 2 are used to
reduce the space taken up by addresses. In addition, since multiple
messages have now been exchanged and some of these may have been
sent by this user:
[0141] i. Where Read Receipts have been requested--but not yet
received, this is indicated. For example, by greying out the text
of said participants (37). Clicking on, hovering over,
right-clicking or otherwise activating said indicator or
participant name may show further details--such as time the message
was read It may also offer appropriate shortcut options triggered
by the absence of aforementioned receipts such as "Resend to
different address", "Resend to Superior"--the latter being possible
where a corporate hierarchy has been or can be determined from a
directory or manually configured organizational chart.
[0142] ii. If a "Mail delivery failed" message has been received,
this indicates that an intended recipient has not received and will
not receive it. This can be indicated by an icon (38) or other
mechanism. Clicking on, hovering over, right-clicking or otherwise
activating said indicator or participant name may show full details
of the message such as the reason for the rejection. It may also
offer appropriate shortcut options such as "Resend to different
address" etc.
[0143] iii. If an "Out of Office" reply has been received this may
be indicated. For example, by an icon next to the individual's name
(39). Clicking on, hovering over, right-clicking or otherwise
activating said indicator or participant name may show the text of
said Out of Office reply. It may also offer appropriate shortcut
options such as "Resend to different address" etc.
[0144] iv. Each constituent message may optionally show only a
subset of the full list of addresses it would show if it were to be
presented in isolation. Said subset is typically those addresses
that differ from the message above (or below according to user
preference). In this example we can see (40) that the original
message was sent "To:" Jane S and Joe B (not in italics) and has
been received by them (not greyed out) even though they are only
being copied on the subsequent responses (as they are shown in
italics in (37)).
[0145] v. To see the full list of recipients of a constituent
message (or just those not already showing) the user may click on
or otherwise select a control--shown here as an ellipsis (41). In
the example shown, the differentiated color and underscore indicate
an HTML link and hence signal to the user that this is a clickable
control.
[0146] d) The time of each constituent message is shown (42), (43).
In addition to the features of FIG. 2, an additional, third time
representation is supported for the previous messages in the
thread. This shows the time difference between successive messages
(as shown by (43)). According to user preference this time span may
be relative to current time; to the most recent message or to the
preceding message. In addition to a more efficient view of the
latest response, FIG. 5 is therefore a single, consolidated view of
the entire thread and hence avoids the need to see intermediate
responses, Read Receipts, Out of Office Replies and Message
Rejections.
[0147] FIG. 6 shows the raw message content as received by the
email application for the message of FIGS. 4 and 5. To allow a
larger font to be used, the text is broken into two containing
frames but that on the right continues below that of the left as
part of the same message. Various fields within this are used to
determine the information shown in FIG. 5. Note that much of the
"missing" information discussed under FIG. 4 is typically still
present in this user's (John Doe's) mailbox--as he will typically
still have the original message in his inbox folder; his own
responses in his Sent Mail folder and subsequent responses from Tim
Headman also in his inbox.
[0148] FIG. 7 shows an exemplary method by which a received message
may be matched with one or more previously received, sent or stored
messages and an overall set of messages relating to the thread as a
whole can be determined. Note that In addition or as an alternative
to the use of the Message IDs shown in FIG. 6, the application may
also insert hidden (or visible) information within the body of the
message and/or additional header fields to facilitate the matching
of messages to threads as described below.
[0149] Using this process on all received messages, including Out
of Office Replies, Delivery Failure Notifications, Read receipts
etc. it is possible to group all received emails into the
appropriate threads.
[0150] The method of FIG. 7 builds the set of messages in each
thread as follows:
[0151] a) The ID of the newly received message is extracted (186)
from the MIME content (183). If this message is not already known
and stored in a "thread" (194) then a new "thread" is created and
this message is placed in it.
[0152] b) The set of all related message IDs is extracted (189)
from the MIME content (184) and held in memory.
[0153] c) The ID of the message (if any) that this newly received
message was a response to is extracted (190) from the MIME content
(185). The application then attempts to locate within the existing
sets of messages already assigned to threads. Should it find the
message already in a thread, then this message is also assigned to
that thread rather than continuing to build a new set of messages
for a new thread. If the message is not already known as part of a
thread, the application accesses the message store(s) (187) to
which it has access. If the message can be found, then the process
of FIG. 7 is repeated (191) for that message. This recursive
process ensures that threads are fully populated even if the
application is introduced after several messages have been sent in
a thread and hence are not yet linked together.
[0154] d) The recursive processing of the "In Reply To" message is
likely to have resulted in most, if not all, of the other Message
IDs being added to the set of messages in the thread. However, if
any of these IDs are not present, one is selected (192) and the
same recursive process (191) applied to it as was done for the "In
Reply To" message.
[0155] e) When all related messages have been processed, the set of
messages (195) should be a complete representation of the related
messages within a thread.
[0156] Note: To avoid indefinite looping, a maxi mum depth and/or
number of messages in a thread may be set. On reaching such a limit
the method will no longer make recursive calls.
[0157] Where related messages are not accessible or where Message
IDs are not available, the application may classify messages to
threads using other means. For example,
[0158] a) Comparing the subject of the message in question with
that of previously sent, received or stored messages or those
already grouped according to the invention. Preferably, this
matching process takes into account the presence of common prefixes
such as "RE:" or "FW:" that signify a reply or for war ding of a
message. More preferably the algorithm also eliminates changes to
the "subject"--such as the insertion of well- known strings such as
"SPAM?".
[0159] b) To provide better discrimination, particularly in cases
where the same subject line is used for multiple threads, the list
of recipients and/or sender may be used to further differentiate
and group messages.
[0160] c) The body of the messages may be compared. On finding the
body of one message within that of another one may infer that the
latter is a response to the former. Advanced matching techniques
may take account of indenting or other marking techniques used by
email clients when composing responses that include the original.
These mean the match is not identical but is predictable and can be
determined in most cases by comparing the possible outcomes of a
reasonable number of commonly used permutations.
[0161] d) Even where no similar messages are available for
comparison, the message itself can be analyzed to determine with a
good degree of accuracy, the sequence of responses shown within it.
A relatively small number of email client applications are in
widespread use and each of these has well established conventions
of how they sign messages, how the original message is presented
within a response etc.
[0162] e) Ideally, some form of unique topic or thread identifier
is used. This may be a trouble ticket number or inquiry number
already present in the message or its subject--or it could be
inserted deliberately to facilitate this mechanism. Such a token
may be hidden or invisible to the human reader if its sole purpose
is to link related messages. Where a company uses a particular CRM
system or convention for customer account numbering, a filter that
recognizes such patterns may be defined. A weighting method or
statistical techniques such as Bayesian filtering may then be used
to determine the significance of the address list, the subject etc.
to improve the accuracy and discrimination of clustering of
messages into threads.
[0163] FIG. 8 shows an exemplary method by which the received
message and any available related messages may then be parsed and
processed so as to deliver the enhanced presentation of FIG. 5.
[0164] a) The method is called recursively on any related messages
that have not already been parsed by this method.
[0165] b) The email client used is determined (196) by reference to
the MIME source of the message (199).
[0166] c) The body of the MIME message (200) is then parsed
according to the known formatting applied by widely used email
clients. In this example, each ">" at the start of a line of
text indicates this came from one email further back in the chain.
Lines with no ">" characters at the start are newly added as
part of this message. Furthermore, in this example, where
successive non-blank lines start with the same number of ">"
characters, it can be inferred that there is no hard carriage
return between the content of these lines. A close approximation of
the original text of each message in the response sequence can be
determined.
[0167] d) Even without the original text of the other emails, a
good approximation can be made of this and any inserts into it
(e.g. (201)) noted. Where the original text of the other messages
is available (194) a more detailed comparison can be made to
identify and highlight any other changes (198).
[0168] e) The newly added content that is not interleaved within an
earlier response is now formatted for display (202). This and the
subsequent steps may include any or all of the previously discussed
display optimizations,
[0169] f) A combination of user preferences (172) and the newly
determined knowledge of which, if any of the previous responses
have had material added or altered determines (203) which of the
related messages are to be shown in expanded or compressed
(summarized) form.
[0170] g) The display of those (now expanded) previous messages
into which text has been inserted or modified is created (204)
using knowledge of the differences between original and current
message. Preferably the changes are summarized in a familiar
form--such as that used in "red-lining" documents.
[0171] h) Linking lines, icons or other devices are created (205)
to link the new message to the appropriate points within the
previous messages.
Viewing Multiple Threads
[0172] Consider now the case of a user who has been offline for
some time or has otherwise received multiple incoming messages that
have not yet been viewed. When he next comes to view his inbox,
there may be several email threads active, each of which may have
had several responses. The longer the user has been online, or the
more overloaded he is, the greater the number of threads and the
greater the average number of interactions on each thread. Against
this, the larger the list of emails, the more likely it is that
someone else has already responded--making it no longer necessary
for this user to do so. It is therefore much less efficient to
catch up with the latest exchanges by simply reviewing each in turn
and hence "clear one's in-box" than it need be.
[0173] FIG. 9 shows a typical email client application window that
allows the user to click on a column header to sort the email by
one of several fields--such as received date, sender or topic. The
Upper section (43) shows a relatively simple set of messages with
one thread roughly corresponding to the previous examples. The 11
messages resulting from this "thread" are interleaved with 3
single, unrelated messages. The other three messages are swamped by
those from the "Sales Meeting" thread and are difficult to pick out
at a glance.
[0174] Note that due to the incremental evolution of these examples
and a desire to show specific enhancements rather than overwhelm
the reader with all permutations, the timestamps, Return Receipts
etc. are not entirely consistent with those shown in FIG. 5).
[0175] Some applications, such as Mozilla.RTM. Thunderbird.RTM.
2.0.0.9 from which this screenshot was taken, also offer sorting
messages "by threads". By clicking on a dummy column header (44),
the messages are grouped into threads as shown in (45).
[0176] The existing columnar and simplistic threading approaches
shown are not optimal--especially when catching up on a large
number of unread items. Specifically, they have the following
disadvantages:
[0177] i. Where message titles are not consistent, messages are not
always grouped into the appropriate "thread" even though they
relate to one another.
[0178] ii. Return Receipts are not grouped with the originating
thread to which they refer. They actually cause so much
clutter--especially when dealing with large groups of
recipients--that this can be a significant deterrent to their
use.
[0179] iii. Similarly, delivery failure messages are not grouped
with the thread to which they refer.
[0180] iv. Said delivery failure messages are cryptic and difficult
for non technical users to understand
[0181] v. Until the first response (whether a genuine response or a
read receipt, failure notification etc.) is received, there is
nothing to indicate that a thread is active. The user would have to
view Sent Items to remind himself of the outstanding thread(s) that
he has initiated.
[0182] vi. Although the user can choose which columns are shown,
many of these are sparsely populated (e.g. Attachments, Priority
etc.) and showing a large number of them is not practical given a
limited screen width.
[0183] vii. It is not clear who has received or read which
messages--especially intermediate responses made by others in the
chain.
[0184] FIG. 10 shows the same set of messages presented by an
example of the present invention. Features to note:
[0185] a) A message may appear in the list even though no responses
have been received for it. The list is therefore not simply
providing a new view of the traditional "Inbox" but also includes
some messages that would previously only show in the "Sent" and or
"Outbox" folders.
[0186] b) The nature of each message may be indicated (162) so as
to differentiate email, Instant Messaging, telephone, video etc.
This symbol may also indicate whether the interaction was a two-way
or multi-way one.
[0187] c) The background color, texture or other graphical
indication of the message thread's subject column (58) may indicate
message threads that this user initiated (59); is a participant in
(was sent the message as a "To" address) (60); was merely copied on
(61) or was blind copied on (62).
[0188] d) Messages are grouped into threads (46) even though the
Subject may differ--with "RE:" or "FW:" prefixes.
[0189] e) The priority of a message thread may be indicated by text
color (57), background color an icon or other graphical device.
[0190] f) As in FIG. 9, whether or not a message has been read can
be indicated e.g. bold in the Subject column (58) indicates an
unread message.
[0191] g) Within a thread, rather than needlessly repeating the
known subject, the start of the text of subsequent messages is
shown (54).
[0192] h) The user may select a thread--and have focus within that
thread directed to a specific Message--by, for example,
double-clicking on any line within the Subject column (58). This
will typically bring up a window such as that of FIG. 5 showing
this thread in more detail.
[0193] i) "Housekeeping" messages such as Out of Office Replies,
Return Receipts, Delivery Failure notifications etc. are not shown
as separate messages. Rather, they are used to mark recipients
appropriately. For example, in FIG. 10, an orange address (47)
indicates an Out of
[0194] Office Response; greyed out address (48) indicates no return
receipt yet and a red address indicates a delivery failure (49).
Clicking on an address or hovering over it will pop up an
explanation of the status along with appropriate text e.g. content
of the Out of Office Reply or the reason for non-delivery. It may
also offer options to resend etc. as previously.
[0195] j) As in previous examples,, addresses are grouped according
to domain Background color and logos are used to identify companies
and groups of companies. However, in this view, three columns are
used to highlight the sender (55), internal (50) and external
recipients (51) respectively. As before, a set of mappings of
domain name to company name, type, color and logo can be used. Note
that, although only shown (70) in the "From" column of FIG. 10,
each individual address shown within any column may have specific
formatting and/or graphic symbols associated with it.
[0196] k) As in previous examples, where lists of addresses have
been truncated to fit the available space, selecting by hovering
over, clicking or otherwise on an ellipsis control (56) will show
the full list represented by it.
[0197] l) Blind copy (bcc) status, by its very definition, can only
be known if one is a blind recipient oneself or if one is sending a
message. As it is important to be aware of this status, it may
warrant a more obvious differentiation than the bold/italic/color
indications previously discussed as differentiators between "To",
"cc" etc. The use of parentheses (68) or some other graphical
symbol or construct is recommended to highlight this status (where
it can be known).
[0198] m) Optionally, the display of addresses (or their
abbreviations) in the two "To/cc" columns may be formatted such
that the order and/or spacing of the addresses is consistent--even
if the list of address changes from one message to the next within
a thread. This serves to highlight any changes to the list as those
deliberately dropped from specific exchanges will be conspicuous by
the gap appearing on a particular line where the lines above and/or
below show a name. Optionally the gap itself may be highlighted
e.g. (69) indicates that Jane S was not copied on this message. The
user can therefore infer that her address was deliberately added
back by Tim in his subsequent response. FIG. 10 actually shows a
further refinement where the fixed order and spacing applies to all
participants except those actually sending messages--whose
addresses are brought to the front of the list first.
[0199] n) As in previous examples, date and/or timestamps can be
shown as absolute (52) or relative times or as time differences
between successive messages within a thread (53). As before,
adherence to service levels can be indicated by background color or
other graphical device.
[0200] o) The order of rows within the table can be altered by
clicking on sort buttons at the top of each corner. FIG. 10 only
shows these (63), (64) on the Subject column (58) but they may be
present on any of the columns. In addition to the standard
alphabetical or numerical sort provided by these buttons,
additional sort orders can be achieved by clicking on the "rainbow"
icon (65). This sorts the table according to the other attributes
shown in that column--typically the background color followed by
one or more secondary/tertiary sort(s) on appropriate field(s). For
example, in the Subject column, this would sort by background color
(this user's participation in the thread--sender; recipient; blind
copied; copied to) followed by text color (priority), followed by
subject. In the date/time column, this would sort by color
(adherence to required response--red/green/amber/grey). Repeated
clicking on the rainbow icon (65) may result in a simple reversal
of the primary sort field order or may invoke the next of a cyclic
list of many possible sort orders (since there is no one definitive
way to order colors). For example, where there are only four
possible colors four successive clicks of this control may sort
each of the four colors to the top of the list in turn.
[0201] p) For each message thread shown, there is a control (66)
that the user can click to remove the thread from the active list.
This does not delete the message(s) associated with the thread.
Instead, it simply removes them from this task oriented view of
active threads.
[0202] q) One of the benefits of the display of FIG. 9 is that
every incoming message--even if it just an Out of Office Reply or
Read Receipt--is presented as a new incoming message and hence
invites the user to check and acknowledge it. The display of FIG.
10 deliberately avoids this but in some cases it is important to be
made aware of changes to the status of the thread due to such
responses. A graphical device, such as a star (67) may be used to
indicate a change in the status of a recipient e.g. has received,
will not receive etc. By hovering the cursor over said star, the
user is informed (e.g. via a pop-up or alt-tag) what the star is
highlighting (e.g. "Out of Office Reply received from Sam Outsider
Sorry I'm out till Monday."). Clicking on said change indicators
acknowledges the alert and removes it. In one variant a single
change indicator (67) may accumulate changes for the whole column
within which it is displayed--allowing review and acknowledgement
of all that relate to that column at once. In a further variant,
said change indicator (67) may relate to the entire row or even an
entire thread.
[0203] To facilitate the display of FIG. 10, a number of changes
are preferably made to the window or form within which one composes
a message. Specifically:
[0204] a) The "Send" button is replaced by two buttons or
controls--forcing the user to choose one or the other. These
are:
[0205] i. "Fire and Forget"--sends a message as at present. Does
not subsequently show it in the list of messages unless someone
subsequently responds to it. Does not request read receipts.
[0206] ii. "Send and Track"--sends a message as at present but also
requests read receipts and immediately makes the sent message
appear in the list view of FIG. 10--even if no-one has yet
responded to it.
[0207] b) An optional control or group of controls is presented.
These allow the user to specify a timeframe within which a response
is expected. Typically this will include a small number of preset
options (e.g. 1 hour, 24 hours, 72 hours) but may also allow for a
specific duration e.g. days, hours or a specific date/time deadline
to be entered. This expected response period is then used to drive
the color coding and other actions associated with the date/time
fields of FIGS. 2, 5 and 10.
[0208] c) A checkbox labeled "Done when all have read" or words to
that effect. Typically the default setting for this option is for
it to be unchecked. If checked, the application will automatically
remove the thread from the list of FIG. 10 as soon as read receipts
have been received from all recipients.
[0209] d) One or more checkboxes from a set that determine whether
or not the user wants to be positively advised (e.g. via a change
indicator (67)) when messages such as Out of Office Replies,
Delivery Failures, Read Receipts etc. are received. The alternative
being that the display simply shows the current status and the user
looks to see if, for example, everyone received the message. If
using the response timer (b) above, then any message that was not
received by all intended recipients after the due time will be
highlighted as a whole--and the specific out of adherence parties
also highlighted--inviting the user's attention.
[0210] e) Where addresses are shown (e.g. as part of a "reply" or
"reply all" command) these are traditionally either preceded by a
drop-down combo box allowing the user to choose "To", "cc" etc. or
are grouped into a list within separate text entry areas for "To" ,
"cc" etc. This makes it cumbersome to simply switch between "To"
and "cc" or vice versa. The other options ("bcc" etc are far less
frequently used. It is therefore advantageous to show a control
that allows single-click changes between the most common options. A
further control may be provided to allow the other options to be
selected.
[0211] f) Within the grid of cells, the user may select one or more
individual messages and/or complete threads (e.g. using similar
techniques to those use in Microsoft.RTM. Word to select a line or
paragraph). In combination with right-click menus, menu items,
toolbar buttons, drag and drop or other control mechanisms this
allows the user to:
[0212] i. Delete or move a set of messages with a single
instruction--rather than having to act on each one.
[0213] ii. "Merge" one or more threads with previously read
messages and/or other "threads" that relate to the same topic--even
though the automated system did not consider them to be from the
same thread. A simple drag and drop mechanism may be used for
example. By dragging one thread onto another and "dropping" it, the
two may be merged and redisplayed as a single thread.
[0214] iii. "Split" a thread into two or more separate threads--for
example where the user recognizes that the automated clustering has
grouped two sets of messages together that are not really part of
the same topic.
[0215] FIG. 11 shows an exemplary method by which the presentation
of FIG. 10 may be derived. Said method is described below:
[0216] a) Any newly received messages are processed (206) according
to the algorithms already described.
[0217] b) Details of the threads to be displayed (by default only
those marked "active") are retrieved (207) from the local data
store (194) and the content of the messages involved is retrieved
where necessary from existing message folder(s) (187).
[0218] c) The set of parties who are contributing to each thread is
determined (208) so that the fixed ordering option can be
used--leaving gaps where parties are not included on a specific
message.
[0219] d) The other summary information--subject, first line of
content and timestamp is constructed (209) for each constituent
message. The timestamp preferably uses shortened, relative
timestamps thus avoiding redundant information being
displayed--such as the year and month of receipt when this is the
same for all messages; removing the date portion altogether where
this represents "today". Alternatively use relative time labelling
e.g. +1 hour, +7 days etc.
[0220] e) The entries are then sorted (210) according to the
current or preferred sort order. Grouping and ordering of threads
and the messages within them is not only performed according to
time of receipt, sender, priority etc. as existing email clients do
but also according to other factors which are likely to influence
the order in which the user should review the messages. For
example, whether or not this user was sent the message directly
("To") or merely copied it ("cc"); or blind copied ("bcc") whether
or not anyone else has already responded to it.
[0221] f) The sorted items are then displayed (211) in the
grid.
[0222] FIG. 12 shows an alternative display. Instead of presenting
each "thread" and acting on but effectively removing the Out of
Office Replies etc. the user may be presented with a list of
messages that is very similar to that of FIG. 9. Features to note
where this display differs from that of FIG. 10 include:
[0223] a) The relationships between messages may be shown to the
user by means of highlighting, joining with lines or other
graphical indicator (212) that alerts the user to the fact that
several of the items are related This approach is similar to that
used in Microsoft Excel.RTM.--in which the cells used in a formula
to derive the value of another cell are highlighted whenever the
derived cell is selected. FIG. 12 shows an example of how this
linkage would appear when the user selects or hovers his cursor
over any line in the "Sales Meeting" thread e.g. (73).
[0224] b) Cryptic standard subject headings e.g. Return Receipt
(displayed)--Re: Sales Meeting" are translated into a more user
friendly "Sales Meeting: VIEWED". Likewise for Out of Office
Replies and Delivery Failure Notifications.
[0225] c) The replacement subject lines are deliberately designed
so as to preserve the original subject heading--making the screen
much clearer when sorted on Subject column (71).
[0226] d) Messages within a thread are automatically numbered once
more than one has been sent. This number is included within the
Subject line (75).
[0227] e) Where space exists after the Subject, the start of the
content may be displayed (72).
[0228] f) Whether or not the original message sent by this user to
start a thread appears in the list is selectable by user
preference. In FIG. 12 the user has elected not to show it.
[0229] g) The user may select a thread--and have focus within that
thread directed to a specific Message--by, for example,
double-clicking on any line within the Subject column (71). This
will typically bring up a window such as that of FIG. 5 showing
this thread in more detail.
[0230] h) For supervisory messages, the "From" column is, by
default, not used.
[0231] i) For supervisory messages, the To/cc columns are, by
default, filled in with the address(es) specifically affected by
that message. The user may still click on the ellipsis link (73),
(74) to see the full internal or external address list e.g. to
check if a suitable alternative recipient has already been included
or not.
[0232] j) Read Receipts from a recipient who has subsequently
responded are largely superfluous and hence optionally, can be
suppressed. Note that the 18:03 Read Receipt from Tim Headman on
FIG. 9 is not shown in FIG. 12. It is only if the time of receipt
is important that a user would choose to show such messages. If the
user occasionally needs to see when someone received a message--as
distinct from the time of their reply--they can do so by hovering
over or selecting their address in one of the "To/cc" columns at
which point details including time of receipt would be shown.
[0233] k) Optionally, where read receipts have been received for
multiple messages within a thread, these can be merged into a
single entry showing the individual message numbers and/or ranges
to which they refer (76).
[0234] l) The user may acknowledge a message by clicking on the
appropriate control (77) on that line. The message will then be
removed from this view. Optionally, to avoid frequent refreshes and
unnecessary network and server traffic, the system may simply note
such selections and apply all of them at a later point e.g. when
the user clicks a further control, such as (78) or automatically
when the screen is next refreshed for other reasons.
[0235] It will be appreciated by one of skill in the art that many
of the details and features described on FIG. 10 may also be
applied to the presentation style shown in FIG. 12 and that hybrid
approaches involving elements from each can be derived.
[0236] Preferably the method by which the presentation of FIG. 10
or 12 is achieved entails processing each message as it arrives and
updating a set of thread details that are therefore continually
kept up to date--thus minimizing any delay that would otherwise be
incurred when the user asks to view the set of messages.
[0237] Note that a system supporting multiple users may benefit
from server-side and/or pre-emptive processing and caching of the
thread model since the view presented to multiple users will be
similar in many respects. This also allows the addition of features
not normally possible in a standard email system. For example, such
a system may know exactly who has read and responded to each email
in a thread in more detail than can be obtained simply from
standard email read receipts. The length of time each individual
has spent viewing or composing a reply may also be of interest to
others viewing the thread.
Splitting and Merging Threads
[0238] Now consider the more complex case of multiple independent
responses to an original message. There is no longer a single
"master" message being built that contains all of the previous
responses. Instead one will have two or more messages that may at
first sight look similar but differ somewhere in the
bodies--perhaps several pages down. It is much harder now to keep
track of everything that has been said on the topic.
[0239] FIG. 13 shows the presentation of an email thread that has
first split into two branches or subthreads as two of the original
recipients respond independently. One of these responses is further
differentiated in that was only sent to a subset of the original
participants.
[0240] The example shown here is deliberately kept as simple as
possible with respect to the other features that have already been
discussed. For example, Read Receipts are not used; only internal,
abbreviated addresses are shown whereas combinations of internal
and external domain addresses are present in the general case.
These features can, of course, be combined with the previous
features in the overall system.
[0241] This scenario, in chronological order, is as follows:
[0242] a. John Doe sends the initial message to Jane Smith and Joe
Brown (his co-workers) and copies his line manager, Tim
Headman.
[0243] b. Jane clicks "Reply All" and responds.
[0244] c. Tim clicks "Reply" and responds--to John only.
[0245] FIG. 13 shows how the above scenario appears to John Doe as
he logs in to find the two independent responses waiting for him.
Note:
[0246] a) The two unread responses are both expanded automatically
even if the user's preference is only for the one most recent
message to be expanded.
[0247] b) Both responses are indented from the left to the same
extent--indicating that they are peers rather than nested
responses.
[0248] c) A "Privacy" control (86) may also be shown alongside each
constituent message. Unlike those shown at the right, this is a
toggle control that switches between two states (e.g. shows open
padlock as in FIG. 13 or closed padlock once clicked). Clicking
this control toggles the state of this constituent message between
"Private" and "Public". User and/or system preference may determine
the default privacy state for a received message. A message
received with a "Confidential" indication within it will
automatically be marked as private. This setting determines whether
or not the constituent message is included in any responses made to
the thread as a whole (see below).
[0249] d) As there is room to fit all of the addresses within the
address boxes shown, there is no need to display the ". . . " link
as could have been done for addresses that are the same as the
preceding (lower) message. (Tim is cc'ed on both this and the one
below).
[0250] e) Next to each message are one or more controls (79), (80),
(81}, (82). The user can click on the appropriate control to Reply
to the sender of this message (79); Reply to all addresses that are
listed in this message (SO); forward the individual message to one
or more addresses (81) or remove the message from accumulating
thread content (82)
[0251] f) Optionally, the aforementioned controls are selectively
disabled (e.g. greyed out or removed as in FIG. 13) where they are
not applicable. For example, on messages one sent (83), there is no
"reply" control; on messages with a single recipient (84) there is
no "reply all" control. This customization of the control set to
suit the message it is shown alongside helps the user to appreciate
that these controls relate to the specific message rather than to
the accumulated thread content. Using these controls the user can
participate in either of the two branches that have started with
Jane's and Tim's responses as follows:
[0252] i. Reply to the sender of this message only. (Not offered
for messages sent by onesself),
[0253] ii. Reply to the sender of this message, copying all those
who also received it. (Not offered for messages where the user was
the sole recipient). Where a message has been marked "Private" a
confirmation dialog will be shown to the user before proceeding
with such a response.
[0254] iii. Forward this message to one or more addresses to be
entered manually. Where a message has been marked "Private" a
confirmation dialog will be shown to the user before proceeding
with such a response.
[0255] iv. Remove this message from the thread. Does not delete it,
simply hides it from view. Returning to a traditional view of the
"Inbox' or other containing folder would still show it.
[0256] g) A similar set of controls is shown at top right (85).
These perform the same functions but for the thread as a whole. The
user can therefore use these controls to:
[0257] i. Reply to the sole other participant in a thread (not
offered in threads with multiple addresses shown)
[0258] ii. Reply to all. This is used to bring all participants in
the thread up to date with the discussion as not all will
necessarily have received all of the messages in the thread. A
message will be composed that contains all of the constituent
messages in the thread apart from those that have been removed from
the thread (e.g. via control (82)) or made "Private" via control
(86). This message is addressed to the same "To", "cc" and "bcc"
lists as were used on the original message that started the thread.
Any additional participants that have been added subsequently are
then added to the "cc" list by default. The user has the
opportunity to review and manually alter the address lists and/or
message content before sending the message.
[0259] iii. Forward. This is used to bring a new party up to date
with the exchanges. The message content is built as for "Reply to
all" above but the address list is left blank for the user to
populate manually.
[0260] iv. Remove this thread. This is used when the user
determines that the thread is no longer "active" and need not be
viewed by him any further. As with the removal of constituent
messages, this action does not, by default, delete the thread but
merely marks it as inactive. A separate view may be provided to
list or browse through inactive threads and mark them "active"
again.
[0261] In more complex threads than that shown in FIG. 13, the user
may be given the option of ordering messages down the page
according to thread/sub-thread (a "nested" view) or
chronologically--in which case links between constituent messages
in a thread may be shown using connecting lines similar to (36) of
FIG. 5 or dynamically appearing linkages and/or highlighting such
as Microsoft.RTM. Excel.RTM. uses to show which cells are used in a
formula.
[0262] In a preferred embodiment, the messages generated by (g)
above may be individually tailored for each recipient. Rather than
sending the same composite message--that contains all constituent
messages (bar those marked "Private")--participants in the thread
will, by definition already have been sent at least some of the
content of the constituent messages. Knowing how each constituent
message was sent to, the application can remove much of the
therefore redundant information. If the recipient is also a user of
this application, then the view they will subsequently see of the
thread as a whole will be performing the same removal of redundant
information--hence there is no loss in value yet transmission costs
and time are reduced. If, on the other hand, the recipient is not a
user of the application, they will not have access to the display
capabilities and would appreciate the whole thread being visible
within a single message. Optionally the application may enhance the
content of the message (e.g. using html and/or scripting) to show
the user a view that has some of the benefits that a user of the
application sees. For example, placing each constituent message
within a collapsible and scrollable frame. Preferably, the
application includes a reference to said application--thus
encouraging anyone receiving such a message to consider using said
application themselves.
[0263] Note that an additional benefit of using the thread level
controls (85) is that the diverging branches can be brought back
together again--as everyone is sent a common message (which may not
include all constituent messages--some having been marked private)
but is now the "common" view of more participants.
[0264] It will be appreciated that all or any subset of the same or
functionally equivalent controls, linkages and graphical devices
can be added to the more compressed list view of FIG. 10.
[0265] In the line per message view of FIG. 12, messages may be
identified as belonging to specific branches using decimal, mixed
alphanumeric or other conventions (e.g. 2-1 would be the first
response to message 2 where 2-2 is a second response to the same
message.)
[0266] The invention detects such splits and merges primarily by
changes to the address lists on the messages. The exclusion of one
or more participants or the re-inclusion or one or more is an
indicator of such a "split" or "merge" respectively.
[0267] Where the participant list changes, the system of the
invention may choose to display the list of remaining participants
and/or the list of those no longer participating. For example in an
exchange between 20 individuals, if a branch appears with the 18
internal participants but not the two customers this may be more
informatively and succinctly highlighted as "excluding" the
latter.
[0268] Alternative methods can be adopted where this is a common
requirement. For example, a change to the subject (typically adding
a suffix) can be used to force a deliberate split even if the set
of participants is unchanged. This is particularly effective for
the user of the application who can indicate his intent to the
application while responding.
[0269] Preferably, many of the formatting features described above
can be incorporated within the html form of the body of any
response being sent by the user of the invention. In this way,
those using traditional email clients become aware of some of the
power of the invention.
Facilitating Use of Most Appropriate Medium
[0270] The foregoing features of the invention address the
presentation of messages as threads of Interaction--but only within
a "store and forward" messaging system. By presenting the stored
messages more effectively, the user is better placed to assimilate
the information and be able to respond. However, there is a great
temptation to respond through the same medium. Even if some
messages have audio or video attachments, the application typically
feels like an email system and encourages response via email.
[0271] The term "Unified Communications" currently relates to the
infrastructure and devices used in communicating over a variety of
mechanisms. However, the client application or end-user tool for
interacting with these rarely, if ever, spans both real-time
("synchronous") and messaging ("asynchronous") communications. Even
recent high profile products such as Microsoft.RTM. Office
Communicator 2007 and Microsoft.RTM. Outlook are aimed firmly at
one or other of these. There are links between them; such as
Voice-Mail but each still has its own user interface. This makes it
more difficult for users to jump between these two modes than it
need be.
[0272] For example, as soon as more than one other party is
involved in an email exchange, there is a reluctance to simply
"pick up the phone" and call the sender--because the other party
will not then have the benefit of hearing your response. Because
they do not see an email corning back they may assume that you have
not dealt with the issue and chase you. So rather than resolving
it--as you know you could since the sender is showing "online" and
is contactable via voice. The easiest thing to do, since one is
interacting with an email application is to send an email to both
parties. This can be costly--not just because the issue is not
resolved to the benefit of the sender until they next pick up their
email but in the meantime, there is a chance the other recipient
calls them--duplicating or even contradicting your response.
[0273] FIG. 14 shows a presentation designed to encourage the use
of the most appropriate medium. The identities of the participants
shown in any message trail or "wrapper" described above are
preferably shown as hot links. These show either directly, through
one or more graphical devices and/or through pop-up information
when selected some or all of:
[0274] a) the mechanisms by which that individual can be
contacted
[0275] b) the current availability of the individual on any or all
of these ("presence")
[0276] c) the individual's preferred communications method
[0277] Alternatively, some or all of the above may be shown on the
display by default--as in FIG. 14 where:
[0278] a) Means of contacting the individual(s) involved in the
individual message are shown (e.g. cell phone (87), "landline"
(88), telephone conference (89), Instant Messaging (90), Audio via
VoIP (91) etc.)
[0279] b) The mechanisms shown are appropriate to the number of
individuals (e.g. telephone conference is only shown where more
than one other participant is involved (89).
[0280] c) The order in which icons are shown may be influenced by
personal and/or corporate preferences. These may take into account
cost/benefit issues e.g. speed of interaction versus cost of
interaction may dictate that web chat is preferred as it is
negligible cost and rapid. Cellular call being more expensive might
be less preferred. "Personal" preferences in this case may include
not only the user of the application but also the recipients (e.g.
who may have indicated that they are abroad and hence incurring
expensive roaming charges on their cell-phone.) Where the other
recipients are also users of this application, such preferences may
be hidden within the text and/or headers of the message previously
sent.
[0281] d) Services that show presence, such as instant messaging
may incorporate this into the icon or other indication used e.g.
(91) may show "Online", "Away" etc. Most such services use their
own set of icons to show these states and, as users will be
familiar with these, the same or similar indications can be used to
good effect here.
[0282] e) Note that many users subscribe to multiple Instant
Messaging services. The application may show each individually or
combine presence information from several to show the most likely
method of reaching that individual. For example, a party that uses
Skype .RTM. and AOL Instant Messenger.RTM. may only be online on
one of the two--so that one would be shown in preference.
[0283] f) Where a message exchange involves multiple parties, not
all may be reachable by all methods. In such cases the application
may rank icons according to the "coverage" (number of participants
that can be reached--optionally giving priority to "To" over "cc"
participants). Those methods providing partial coverage may be
grayed out and/or provide indication (within the icon, on pop-up
text or menu for example) to show how many and/or exactly which
participants can be reached via those mechanisms.
[0284] g) When the user clicks on or otherwise activates one of
these controls, a call is made or message prepared (as appropriate)
for the parties shown in that message exchange. The user can
subsequently deselect these and/or add other parties but the
default participant list is as per the message exchange next to
which the icon is shown.
[0285] h) A similar set of icons is shown at the top of the window
and relate to the thread as a whole hence potentially all
participants in the thread shown. Note icon (92) which allows
creation of a new, blank email message as distinct from the Reply
All icon below it which, as previously discussed, will
automatically construct a comprehensive record of the interactions
to date and use this as the starting point for an email to all
participants. As with (f) above, these communication methods may
not be able to reach all parties and hence may show partial
coverage information. Unlike the icons on individual message
sections below, clicking on one of these icons will initiate
interaction with the full list of participants in the thread
(again, by default with the user then able to add and/or remove
participants).
[0286] Contact with an individual or group may therefore be
attempted via any of the mechanisms shown for example by clicking
on a telephone icon next to that user's name. Alternatively, where
not all options are shown already, clicking on a name may bring up
all possible communication methods allowing the user to select one
of many, preferably ranked according to the user and/or recipient's
preferences. Note that the set of communication methods is not
exhaustive and is intended to allow the addition of others as
required. For example, a video call as discussed later.
[0287] At an intermediate level, since participants are already
grouped by domain (typically corresponding to all those from a
particular company) a further command can instruct the system to
attempt to connect to all of those. For example, when showing the
user the list of addresses with whom communication is to be
initiated, if these are grouped by domain and an additional control
such as a checkbox embedded in a surrounding frame may allow the
selection or de-selection of all addresses within that domain.
[0288] In a preferred embodiment, even when the user opts to
communicate with one or more parties via a real-time mechanism,
that interaction is automatically recorded--for example as an audio
file, a text record or a recording of the user's desktop--and can
be made available to the other parties involved in the interaction.
In an alternative embodiment the user may be prompted to decide
whether or not the interaction should be recorded and/or may start,
stop or pause recording at any time.
[0289] FIG. 14 shows an icon (86) that indicates whether contacts
(by whatever mechanism) will be recorded (if possible) or not. The
user may click to toggle the state of this prior to and/or during a
contact. A further control (166) may be provided to allow the user
to "break" the recording of a real-time interaction. This will
split the recording at the point the control is used--with a
recording prior to it and a new one after it. This allows the user
to make many, focused and relevant recordings within one
interaction which may cover several topics, not all of which need
to be recorded or which need to be forwarded to different
people.
[0290] If the user of the invention places a telephone call but the
recipient is not available or chooses not to answer the call, the
call may be diverted to a voicemail system. However, some phones
are not provided with voicemail; some users choose not to turn it
on and even with those that do, leaving a message on the
recipient's voicemail system will result in the message being left
for only that one recipient. Hence, whether the call diverts to
voicemail or not, the user of the invention may choose--for
example, by clicking on a button on the user interface of the
application--to record a message using the invention rather than
leave a message on the recipient's voicemail. Should they choose
this option, the telephone call may be terminated and their speech
is recorded from their headset/handset/microphone. Optionally they
may choose to leave a voicemail and have a local recording made of
them doing so.
[0291] The user may indicate that they have completed recording
their message--for example, by clicking on another control on the
application's user interface or in the case of leaving a voicemail
whilst recording a local copy, by hanging up the telephone
connection. One such control will preferably attach the file
containing the voice recording (or a link to it if it can be
accessed remotely) to an email ready to be sent to the person that
the abortive telephone call was placed to. The other participants
in the thread may then be copied the recording and can choose to
listen to it at their convenience.
[0292] Preferably, the progress of any recording--whether a live
telephone call, multi-way conference call or leaving a message--is
shown graphically as it is recorded--for example as an audio
waveform. FIG. 15 shows the user interface during a telephone call
that has been initiated as part of an ongoing thread which already
contains three email messages.
[0293] It is common practice for an audio file to be shown in this
manner (9S) but in a preferred embodiment of the invention, the
horizontal axis represents time at a fixed scale. This scale can be
set according to user preferences but is preferably set so as to
allow the user to see the gaps between sentences as sufficiently
wide areas in which to click with accuracy--so as to start playing
from the start of the following utterance.
[0294] As a recording progresses, the audio waveform extends to the
right within the area allocated to it. Should the duration exceed
that which can be shown within the horizontal area available, it
may (according to user/system preference) "wrap" to a new
horizontal line below the first one. This makes audio appear more
like text. The user can see at a glance how long it is and can see
long pauses. It also facilitates annotation of the recording with
the full or partial textual transcript should LVSR or keyword
spotting (respectively) have been performed on it. In FIG. 15
manual annotation is shown (104) and is indicated as such by icon
(103).
[0295] Optionally, a long pause (defined, for example, as a period
of time greater than a pre-set threshold within which the
instantaneous, cumulative or time-averaged audio amplitude or
energy does not exceed a threshold) will cause the audio display to
wrap to another line providing some visual feedback similar to that
achieved by paragraph breaks in text.
[0296] During the recording, the user may annotate the recording
with actions such as clicks, text entries (104) or selecting from a
number of pre-defined options (105). This may be a deliberate
action on the part of the user--clicking on an icon within the
application (105). Alternatively, such annotations may be the
result of integration with other applications. For example, on
bringing up a customer credit application form on screen, the audio
recording may be marked to that effect. This can be achieved by
integration with the application responsible for displaying said
form or by "screen-scrape" techniques whereby the presence of a
pre-defined text string or graphical pattern on screen can be used
to infer the presence of said form.
[0297] Initially such markers or typed entries will be
automatically time-stamped and hence can be related to that instant
of the communication. However, in a preferred embodiment, the user
may adjust the timestamp--for example by dragging the on-screen
marker (107) along the time-axis or by stretching a marker (105)
(e.g. via drag and drop) so as to cover a period of time rather
than a point in time. Certain markers (such as the "Smiley" in FIG.
15) may be automatically defined as covering a period rather than a
point in time. Such controls may show as two-state buttons
appearing depressed or activated when clicked the first time and
returning to their original state on a subsequent click. When
activated a line (108) may be seen moving with the newly recorded
audio until the control is clicked again. Multiple such lines may
be active at the same time within a call.
[0298] In addition to the predefined markers (105) the user may
simply click within the audio waveform. Preferably this results in
numbered markers (106) which the user can subsequently delete, move
via drag and drop, add comments to etc.
[0299] These markers will be stored or otherwise associated with
the recording and can subsequently be used to select, edit and/or
highlight sections of the recording. These sections can then be
attached to email messages allowing the user of the invention to
pass on selected, relevant sections of the recording as an
alternative to the entire recording. This allows the user to remove
extraneous or irrelevant content and hence make the review process
more efficient for those to whom the attachment is sent.
[0300] Also note on FIG. 15:
[0301] a) While the call is in progress, the application provides
shortcuts to appropriate features e.g. for a telephony call as
shown here, Transfer (100), Hold (101) and Release (102) functions
are immediately available. More advanced features such as those
available on softphone implementations may also be provided.
[0302] b) To add other parties into the call, the user may click on
icons (97) that show appropriate mechanisms by which others can
join this interaction. In this case, a phone call may be
conferenced with a Skype.RTM. audio call and/or other telephone
calls.
[0303] c) The user may choose, during or after the call, to delete
the audio recording--leaving just the text annotations (if any) and
the basic information about the call (participants, time, duration
etc.). This can be done using control (96).
[0304] d) Control (93) which allows the user to compress the entire
display for previous messages in this thread to a summary form.
[0305] e) Indication (116) that the interaction was two way. This
will be shown on subsequently viewing this thread as well as during
the call.
[0306] f) Indication (99) of the method used and, from its
background color in this case, the fact that the interaction is
still active. Said indication may be included in subsequently
forwarded messages so that other participants in the thread are
aware of this information.
[0307] g) Recording control (94) is effectively tri-state. Here it
is showing not only that recording is enabled (as it did in FIG.
14) but that it is actually recording now. Clicking this control at
the point shown in this Figure would stop recording without hanging
up the call.
[0308] On receipt of an attachment that is identifiable as an audio
and/or video recording, the invention will display this in a
similar fashion--giving the user an instant appreciation of the
duration of the call and an ability to start playing it from
anywhere within the recording. This is more efficient than having
to recognize the type of attachment, select and open it and then
have a separate media player obscure the body of the email.
[0309] In a preferred embodiment of the invention, the recording is
made with separate channels or "tracks" for the user's voice and
the other parties on the call (a "stereo" recording).
[0310] In a further preferred refinement of the invention, the
recording is made with separate channels or "tracks" for each
participant in a multi-way call.
[0311] By recording the audio in separate tracks it is easier for
analysis tools such as a large vocabulary speech recognizer (LVSR)
or a keyword spotter to interpret the audio and produce a whole or
partial transcript of the recording. Preferably the transcript may
be shown alongside the recording. This may be used by the user
making the recording to select portions of the recording for onward
transmission to other parties. It may also be sent with the audio
recording to help the recipients to identify the sections of
interest to them--by identifying text and the time offset at which
it occurs as an alternative to having to listen to the entire audio
recording.
[0312] The display of the audio may be further enhanced in cases
where different participants can be identified. For example, the
audio waveform display may wrap to a new horizontal line on the
display whenever the active speaker changes. This can be
determined, for example, by determining which of the tracks
contains the most audio energy within a sliding time window.
Alternatively, many large vocabulary speech recognition tools first
determine the identity of the speaker and a change in this can be
used to move to the next "line".
[0313] Similar "line-wraps" may occur when text is entered (e.g.
with carriage return or enter key as used in instant messaging to
complete and send a message). The system will preferably break the
horizontal run of audio recording, insert a line onto which the
text is typed and continue the current audio beneath that point.
The user can then continue to type into the text area associated
with the time they started this annotation or click ahead of the
growing audio waveform to move their next annotation point back to
current time.
[0314] FIG. 16 shows a telephone call that has progressed for a
longer period. Notes were entered at the start but during the time
period shown by the second line of audio (114) no additional notes
were made. Hence the next wrap (111) was to immediately below
(114). Further notes have then been made (113). Note how the period
covered by the "smiley" icons (110) extends over two tracks.
[0315] Also note on FIG. 16 that control (93) of FIG. 15 has been
activated so the previous messages show as a single line summary.
Control (109) would expand these again if needed.
[0316] The scissors control (95) on FIG. 15 allows the user to
manually split the recording of an active call. Had the user done
so during the latter half of the phone call shown in FIG. 16, a
display such as that of FIG. 17 would be shown. In this figure:
[0317] a) The user has clicked on the scissors control (118) at the
point where he started to discuss Jane's performance. Rather than
using the "smiley" annotations within the recording, he has chosen
to split the recording at this point in order to send this portion
to Jane (in this case using "Forward" via email icon once the call
is over. This only appears once a recording is complete-e.g. (117)
and hence is not yet shown for the lower message. The split is
indicated by a separator (116) and the creation of a new message
(above or below the previous one according to user preference).
[0318] b) Control (132) shows that the top message is now no longer
active but still indicates the method of communication (called
Tim's cellphone).
[0319] c) The now complete telephone call, being a real-time
interaction with a duration shows a time period (119) unlike an
email which has a "point" time of arrival. Alternative displays
such as "start time and duration" are possible.
[0320] d) The current call may show start time and no end time or
duration (120) or may show current time and/or duration and be
updated as this progresses.
[0321] The above approach may be used for voice
communications--regardless of whether these occur via traditional
telephony, VoIP or other- services.
Instant Messaging
[0322] As with live and attempted live telephone or video calls, so
similar features apply to instant messenger or "chat" applications.
Should the user of the invention decide to communicate with one or
more or the participants on the thread using a web-chat application
such as Instant Messenger.RTM. they may find that one or more of
the recipients--even though they are showing as "online" or
"available"--does not actually respond. In such a case, the effort
of typing one or more messages into the chat application is wasted
as these must then be copied into an email to ensure transmission
to all parties.
[0323] A similar problem occurs when only some of the participants
subscribe to the messaging service chosen. An email may need to be
sent to the other participants to let them see what has been agreed
between those who did speak in "real-time". Many such systems--for
example SKYPE.RTM.--provide a readily available interface that
allows developers to layer such applications onto the underlying
mechanism.
[0324] The invention therefore optionally includes integration to
one or more instant messaging mechanisms. This allows a real-time
discussion to be undertaken via an instant messaging mechanism
while at the same time building a textual, time-stamped record of
the interaction. This may be sent as the body of an email during or
after the discussion.
[0325] When recording interactions made over such "instant"
messaging services, the exchanges are actually not "instant" but
many are nearly so. There is a benefit in grouping exchanges
"paragraphs" in a similar way to that described for audio
conversations. By setting one or more threshold intervals (e.g. 5
minutes and 1 hour) the exchanges can be summarized more
efficiently than is typically achieved by simply cutting and
pasting from a chat window. In the latter case, every exchange is
often time-stamped but in many cases just knowing the start time of
the group of exchanges is adequate and makes it easier to read the
record of what was exchanged.
[0326] FIG. 18 shows a very similar interaction to that of FIG. 17
but this time using instant messaging:
[0327] a) The recording (121) and "chop" (128) controls operate as
they did for telephony calls controlling and showing whether or not
recording is enabled/disabled/taking place and allowing the
recording or "timestamped and annotated transcript" as it is in
this case to be split into multiple messages as the interaction
progresses.
[0328] b) The messages exchanged are shown (122) and new messages
can be typed into this window or a separate text control.
[0329] c) Interaction is occurring within this application rather
than the window of the instant messaging system. The latter may be
used--in which case this application will be recording the
interaction as it progresses rather than showing it.
[0330] d) Timestamps (126) may be optional. It is not always
important to see how rapidly someone responded.
[0331] e) Names (or abbreviations thereof) may be optional. Simply
coloring the text consistently for "sent" and "received" may be
adequate--especially in a two-way interaction. The color keying may
be shown by coloring the addresses in the list above the text area
the same way as the text is being colored. This is especially
powerful for multi-way conferences.
[0332] f) To save space and for consistency with the other use of
addresses, the names shown (127) may not be those known to the
particular instant messaging service being used. Instead these are
consistent with the abbreviations defined by the application and
shown in address lists.
[0333] g) The type of interaction and whether or not it is still
active is shown as before (125).
[0334] h) One or more controls that would normally be present in
the IM client are provided for ease of use (124).
[0335] i) Additional participants may be added to the interaction
using any means appropriate (123). This may include other types of
IM or even cellular text messaging (SMS).
Cellular Text Messaging
[0336] Similar issues occur with text messaging (e.g. Short Message
Service or "SMS") via mobile phones. If this method is chosen to
advise some participants of an update to the discussion, the other
participants will probably want to receive an email with the same
or similar content. Hence the invention can also copy the text
messages sent to an email.
[0337] Optionally, the invention may, by default, select the
remainder of the participants--those who did not participate in the
real-time or preferred method of communication--as the recipients
for the corresponding email.
Incoming Contacts
[0338] The above examples have shown how an existing message thread
is extended by the user of the invention's initiating a further
interaction. A similar approach is taken if the user of the
invention takes an incoming call such as a telephone call.
Regardless of whether it is associated with an existing message
thread--this too will be recorded by default (according to user
preference).
[0339] An example is shown in FIG. 19. Note:
[0340] a) The subject may be initially blank. The user can type the
name of a new thread (130) or select an existing thread using
drop-down list (167) once he understands the context of the call.
Preferably, the list of threads will show active threads but the
user may choose to select (and hence re-activate) a previously
closed thread.
[0341] b) Preferably, integration with the communications system
allows the calling party to be identified. If this is only as a
telephone number, then a reverse lookup into one or more
directories may be used to determine their name which will be shown
(131). If a name cannot be determined or is wrong or ambiguous, the
user may enter a name and/or select one from an existing list.
[0342] c) The same means of marking the recording as it progresses
apply.
[0343] d) Appropriate controls (129) are shown to allow the user to
perform functions during the interaction.
[0344] e) During or after the call completes, the user can choose
to send all or edited sections of the recording to one or more
recipients.
Visual Interactions
[0345] It will be appreciated by one of skill in the art that video
calls be handled in a similar fashion. FIG. 20 shows an exemplary
presentation of such a call:
[0346] a) The type of interaction is indicated (136) and may be
shown as part of the summary view of the interaction.
[0347] b) Instead of or as well as showing the level of audio one
may show the amount of video activity over time (137). This can be
calculated, for example, using motion detection algorithms or
others that compare successive frames or simply from the bandwidth
used where a variable bit rate signal is transmitted.
[0348] c) In addition to inferring "paragraph breaks" from pauses
in the audio, a lack of screen activity or, conversely a
significant change as occurs on scene boundaries can be used to
identify significant points within a video stream. These may cause
"line-wraps" as with the telephone call example earlier.
[0349] d) In addition to showing the level of video activity, small
"thumbnail" views (134) of the video may be shown--at scene change
boundaries and/or regular intervals to give the viewer a
storyboard" overview of the recorded video. This helps them to know
which section(s) of the video they wish to view.
[0350] e) As with the telephone call, the user can annotate the
recording by clicking on it during the recording.
[0351] f) As with the telephone call, the user can replay a
completed recording from any point within it by clicking on the
waveform(s) versus time display(s) e.g. (137) at the point they
wish to play from.
[0352] g) Audio and video recordings may be deleted independently
of each other using the control (135), (133) next to the
appropriate waveform display.
[0353] It will be appreciated that other collaborative
communications such as shared web-browsing, desktop sharing, remote
PC access, electronic white-boarding and the like can be handled in
the substantially the same way as video calls though in some cases
additional information such as the text being displayed can be
extracted from the streams used to record the interactions. This
provides further indexing and search capabilities similar to those
discussed in the case of LVSR transcriptions of audio calls.
Further Integrations
[0354] This specification has concentrated on the presentation,
control and recording of interactions. As with any email
application, the invention may be further enhanced by integrating
it with other applications. These include but are not limited
to:
[0355] a) Calendars and scheduling tools
[0356] b) Task List/To Do list management
[0357] c) Transcription tools (from word-processors to court
reporting tools)
[0358] d) Customer Relationship Management (CRM) systems
[0359] e) Call Centre applications e.g. Quality Monitoring, Call
Flow design, Helpdesk applications etc.
[0360] f) Enterprise Resource Planning (ERP) systems Integration
may be achieved in a variety of ways. For example, including menu
items, toolbar buttons or other controls within the presentation
formats shown in this specification or conversely by embedding
these message handling tools within the larger application.
System Design
[0361] One of skill in the art will appreciate that the invention
may be realised through software, hardware or a combination of
these. It may be implemented on a single computer or the
functionality distributed across multiple computers that need not
be co-located. Standalone application, client server or web and
browser based implementations are all feasible.
[0362] One should also note that the specification describes a wide
range of options and possibilities. As is standard practice in such
applications, the preferred settings for these may be set both at
an overall (shipping default), a corporate (e.g. set at install
time) and personal (e.g. change via a "Tools>Options" menu or
similar) level.
[0363] It will be appreciated by one of skill in the art that
although the description of the invention concentrates on the
on-screen display of information, many of these techniques are also
advantageous when printing, copying or pasting the message for use
in other applications. Likewise, the same techniques are useful
when a permanent record of such an exchange is stored in an
archival system or as part of a larger system such as a Customer
Relationship Management (CRM) system. By reducing both the data
size and the physical presentation area needed yet maintaining the
information content, it is both easier to store and for a reader to
grasp the meaning of the information.
[0364] Audio recording can be achieved in many ways. The user's
telephone handset/headset can be connected via appropriate
circuitry to the line in and/or microphone connections on the
computer. Where an IP telephony system such as Avaya.RTM.
Communication Manager is available, a "softphone" can be
instantiated on the PC using Avaya's Distributed Media and Call
Control (DMCC) Application Programming Interface to receive a copy
of the audio stream which can then be stored to a file on disk. A
separate recording system may be available and this application may
instruct it to record and/or have access to recordings that may be
made automatically by it.
[0365] FIG. 21 shows an exemplary system design, in which the
invention is instantiated on one or more Application Server(s) 143.
As with each of the elements of this diagram, multiple independent
heterogeneous examples of each component may exist; each may be
distributed across the network and/or partitioned for scalability.
The system is further described below:
[0366] a) Optionally, the application server (143) is configured
with access details for one or more directories and/or address
books from which it can expand upon the addresses (email addresses,
instant message "handles", telephone numbers etc.) found within the
messages it is to process. Such directories may exist within the
enterprise (144) and/or (147) on the internet (148) (e.g.
Skype.RTM. directory services). These often conform to standards
such as the Lightweight Directory Access Protocol (LDAP) or
similar--allowing easy interrogation by the application server.
Additional directories (or "address books") may be found on the
client PCs (142) and/or Mail Servers (146). The application will
also interrogate these where it is made aware of them and is
granted access--so as to build as complete a picture as possible of
the individuals who are communicating.
[0367] b) The application server will normally have access to at
least one Mail Server (146)--though in an alternative
implementation it may encompass Mail Server functions within the
application itself. Mail Servers may be located on the company's
network (145) or on the Internet (148) e.g. hotmail. It is not
unusual for a system to require links to many mail servers,
particularly in a large enterprise.
[0368] c) Links to Instant Messaging (IM) services (150), whether
corporate or public (latter shown in FIG. 21) typically provide
presence services as well as directory and connection establishment
services within them. The Application server may establish logons
with the IM server on behalf of one or more users or these may be
established directly on their computers (142) with links back to
the Application server advising it of status, presence etc.
[0369] d) Note the Firewall between internet (148) and Corporate
network (145). In addition to blocking some traffic between these,
there is often Network Address Translation (NAT) occurring at this
point where the Application Server is supporting users on the
internet as well as within the firewall, it may be necessary for
the remote components of the application to pass back addressing
information allowing the mapping of streams across this boundary.
Where remote clients are using a VPN to access the corporate
network, however, there is no need to translate as the VPN makes
them appear to be inside the corporate network.
[0370] e) Telephony integration is typically achieved by connection
to one or more internal telephone switches (or Private Automatic
Branch Exchange, PABX) (139). Such switches normally provide
Computer Telephony Integration (CTI) services which are often
connected via a dedicated CTI server (140). Regardless of this
topology, the application may determine activity on the switch by
using the published CTI mechanism(s). On many switches it is also
possible to emulate a telephone and/or observe a telephone using a
"softphone". This may be instantiated at the user's computer (142)
or on the server (143). Increasingly, such interfaces allow not
just visibility and control over telephony functions but also
provide access to the media streams (the voice). The user may have
a real telephone (analogue, digital or IP) (141) and/or telephone
capabilities (a "softphone") on his computer (142). Most telephony
systems also support a Voicemail capability--whether on a separate
server (213) as shown or embedded into the switch (139). Standards
exist for integration with Voicemail systems--either through the
switches main CTI interface(s) or a dedicated interface to the
voicemail itself. In this way the application server can leave
messages, retrieve messages, determine whether a user has messages
waiting etc. It will be appreciated that the same or better degree
of integration is possible in "soft" IP telephone systems (e.g.
those using Session Initiation Protocol (SIP)) as in the more
traditional topology drawn in this example.
[0371] f) Recording of calls and messages can be performed in a
variety of ways. The aforementioned Voicemail (213), Mail servers
(146) and Instant Messaging services (150) typically include some
degree of recording and/or archival. Increasingly, telephone
systems (139) also include recording features. Alternatively, a
dedicated recording system (150) may be connected. Such systems are
well known and may be recording voice, data, email, screen content,
video etc. It may be tapping into communication paths unobtrusively
and/or using the switching capabilities of the other components
discussed to join the interactions as a (usually silent)
participant. The application (143) may interface to the recording
system (150) to control its activities and/or make use of
recordings it has made or is making. Alternatively, some or all
recording functions may be performed within the application itself
by implementing aspects of the recording system. Likewise some
recording may be undertaken at the user's desktop by applications
running on the user's computer or additional hardware located
nearby.
[0372] g) The user's computer (142) will typically access the
application through a browser or a client application. In both
cases communication with the Application Server (143) is
established. Interaction with the other components in the diagram
may be direct or via the application server (143). The former
allows better scalability and minimizes round-trip delays.
Optionally the user's computer will have sound input and/or output
devices to allow audio communication directly from it. Otherwise a
separate audio device such as a telephone (141) or VoIP phone will
be required for audio calls. Ideally, the computer (142) also has
video capture capabilities allowing video calls. to be placed. Note
that this is not necessary for one way video calls. The incoming
video can be displayed on screen even if the user has no means of
capturing their own image to send.
[0373] h) By designing the system to accommodate an arbitrary
number of communication mechanisms, each of which is either
real-time or not; supports audio or not; supports video or not etc.
an extensible system can be deployed such that the addition of a
new communications mechanism is a relatively straightforward
exercise.
* * * * *