U.S. patent application number 17/474467 was filed with the patent office on 2022-03-17 for system and method for establishing and managing multiple call sessions from a centralized control interface.
The applicant listed for this patent is Damaka, Inc.. Invention is credited to Sivakumar Chaturvedi, Rashmi Lohita.
Application Number | 20220086197 17/474467 |
Document ID | / |
Family ID | |
Filed Date | 2022-03-17 |
United States Patent
Application |
20220086197 |
Kind Code |
A1 |
Lohita; Rashmi ; et
al. |
March 17, 2022 |
SYSTEM AND METHOD FOR ESTABLISHING AND MANAGING MULTIPLE CALL
SESSIONS FROM A CENTRALIZED CONTROL INTERFACE
Abstract
Disclosed are a system and method for establishing and managing
one-to-one and conference call sessions through a virtual waiting
room. Conference calls may be established initially or created as
additional people are invited to an existing call. Functions such
as screensharing, chat messaging, and file sharing may be provided.
Media, including video, text, and images, may be selected and sent
to participants while they are on hold or during an active call
session.
Inventors: |
Lohita; Rashmi; (Allen,
TX) ; Chaturvedi; Sivakumar; (Allen, TX) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Damaka, Inc. |
Richardson |
TX |
US |
|
|
Appl. No.: |
17/474467 |
Filed: |
September 14, 2021 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
63077892 |
Sep 14, 2020 |
|
|
|
63176419 |
Apr 19, 2021 |
|
|
|
International
Class: |
H04L 29/06 20060101
H04L029/06 |
Claims
1. A method for managing multiple independent communication
sessions between a control device and at least first and second
client devices, the method comprising: establishing, between a
control device and a first client device, a first communication
session; establishing, between the control device and a second
client device, a second communication session; participating, by
the control device, in the first communication session while the
second communication session is on hold; placing, by the control
device, the first communication session on hold; injecting, by the
control device, a first visual media file into the first
communication session, wherein the first visual media file is to be
displayed on the first client device while the first communication
session is on hold; and switching, by the control device, to the
second communication session while the first communication session
is on hold.
2. The method of claim 1 wherein injecting the first visual media
file into the first communication session further comprises:
receiving, by the control device, a selection of the first visual
media file from a plurality of available visual media files; and
sending the first visual media file to the first client device,
wherein the entire first visual media file is received before being
displayed.
3. The method of claim 1 wherein injecting the first visual media
file into the first communication session further comprises:
receiving, by the control device, a selection of the first visual
media file from a plurality of available visual media files; and
streaming the first visual media file to the first client device
while the first communication session is on hold.
4. The method of claim 1 wherein injecting the first visual media
file into the first communication session further comprises:
receiving, by the control device, a uniform resource locator (URL),
wherein the URL represents a location of the first visual media
file on a server accessible to the first client device; and sending
the URL to the first client device.
5. The method of claim 1 wherein the first and second client
devices are invited to communicate with the control device via an
identical uniform resource locator (URL).
6. The method of claim 1 further comprising: receiving, by the
control device, a request for a third communication session from a
third client device; and adding, by the control device, the third
client device to a virtual waiting room, wherein the third
communication session is automatically placed on hold when added to
the virtual waiting room.
7. The method of claim 1 wherein the first client device is able to
participate in the first communication session using only a
standard browser present on the first client device, wherein no
plug-in or other application download is required to participate in
the first communication session.
8. A method for injecting a visual media file into a communication
session between a control device and a client device, the method
comprising: establishing, between a control device and a client
device, a communication session; placing, by the control device,
the communication session on hold; injecting, by the control
device, a visual media file into the communication session, wherein
the visual media file is to be displayed on the client device while
the communication session is on hold; and removing, by the control
device, the hold to continue the communication session.
9. The method of claim 8 wherein injecting the visual media file
into the communication session further comprises: receiving, by the
control device, a selection of the visual media file from a
plurality of available media files; and sending the visual media
file to the client device, wherein the entire visual media file is
received before being displayed.
10. The method of claim 8 wherein injecting the visual media file
into the communication session further comprises: receiving, by the
control device, a selection of the visual media file from a
plurality of available media files; and streaming the visual media
file to the client device while the communication session is on
hold.
11. The method of claim 8 wherein injecting the visual media file
into the communication session further comprises: receiving, by the
control device, a uniform resource locator (URL), wherein the URL
represents a location of the visual media file on a server
accessible to the client device; and sending the URL to the client
device.
12. The method of claim 8 further comprising injecting, by the
control device, a second visual media file into the communication
session while the communication session is not on hold, wherein the
second visual media file is to be played on the client device while
the communication session is not on hold.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit of U.S. Provisional
Application Ser. No. 63/077,892, filed on Sep. 14, 2020, and
entitled SYSTEM AND METHOD FOR ESTABLISHING AND MANAGING MULTIPLE
INDEPENDENT CALL SESSIONS FROM A CENTRALIZED CONTROL INTERFACE,
which is incorporated by reference herein in its entirety.
[0002] This application claims the benefit of U.S. Provisional
Application Ser. No. 63/176,419, filed on Apr. 19, 2021, and
entitled SYSTEM AND METHOD FOR HIGHLY SCALABLE BROWSER-BASED
AUDIO/VIDEO CONFERENCING, which is incorporated by reference herein
in its entirety.
BACKGROUND
[0003] The manner in which communication sessions with remote
parties occur is currently limited in functionality and
flexibility. Accordingly, what is needed are a system and method
that addresses these issues.
BRIEF DESCRIPTION OF THE DRAWINGS
[0004] For a more complete understanding, reference is now made to
the following description taken in conjunction with the
accompanying Drawings in which:
[0005] FIGS. 1A and 1B illustrate embodiments of environments
within which a control device may manage separate communication
sessions with one or more client devices;
[0006] FIG. 1C illustrates one embodiment of a virtual waiting room
and call system within the environment of FIG. 1A;
[0007] FIGS. 1D-1H illustrate embodiments of different arrangements
of the components of the virtual waiting room and call system of
FIG. 1C;
[0008] FIG. 2-5 illustrate embodiments of a graphical user
interface (GUI) that may be used by the control device of FIGS. 1A
and 1B;
[0009] FIG. 6A illustrates one embodiment of a GUI by which call
requests may be assigned to a virtual waiting room;
[0010] FIG. 6B is a flow chart illustrating an embodiment of a
process that may be executed by a control device;
[0011] FIG. 6C is a flow chart illustrating an embodiment of a
process that may be executed by a client device;
[0012] FIGS. 7A-7E illustrate processes by which media may be
injected into a session, either directly or via a URL, to be played
when the session is active or on hold;
[0013] FIGS. 8-10 are flow charts illustrating embodiments of
processes that may be executed by a control device to provide media
for display on a client device;
[0014] FIG. 11 is a flow chart illustrating an embodiment of a
process that may be executed by a client device;
[0015] FIGS. 12A and 12B illustrate embodiments of a medical
facility environment within which a control device may manage
communication sessions with multiple client devices;
[0016] FIGS. 12C and 12D illustrate embodiments of FIGS. 1A-1C,
12A, and 12B that use one or more servers for conference calls;
[0017] FIGS. 13A-17F are embodiments of sequence diagrams
illustrating various processes that may be executed within the
environments of FIGS. 1A, 1B, and 12A-12F;
[0018] FIGS. 18-26 illustrate embodiments of a GUI that may be used
by a control device;
[0019] FIGS. 27-29 illustrate embodiments of a GUI that may be used
by a client device; and
[0020] FIG. 30 is a simplified diagram of one embodiment of a
computer system that may be used in embodiments of the present
disclosure as a control device, a client device, and/or a
server.
DETAILED DESCRIPTION
[0021] It is understood that the following disclosure provides many
different embodiments or examples. Specific examples of components
and arrangements are described below to simplify the present
disclosure. These are, of course, merely examples and are not
intended to be limiting. In addition, the present disclosure may
repeat reference numerals and/or letters in the various examples.
This repetition is for the purpose of simplicity and clarity and
does not in itself dictate a relationship between the various
embodiments and/or configurations discussed.
[0022] Referring to FIGS. 1A and 1B, embodiments of environments
are illustrated within which various aspects of the present
disclosure may practiced. Referring specifically to FIG. 1A, an
environment 100 includes a control device 102 that has established
communication sessions with a client device 104, a client device
106, and a client device 108. The control device 102 is coupled to
the client device 104 via a session 110, to the client device 106
via a session 112, and to the client device 108 via a session 114.
Although only three client devices are illustrated, it is
understood that any number of client devices may be in
communication with the control device 102, subject to technical
limitations such as bandwidth, processing power, and similar
factors.
[0023] FIG. 1B illustrates an environment 116 in which the control
device 102 has established only the communication session 110 with
the client device 104. It is understood that the environments 100
and 116 may be viewed as the same environment in different states.
For example, the control device 102 of FIG. 1A may drop two
sessions to reach the single session of FIG. 1B. Similarly, the
control device 102 of FIG. 1B may add two sessions to reach the
three sessions of FIG. 1A.
[0024] The control device 102 and client devices 104, 106, and 108
may be mobile devices (e.g., tablets, smartphones, personal digital
assistants (PDAs), or netbooks), laptops, desktops, workstations,
smart televisions, and/or any other computing device capable of
receiving and sending electronic communications via a wired or
wireless network connection. Such communications may be direct
(e.g., via a peer-to-peer network, an ad hoc network, or using a
direct connection), indirect, such as through a server or other
proxy (e.g., in a client-server model), or may use a combination of
direct and indirect communications.
[0025] While multiple sessions may be combined into a single
session (e.g., as a conference call or a whiteboard sharing
session), it may be desirable in some scenarios described herein to
maintain the sessions 110, 112, and 114 as independent sessions.
Accordingly, with independent sessions, only the control device 102
has access to the session established with each of the client
devices 104, 106, and 108, and the client devices do not have
access to the sessions of the other client devices. In other words,
neither the client device 106 nor the client device 108 may access
the session 110 unless the control device 102 specifically merges
the sessions, and such functionality may or may not be available to
the control device depending on the particular configuration of the
control device and/or client devices. For example, such
functionality may be disabled to prevent a user of the control
device 102 from inadvertently merging sessions. This may be of
particular value in environments where privacy requirements are
high and possibly regulated by law, such as telemedicine and
counseling environments.
[0026] In other embodiments, a client device may be provided the
needed functionality to invite another device to a session. In such
embodiments, the other device may not be part of an existing
session, but would be invited to participate in the session. Each
session may be end-to-end encrypted for both signaling and media.
This enables sessions to be channeled through third-party servers
without compromising private information.
[0027] A session may be accessed by the control device 102 and
client devices in various ways. For example, a session may be
accessed in a web based manner via a web browser (e.g., based
solely on a browser's native capabilities or using scripting, an
applet, and/or other browser related functionality), or may be
accessed in an application based manner using a downloaded
application that is installed and/or used to execute a temporary
file. It is understood that the control device and client device
may access a session in different ways, such as an installed
application for the control device and browser access for the
client device or vice versa.
[0028] The terms "control device" and "client device" as used
herein do not in themselves dictate a server/client or other
particular relationship between the control and client devices. As
will be seen in following examples, the terms are generally used in
a relational manner to indicate that the control device is
controlling one or more sessions via a control interface, and each
client device is communicating with the control device via one of
those sessions. In some embodiments, the control device 102 may
have access to functionality not available to the client device,
such as the ability to inject videos and/or other media into the
session when the session is ongoing or placed on hold. In addition,
it is understood that each of the control device and client devices
are generally used for interactions with a user, and the user
information may be used or displayed in any given example herein.
For example, the control device 102 may be identified as such, or
may be identified by a user name or other designation.
[0029] For purposes of example, each of the communication sessions
110, 112, and 114 is described with respect to a hybrid
peer-to-peer model, but the present disclosure is not limited to
such models. It is understood that each of the sessions 110, 112,
and 114 may include multiple connections or channels for different
functions, such as a connection/channel for an audio/visual call
and another connection/channel for document sharing. Alternatively,
a single connection/channel may be used to provide multiple
functions. Examples of hybrid peer-to-peer communications and
functionality that may be used herein are disclosed in U.S. Pat.
Nos. 7,656,870; 7,623,516; 7,623,476; 7,933,260; 8,050,272;
8,218,444; 8,446,900; 8,694,587; 8,892,646; 9,027,032; and
9,356,997.
[0030] From a technical perspective, each device may act as a
client, a server, and/or as something else depending on the
underlying technological framework used to establish and maintain
the communication sessions. For example, in a hybrid peer-to-peer
framework, both the control device 102 and a client device may take
the roles of a server and a client, depending on which way
information is being transferred at a given time. In a traditional
client-server framework, both the control device 102 and the client
device may be clients interacting with a server that is separate
from either device. Accordingly, it is understood that the
functionality described herein may be translated to various
technological frameworks, with appropriate method steps and message
flows that correspond to the particular framework in which the
present disclosure is implemented.
[0031] Referring to FIG. 1C, one embodiment of a communications
framework 120 is illustrated with a virtual waiting room (VWR)
management interface 122. The VWR 122 may be used to manage session
resources 124a-124c that handle the actual communications with the
client devices 104, 106, and 108, respectively. For example, the
session resources 124a-124c may be used for the allocation of
memory, communication sockets, data prioritization, and/or other
resources that are used to send and receive data. The VWR
management interface 122 may provide a visual view of the end
result (e.g., the availability and/or existence of multiple
sessions) and provide management components such as interactive
elements (e.g., buttons) for placing calls on hold, providing users
with uniform communication and collaboration (UCC) functionality,
and similar features. In some embodiments, the VWR management
interface 122 may be further divided to provide a backend interface
that may be used to send invitations to the VWR, schedule calls,
and perform similar functions, and a frontend interface that may
then access the calls waiting in the VWR. The division and
organization of such functionality may differ based on the
particular implementation.
[0032] Referring to FIGS. 1D-1G, embodiments of different
arrangements of the communications framework 120 of FIG. 1C are
illustrated. In FIG. 1D, the communications framework 120 is
located on the control device 102. In FIG. 1E, the communications
framework 120 is located on one or more servers 126 and accessed by
the control device 102. In FIG. 1F, the VRM management interface
122 is located on the control device 102, and the session resources
124a-124c are located on the server 126. In FIG. 1G, the VRM
management interface 122 is divided into a frontend portion 122a
and a backend portion 122b. The frontend portion 122a is located on
the control device 102, and the backend portion 122b and session
resources 124a-124c are located on the server 126. In FIG. 1H, the
VRM management interface 122 is divided into a frontend portion
122a and a backend portion 122b, both of which are located on the
server 126 along with the session resources 124a-124c. In some
embodiments, the control device 102 may also be directly coupled to
the backend portion 122b, as shown in FIG. 1H. The server 126 may
represent one or more virtual and/or physical servers in FIGS.
1E-1H. It is understood that many different arrangements of
functionality are possible, and FIGS. 1C-1H are meant for purposes
of example only and are not intended to be limiting.
[0033] The communications framework 120 may enable all of the
client devices 104, 106, and 108 to request access to the control
device 102 via a single, identical uniform resource locator (URL)
or other uniform resource identifier (URI). Because the
communications framework 120 enables sessions to be established and
managed together, it can identify different session requests and
set each session up independently even though all requests are
received via the same URL. This means that either static or dynamic
URL allocation may be used, and a single URL can be send to
multiple invitees for independent sessions. It is understood that
this is not required and that unique URLs may be used if
desired.
[0034] Referring to FIG. 2, one embodiment of a graphical user
interface (GUI) 200 is illustrated for the control device 102 of
FIGS. 1A and 1B. The GUI 200 may represent, for example, the VRM
management interface 122 of FIGS. 1C and 1F, and the frontend VRM
management interface 122a of FIGS. 1G and 1H.
[0035] In the present example, the GUI 200 may be used to control
multiple communication sessions, such as the sessions 110, 112, and
114. Individual sessions may be added or dropped without affecting
the status of the other sessions. Depending on the embodiment, a
limited number of session slots may be available, or sessions may
be added until constrained by technological or other factors (e.g.,
memory, bandwidth, or available time slots). The GUI 200 may be
web-based and accessible by control device 102 via a web browser,
or may be application based and accessible by the control device
using a downloaded application that is either installed or used to
execute a temporary file.
[0036] In the present example, the sessions are with individuals
using the client devices 104, 106, and 108. The sessions are
separate and remain so unless merged by the control device 102. The
client devices 104, 106, and 108 may not have the ability to
unilaterally merge with another session, but may have the ability
to approve or reject a merge initiated by the control device 102.
In addition, the control device 102 and/or the client devices 104,
106, and 108 may have the ability to create a new session (e.g.,
rather than merging an existing session) and add that session to an
existing session.
[0037] The GUI 200 provides a session list for the user of the
device 102, with a session indicator 202a for the session 110, a
session indicator 202b for the session 112, and a session indicator
202c for the session 114. In the present example, each session
indicator 202a-202c includes a name for the individual
communicating via the respective client device.
[0038] The GUI 200 may further include controls for each of the
sessions 110, 112, and 114, with a control 204a for session 110, a
control 204b for session 112, and a control 204c for session 114.
The controls 204a-204c enable a currently active session to be put
on hold (e.g., the control 204b) or may be used to switch to a
currently held session (e.g., the controls 204a and 204c).
Switching to a session that is on hold may automatically put the
current session hold. Putting the current session on hold may
retain the current session as the active session (e.g., may place
the current session on hold without switching to another session).
Additional controls 206a and 206b may be used to merge existing
sessions.
[0039] A video display window 208 may be provided for a video call
in the active session. As the current session is with Rashmi, her
name may be provided in the window. Controls such as a speaker
button 210, a mute button 212, and a video button 214 may be
provided for control over the audio and video aspects of the
call.
[0040] A screenshare button 216 may be provided to enable screen
sharing. A send button 218 may be provided to send data in the
current session. For example, selection of the send button 218 may
open a dialog box or menu from which one or more files, URLs,
and/or other data may be selected for transmission to the client
device separate from, or embedded into, the ongoing call. The terms
"file" and "media" as used herein include any type of text, image,
model (e.g., a 3-D model or information needed to render/manipulate
such a model), audio, and/or video file or combination of files,
and may include files having multiple media types (e.g., an
audio/video file or a text document with inline or linked
images).
[0041] The data may be local on the control device 102, may be
retrieved from a storage location (e.g., a server or database)
prior to being sent by the control device, or may be accessed by
the client device from a storage location (e.g., a server or
database) using a link or other access mechanism provided by the
control device. The data may be send as a discrete file or
streamed, depending on the particular file and transfer mechanism.
Data may also be provided via a link to the client device.
[0042] The send process may be accomplished in different ways. The
system may be limited using various parameters or may dynamically
adjust how data is sent based on factors such as available
bandwidth and processing power, the type of data to be sent,
whether the call is on hold or active, and similar factors.
[0043] Referring to FIG. 3, for example, a file may be sent during
an active call so that the control and/or client devices may
display the file while the call is ongoing. The file may be an
image, a text file, a video file, or any other file type or
combination of types. The GUI 200 may provide a file view window
300 with a file viewing space 302. Depending on the size of the
file, a slider 304 and slider control 306 may be used to scroll
through the file vertically. A horizontal control (not shown)
similar to the slider 304 for horizontal movement may be provided
in some embodiments. A 3D control (not shown) may be provided to
manipulate 3D files. One or more tabs or other selection
components, such as tabs 308a-308d, may be provided to access
multiple files.
[0044] In other embodiments, data may be sent while the call is on
hold, enabling the client device to display the video file or other
file while waiting for the control device 102 to restore the
session to active status. One embodiment includes sending/beginning
to stream a file and then placing the session on hold (the file may
still be sending/streaming while on hold in some examples). Another
embodiment includes placing the session on hold and then
sending/streaming the file to the client device. Still another
embodiment includes sending a URL to the client device before or
after placing the session on hold, and the client device selects
the URL to download or stream the file. Such data transmissions may
require user acceptance/authorization or may automatically play on
the client device.
[0045] In some embodiments, a URL or a file may be sent or streamed
to the client device from a preset list of available options before
or after the session is put on hold. The selection may be manual or
automatic, with automatic selections having a variety of options
such as predefined, prioritized, based on the current session's
context, and/or random. In some embodiments, user engagement may be
tracked to record whether the file was viewed and, if so, how much
of the file was viewed. In other embodiments, the file may simply
be transferred without tracking user engagement.
[0046] Each feature of the GUI 200, including the session
indicators 202a-202c and controls 204a-204c, 206a, and 206b may
include text, colors, border modifications (e.g., dotted lines,
thicker or multiple lines, etc.), and/or other variations to
indicate the status of the associated session and/or available
functionality. For example, the active session of session indicator
202b may be outlined or filled with green, and the inactive
sessions of session indicators 202a and 202b may be outlined or
filled with yellow or red. Additionally or alternatively, some or
all section indicators may be shaded in color, may display symbols,
etc., to indicate the status of their respective sessions.
Functionality may be grayed out or removed if not available.
Furthermore, the amount of time on hold may be indicated via a
visible timer, changes in the session indicator (e.g., slowly
turning from yellow to red), and/or using other methods.
[0047] In addition, it is understood that many different
arrangements of various elements illustrated on the GUI 200 may be
used. Elements may be removed, added, combined, further divided,
and positioned differently. Accordingly, it is understood that many
different graphical presentation and implementation techniques may
be used with the GUI 200 and the present disclosure is not intended
to be limited to those explicitly illustrated or described.
[0048] Referring to FIG. 4, one embodiment of the GUI 200 of FIG. 3
is shown with session 110 merged with session 112, so a video
window 208b for Siva is now present. Document sharing and other
functionality may continue with both Siva and Rashmi.
[0049] Referring to FIG. 5, one embodiment of the GUI 200 of FIG. 2
is shown with a single call session. Some functionality displayed
with respect to FIG. 2 is not displayed in this example since only
the single call is present.
[0050] Referring to FIG. 6A, one embodiment of a GUI 600 is
illustrated. The GUI 600 may represent, for example, the backend
VRM management interface 122b of FIGS. 1G and 1H. In the present
example, the GUI 600 includes one or more possible call queues 602.
The call queues include a queue for a control device 604a and a
control device 604b (or names associated with those devices). The
queue for the control device 604 has four available slots, with
slots 606a-606c indicated as being busy, and slot 606d indicated as
being available. The control device 604b has no available slots and
is indicated as being offline.
[0051] The GUI 600 is displaying incoming requests 608a-608c. The
incoming request 608a has been selected, showing additional
selection possibilities such as an additional information button
610, an assignment button 612, and a schedule button 614. The
incoming request button 610 may provide additional information
about the caller, whether a particular queue should be used (e.g.,
an appointment for a particular doctor rather than a general call
for first available), and similar user specific details. The
assignment button 612 may be used to assign the call to a
particular available slot (e.g., the slot 606d). This will place
the request in the virtual waiting room for the control device
604a. The schedule button 614 may be used to schedule a later call,
such as with the control device 604b when it comes online. In some
embodiments (not shown), the control device 604b may have slots
available for future calls, and the incoming request 608a may be
assigned to one such slot.
[0052] Referring to FIG. 6B, a flow chart illustrates one
embodiment of a process 620, executable by the control device 102
or a component of the communications framework 120, by which client
devices may be invited to a virtual waiting room and provided media
to display while waiting. In step 622, the control device 102 or a
component of the communications framework 120 sends an invitation
to a client device. In step 624, the incoming client device is
placed in the virtual waiting room. In step 626, media (e.g.,
audio, video, text, images, and/or other types of files) or URLs
representing media locations may be sent to the waiting client
device. In some embodiments, step 626 may occur as part of the
invitation of step 622, between steps 622 and 624, or with step
624.
[0053] In step 628, a list of client devices in the virtual waiting
room may be displayed. In step 630, a selection of a particular
client device is received with any additional options (e.g.,
whether the session is to be initiated as a chat session, audio
only call, or video call). In step 632, the session is initiated.
Step 632 will generally end or suspend the media on the client
device. Playback may continue if the session is again placed on
hold, if the user of the client device indicates they wish to
continue, and/or based on other parameters.
[0054] Referring to FIG. 6C, a flow chart illustrates one
embodiment of a process 640, executable by a client device, by
which the client device may be invited to a virtual waiting room
and provided media to display while waiting. In step 642, the
client device 102 receives an invitation to the virtual waiting
room and joins in step 644. In step 646, media (e.g., audio, video,
text, images, and/or other types of files) or URLs representing
media locations are received by the waiting client device. In some
embodiments, step 646 may occur as part of the invitation of step
642, between steps 642 and 644, or with step 644. In step 648, the
client device plays the media while waiting for the session to
begin.
[0055] Referring to FIGS. 7A-7E, embodiments are illustrated where
the control device 102 is sending media (e.g., audio, video, text,
images, and/or other types of files) to one or more client devices.
With respect to each of FIGS. 7A-7E, as described previously, the
sending of media and/or URLs may occur with or without putting the
corresponding communication session(s) on hold. This process
enables educational material, advertisements, and/or other material
to be displayed on a client device while the session is active or
on hold.
[0056] The particular information may be selected based on the
context of the session. For example, if the session is a medical
consultation, the media may be related to available diagnostic
procedures, potentially applicable drugs, and/or other information
that may be used to inform and/or educate the patient while waiting
for the session to continue. If the session is for discussing the
purchase or rental of a product or service, the media may be
related to options, upgrades, alternatives, availability, special
deals, and/or other information that may be applicable to the
consumer. If the session is a consultation session for advice
(e.g., beauty advice, counseling, recruiting, or professional
development), the media may be related to various options,
products, subscriptions, and/or other information that may be
applicable to the individual seeking advice. The media may play
automatically or may require the user of the client device to
accept the media.
[0057] In FIG. 7A, the media is sent directly to the client device
104. In FIG. 7B, the media is sent directly to the client devices
104 and 106. The client devices 104 and 106 may be in independent
sessions with the control device 102 or may be in a conference call
together. The media sent to each of the client devices 104 and 106
may be identical or different.
[0058] In FIG. 7C, a URL is sent to the client device 104 and the
client device 104 then retrieves the media from a server 702 using
the URL. In FIG. 7D, a URL is sent to each of the client devices
104 and 106, and each client device 104 and 106 then retrieves the
media from the server 702 using the URL. The client devices 104 and
106 may be in independent sessions with the control device 102 or
may be in a conference call together. The URL sent to each of the
client devices 104 and 106 may be identical or different, and may
direct the client devices to the same server or to different
servers.
[0059] In FIG. 7E, the media is sent directly to the client device
106, and a URL is sent to the client device 104, which then
retrieves the media from the server 702 using the URL. The client
devices 104 and 106 may be in independent sessions with the control
device 102 or may be in a conference call together. The media
directly sent and represented by the URL may be identical or
different.
[0060] Referring to FIG. 8, a flow chart illustrates one embodiment
of a process 800 by which media may be sent during a period in
which a communication session is on hold or to be placed on hold.
In step 802, the control device 102 or a component of the
communications framework 120 identifies that the current session
has been placed on hold. In step 804, media options are provided to
a user of the control device 102. In step 806, a media selection is
received. In step 808, the media or URL is injected into the media
session for delivery to the client device 104. In some embodiments,
step 802 may occur after step 804, 806, or 808.
[0061] Referring to FIG. 9, a flow chart illustrates one embodiment
of a process 900 by which media may be sent during a period in
which a communication session is on hold or to be placed on hold.
In step 902, the control device 102 or a component of the
communications framework 120 identifies that the current session
has been placed on hold. In step 904, media is automatically
selected using a predefined priority list, a randomized selection
process, or another method. In step 908, the media or URL is
injected into the media session for delivery to the client device
104. In some embodiments, step 902 may occur after step 904 or
906.
[0062] Referring to FIG. 10, a flow chart illustrates one
embodiment of a process 1000 by which previously sent media may
have been fully consumed or a request may be received by the client
device 104. In step 1002, the control device 102 or a component of
the communications framework 120 detects that previously sent media
has ended and/or receives a request for media from the client
device 104. In step 1004, media is obtained as described with
respect to steps 804 and 806 of FIG. 8 or step 904 of FIG. 9. In
step 1006, the media or URL is injected into the media session for
delivery to the client device 104. In some embodiments, step 1002
may occur after step 1004.
[0063] Referring to FIG. 11, a flow chart illustrates one
embodiment of a process 1100 by which media may be received for use
by the client device 104 during a period in which a communication
session is on hold or to be placed on hold. In the present example,
the session 110 has been placed on hold prior to step 1102, but the
hold may be placed at any time before the execution of step 1110 in
other embodiments.
[0064] In step 1102, media or a URL is received. Although not
shown, an authorization step may be present in some embodiments
that enables a user of the client device 104 to reject the
media/URL. If this occurs, the method 1100 may move directly to
step 1116. The following description of FIG. 11 assumes that
authorization has occurred or was not needed.
[0065] If media is received in step 1102 as determined in step
1104, the method proceeds to step 1108 where the media is played.
If a URL is received, the method 1100 moves to step 1106 to obtain
the media before moving to step 1108. If the hold remains in place
as determined in step 1110, the media may continue playing. It is
understood that step 1110 may be an interrupt rather than a
determination, in which case the media may play until an interrupt
occurs.
[0066] If the hold is removed, the media is ended/suspended in step
1112 and the method 1100 moves to step 1118 where the call is
continued. There may be a closing notice, such as a an audio or
text "Thank you" as the media is being closed. If the hold remains,
the method 1110 may determine if the media has ended (e.g.,
finished playing if a video or audio file). If the media has not
ended, the method 1100 may return to step 1108 and continue playing
the media. If the media has ended, the method 1100 waits for the
removal of the hold in step 1116. Once the hold is removed, the
call continues in step 1118. In some embodiments, the client device
104 may request additional media in following step 1114 or may
receive additional media as indicated in FIG. 10.
[0067] In some embodiments described in the present disclosure,
analytical data may be compiled. For example, if media is played
while sessions are on hold, the media name, number of times the
media is played, whether the media was paused or reversed for
additional viewing, the control device that selected the media, and
similar information may be compiled. Such data may be used to
determine effectiveness of particular media, gauge interest,
identify effectiveness or use patterns relative to other media,
etc.
[0068] Referring to FIG. 12A, in one embodiment, the components of
the environment 100 of FIG. 1A are illustrated in a medical
environment 1200. Accordingly, the environment 1200 includes a
medical facility 1202, which may be all or a portion of a hospital,
clinic, doctor's office, and/or any other medical facility. It is
understood that the medical facility 1202 may operate within a
larger facility, such as a clinic located within a hospital or on a
hospital's grounds. The medical facility 1202 may include or
otherwise access a server 1204, a local electronic storage 1206
(e.g., a database), and/or a remote electronic storage 1208. In
some embodiments, the server 1204 may incorporate one or both of
the local/remote electronic storages 1206/1208 into the server. In
other embodiments, the server 1204 may be located outside of the
medical facility 1202, such as a server provided by a hospital for
a clinic's use or a cloud based server.
[0069] The server 1204 may host at least some of the technology
needed for the communications and virtual waiting room
functionality to operate. For example, if the technological
framework is a hybrid peer-to-peer system, the server 1204 may
function as the server that performs operations needed for such a
system as described in previously referenced U.S. Pat. Nos.
7,656,870; 7,623,516; 7,623,476; 7,933,260; 8,050,272; 8,218,444;
8,446,900; 8,694,587; 8,892,646; 9,027,032; and 9,356,997.
[0070] With additional reference to FIG. 12B, another embodiment of
the medical environment 1200 is illustrated as an environment 1220.
In this example, the server 1204 and electronic storage 1208 (which
may be combined with or separate from the server 1204) are remote
from the medical facility 1202. For example, the server 1204 and
electronic storage 1208 may be provided via cloud services that are
owned or leased by the medical facility 1202.
[0071] With respect to both FIGS. 12A and 12B, if the server 1204
and local/remote electronic storages 1206/1208 are under control of
the medical facility 1202, oversight of legal and ethical
considerations (e.g., compliance with the Health Insurance
Portability and Accountability Act (HIPAA) and other applicable
laws and ethical rules) may be simplified as third parties will not
be in control or have access to any regulated information. It is
noted that the server 1204 and electronic storage 1208 of FIG. 12B
may be viewed as being under control of the medical facility as
long as the medical facility personnel are the only ones to have
access to the remote components and/or regulated information that
is processed or stored by the server/electronic storage. In
addition, as each session may be end-to-end encrypted for both
signaling and media, sessions can be channeled through third-party
servers without compromising private information.
[0072] If media is to be injected into a session while the session
is active or on hold, a URL or list of URLs may be provided to the
server 1204, one or both of the electronic storages 1206/1208,
and/or to the control device 102. This avoids the need to store the
media within the medical facility, although such storage may be
used for at least some of the media in some embodiments. Media may
also be stored on a remote server 1210 that may or may not be
associated with the medical facility 1202.
[0073] In some embodiments, the control device 102 may not be
located within the medical facility some or all of the time. For
example, if the control device 102 is a tablet, a doctor within the
medical facility may use the tablet from home. In other
embodiments, the control device 102 may be limited in functionality
based on a geofence, time parameters, and/or other
restrictions.
[0074] The previously described functionality and interfaces may be
directed to the medical environment 1200. For example, the control
device 102 may be associated with a doctor, a nurse practitioner,
or other medical personnel (all personnel may be generally referred
to herein as "doctor" for purposes of illustration). The client
devices 104, 106, and 108 may be associated with patients of the
doctor. The sessions 110, 112, and 114 are to remain separate as
each may involve the transfer of confidential and sensitive health
data in the form of verbal and visual communications, text chat,
documents, images (e.g., CAT scans and X-rays), etc.
[0075] As described previously, additional client devices may be
added to a session in a conference call if desired. For example, if
the patient is a minor, a parent or guardian may be present. Family
members, friends, other medical personnel (e.g., specialists),
and/or other individuals may be added to a session. If needed,
consent forms may be sent to users of client devices to ensure that
proper documentation procedures are followed, just as they would be
for in-office visits. Accordingly, by sending and receiving online
forms, scans or images of signatures, and similar electronic
documents, the present disclosure enables established procedures to
be maintained even in virtual consultations.
[0076] Reminders may be used if documents are needed, or
functionality may be blocked until certain steps are taken. For
example, audio and video may not be available to a client device
unless a consent form is received. It is understood that such
constraints are customizable and may not be required unless
desired. Patient education (PE) information, advertisements, and
other information may be sent to the client devices as described in
previous embodiments, either directly (e.g., as shown in FIG. 7A)
or indirectly (e.g., as shown in FIG. 7B).
[0077] Referring to FIG. 12C, one embodiment of an environment 1230
illustrates components of the environments 1200 (FIG. 12A) and 1220
(FIG. 12B), although it is understood that the components of the
environment 1230 may be used in any many different implementations
of the present disclosure. In the present example, the server 1204
is represented by a gateway server 1204a and a conference server
1204b, which may include a multipoint control unit (MCU) designed
to facilitate conference calls. In other embodiments, the
functionality of the servers 1204a and 1204b may be performed by a
single server, in which case only the single server would be
present. In the present example, the control device 102 is in
one-to-one sessions (e.g., an active call, on hold, or waiting)
with each of the client devices 104, 106, and 108. Accordingly, the
conference server 1204b is not needed. The gateway server 1204a
handles signaling for the control device and client devices, while
media moves directly between the control device 102 and each client
device.
[0078] Referring to FIG. 12D, one embodiment of an environment 1240
illustrates the components of the environment 1230 in a conference
call scenario. The control device 102 is involved in a conference
call with the client devices 104, 106, and 108. The gateway server
1204a handles signaling for the control device and client devices,
while media moves between the control device and client devices
through the conference server 1204b.
[0079] FIGS. 13A-17F are embodiments of sequence diagrams
illustrating various processes that may be executed within the
environments of FIGS. 1A, 1B, and 12A-12D. Although the particular
examples of FIGS. 13A-17F may contain details relevant specifically
to the medical environment 1200 of FIGS. 12A and 12B (e.g.,
references to doctors, patients, patient education material, and
other medical oriented terminology), the various steps may be
applied to many different environments, including sessions between
consultants and clients, teachers and students, and salespeople and
customers.
[0080] It is understood that the sequence diagrams described herein
illustrate various exemplary functions and operations that may
occur within various communication environments. It is understood
that these diagrams are not exhaustive and that various steps may
be excluded from the diagrams to clarify the aspect being
described. For example, it is understood that some actions, such as
network authentication processes and notifications, may have been
performed prior to the first step of a sequence diagram. Such
actions may depend on the particular type and configuration of a
particular device, including how network access is obtained (e.g.,
cellular or WLAN access). Other actions may occur between
illustrated steps or simultaneously with illustrated steps,
including network messaging for call maintenance (including
handoffs), communications with other devices (e.g., email, text
messages, and/or voice calls (including conference calls)), and
similar actions. In addition, is it understood that single messages
may be illustrated or described, and such messages may actually
represent a series of messages.
[0081] Referring to FIGS. 13A-13D, one embodiment of a sequence
diagram illustrates a process 1300 by which the control device 102
(representing a doctor) within the medical environment of FIGS. 12A
and 12B may communicate with client devices 104 (representing a
first patient) and 106 (representing a second patient) in separate
sessions. It is understood that some steps (such as file sharing or
screen sharing) may occur at different times than those illustrated
in FIGS. 13A-13D (or not at all), and that such steps are included
to show how general message flows for various features may be
incorporated.
[0082] For purposes of simplicity in this and following
embodiments, the control device 102 may be hosting or otherwise
providing the waiting room. In embodiments where a separate device
or server is providing the waiting room (e.g., the gateway server
1204a of FIGS. 12C and 12D), additional messages would be passed
between the control device 102 and the hosting device/server with
information about waiting patients, etc., and client devices would
interact with the hosting device/server as needed.
[0083] Referring specifically to FIG. 13A, in step 1302, the
control device 102 logs into a server 1204, which is the server
1204 of FIG. 12A or 12B in the present example. In steps 1304 and
1306, respectively, the control device 102 sends an invitation to
the client devices 104 and 106 via email, SMS, or other
communication channels. In steps 1308 and 1310, respectively, the
client devices 104 and 106 join the doctor's waiting room (which
may be specific to the doctor or may be a general waiting room in
the medical facility). In step 1312, the server 1204 notifies the
control device 102 of the available status of the client devices
104 and 106. In steps 1314 and 1316, respectively, the server 1204
notifies the client devices 104 and 106 of the available status of
the control device 102. In steps 1318 and 1320, respectively, the
client devices 104 and 106 notify the control device 102 that they
are in the waiting room.
[0084] Referring specifically to FIG. 13B, in step 1322, the
control device 102 sends patient education material to the client
device 106, which the client device 106 displays to its user in
step 1324. As this process has been described in previous
embodiments, it is not further detailed here. In step 1326, the
control device 102 starts a session (e.g., an audio or video call)
with the client device 104. In step 1328, the control device 102
notifies the server 1204 that it is on a call, and the server 1204
updates the control device's status to the client device 106 in
step 1330. At this time, there is an active call between the
control device 102 and the client device 104 as indicated by step
1332. In step 1334, the control device 102 places the active call
with the client device 104 on hold. In step 1336, the control
device 102 sends patient education material to the client device
104, which the client device 104 displays to its user in step
1338.
[0085] Referring specifically to FIG. 13C, in step 1340, the
control device 102 removes the hold on the call with the client
device 104. When the hold is removed, the display of the media sent
to the client device 104 in step 1336 may be paused or otherwise
suspended. At this time, the active call between the control device
102 and the client device 104 continues as indicated by step 1342.
In step 1344, the control device 102 shares one or more files with
the client 104 during the active call. In step 1346, the active
call between the control device 102 and the client device 104 is
ended. In step 1348, the control device 102 updates its status with
the server 1204 as available, and the server 1204 notifies the
client device 106 of the updated status in step 1350. In step 1352,
the control device 102 instructs the server 1204 to disconnect the
client device 104 from the waiting room, and the server 1204
notifies the client device 104 of the disconnection in step
1354.
[0086] Referring specifically to FIG. 13D, in step 1356, the
control device 102 sends a chat message to the client device 106
while the client device is displaying the media. For example, the
chat message may inform the user of the client device that the
doctor is now ready or will be ready in a particular amount of time
(e.g., five minutes). In step 1358, the control device 102 starts
the call with the client device 106. At this time, there is an
active call between the control device 102 and the client device
106 as indicated by step 1360. In step 1362, the control device 102
sends patient education material to the client device 106 during
the active call, which the client device 106 displays to its user
in step 1364. In step 1366, the control device 102 shares its
screen with the client 106 during the active call. In step 1368,
the control device 102 shares one or more files with the client 106
during the active call.
[0087] In step 1370, the active call between the control device 102
and the client device 106 is ended. In step 1372, the control
device 102 updates its status with the server 1204 as available. In
step 1374, the control device 102 instructs the server 1204 to
disconnect the client device 106 from the waiting room, and the
server 1204 notifies the client device 106 of the disconnection in
step 1376.
[0088] Referring to FIGS. 14A and 14B, one embodiment of a sequence
diagram illustrates a process 1400 by which the control device 102
(representing a doctor) within the medical environment of FIGS. 12A
and 12B may communicate with client devices 104 (representing a
first patient) and 106 (representing a second patient) in separate
sessions. It is understood that some steps (such as file sharing or
screen sharing) may occur at different times than those illustrated
in FIGS. 14A and 14B (or not at all), and that such steps are
included to show how general message flows for various features may
be incorporated.
[0089] Referring specifically to FIG. 14A, in step 1402, the
control device 102 logs into a server 1204a, which is the server
1204 of FIG. 12A or 12B in the present example. In step 1404, the
control device 102 sends an invitation to the client device 104 via
email, SMS, or another communication channel. In step 1406, the
client device 104 joins the doctor's waiting room (which may be
specific to the doctor or may be a general waiting room in the
medical facility). In step 1408, the server 1204a notifies the
control device 102 of the available status of the client device
104. In step 1410, the server 1204a notifies the client device 104
of the available status of the control device 102. In step 1412,
the client device 104 notifies the control device 102 that it is in
the waiting room. In step 1414, the control device 102 begins a
call session with the client device 104. At this time, there is an
active call between the control device 102 and the client device
104 as indicated by step 1416.
[0090] In step 1418, the control device 102 notifies the server
1204a to begin recording. In step 1420, the server 1204a instructs
the server 1204b (which may be the same server as the server 1204a
or a different server) to set up a recording session for the
control device 102 and the client device 104. In steps 1422 and
1426, respectively, the server 1204a sends a start recording
message to the control device 102 and client device 104 to begin
recording. This message may contain information needed to send data
to the server 1204b.
[0091] Referring specifically to FIG. 14B, in steps 1426 and 1428,
respectively, the control device 102 and the client device 104 send
session media to the server 1204b to be recorded. In step 1530, the
control device 102 shares its screen with the client device 104,
which may happen many different times during the call. In step
1432, the control device 102 notifies the server 1204a to stop
recording. In step 1434, the server 1204a instructs the server
1204b to stop the recording session. In steps 1436 and 1438,
respectively, the server 1204a sends a stop recording message to
the control device 102 and client device 104.
[0092] In 1440, the active call between the control device 102 and
the client device 104 is ended. Although not shown, the control
device 102 may update its status with the server 1204a as
available. In step 1442, the control device 102 instructs the
server 1204a to disconnect the client device 104 from the waiting
room, and the server 1204a notifies the client device 104 of the
disconnection in step 1444. In step 1446, the control device 102
may retrieve the recording from the server 1204b and play back the
session in step 1448.
[0093] Referring to FIGS. 15A-15F, one embodiment of a sequence
diagram illustrates a process 1500 by which the control device 102
(representing a doctor) within the medical environment of FIGS. 12A
and 12B may communicate with client devices 104 (representing a
first patient) and 106 (representing a second patient) in a
conference call. It is understood that some steps (such as file
sharing or screen sharing) may occur at different times than those
illustrated in FIGS. 15A-15F (or not at all), and that such steps
are included to show how general message flows for various features
may be incorporated.
[0094] Referring specifically to FIG. 15A, in step 1502, the
control device 102 logs into a server 1204a, which is the server
1204a of FIGS. 12C and 12D in the present example. In steps 1504
and 1506, respectively, the control device 102 sends an invitation
to the client devices 104 and 106 via email, SMS, or other
communication channels. In steps 1508 and 1510, respectively, the
client devices 104 and 106 join the doctor's waiting room (which
may be specific to the doctor or may be a general waiting room in
the medical facility). In step 1512, the server 1204a notifies the
control device 102 of the available status of the client devices
104 and 106. In steps 1514 and 1516, respectively, the server 1204a
notifies the client devices 104 and 106 of the available status of
the control device 102. In steps 1518 and 1520, respectively, the
client devices 104 and 106 notify the control device 102 that they
are in the waiting room.
[0095] Referring specifically to FIG. 15B, in step 1522, the
control device 102 sends patient education material to the client
device 106, which the client device 106 displays to its user in
step 1524. As this process has been described in previous
embodiments, it is not further detailed here. In step 1526, the
control device 102 starts a session (e.g., an audio or video call)
with the client device 104. In step 1528, the control device 102
notifies the server 1204a that it is on a call, and the server
1204a updates the control device's status to the client device 106
in step 1530. At this time, there is an active call between the
control device 102 and the client device 104 as indicated by step
1532.
[0096] In step 1534, the control device 102 invites the client
device 106 to join the ongoing call of step 1532. However, the
control device 102 does not host the conference call itself and so
sends a message to the server 1204a that it is joining a conference
call. The message may include the identify of the other
participants (e.g., the client devices 104 and 106) and may serve
as an instruction to the server 1204a to set up the call. In steps
1540 and 1542, respectively, the client devices 104 and 106 send
messages to the server 1204a to join the conference call. In step
1542, the server 1204a connects to a server 1204b, which is the
server 1204b of FIGS. 12C and 12D in the present example.
[0097] Referring specifically to FIG. 15C, in steps 1544, 1546, and
1548, the server 1204b joins the control device 102, client device
104, and client device 106 in a conference call. As the server
1204b manages the media for the conference call, media for the call
(e.g., audio and/or video) flows between the server 1204b and each
of the control device 102, client device 104, and client device 106
as shown by arrows 1550, 1552, and 1554, respectively. File sharing
data also flows between the server 1204b and each of the control
device 102, client device 104, and client device 106 as shown by
arrows 1556, 1558, and 1560, respectively.
[0098] Referring specifically to FIG. 15D, in step 1562, the
control device 102 may send patient education material to the
server 1204b, which then sends the material to the client devices
104 and 106 in steps 1564 and 1566, respectively. The client 104
displays the material to its user in step 1568 and the client
device 106 displays the material to its user in step 1570. Chat
messages also flow between the server 1204b and each of the control
device 102, client device 104, and client device 106 as shown by
arrows 1572, 1574, and 1576, respectively.
[0099] Referring specifically to FIG. 15E, in step 1578, the
control device 102 may initiate screen sharing to share its screen
with the client devices 104 and 106. In the present example,
screensharing uses a third server 1204c, but this server may be
combined with the server 1204b in other embodiments as shown, for
example, in FIG. 15G. In step 1580, the server 1204a notifies the
server 1204b of the screenshare, and the server 1204b (which is
handling the conference call) sends a message to the server 1204c
to set up the screenshare in step 1581. In step 1582, the server
1204c sends a message to the server 1204b with screenshare session
information, and the server 1204b passes this information to the
server 1204a in step 1583. In step 1584, the server 1204a sends the
screenshare information to the control device 102. In step 1585,
the control device 102 sends screen share data to the server 1204c,
and the server 1204c sends the data to the client devices 104 and
106 in steps 1586 and 1587, respectively.
[0100] Referring specifically to FIG. 15F, in step 1588, the
control device 102 instructs the server 1204b to end the conference
call. The server 1204b instructs the server 1204c to end the
screenshare session in step 1589 and sends a message to the server
1204a that the conference call is being ended in step 1590. In
steps 1591-1593, respectively, the server 1204a notifies the
control device 102, the client device 104, and the client device
106 that the conference is over. In step 1594, the control device
102 indicates to the server 1204a that it is available. In step
1595, the control device 102 instructs the server 1204a to
disconnect the client devices 104 and 106 from the waiting room.
The server 1204a notifies the client devices 104 and 106 of the
disconnection in steps 1596 and 1597, respectively.
[0101] Referring to FIG. 15G, an alternate embodiment of some steps
of FIGS. 15E and 15F is illustrated with the servers 1204b and
1204c combined into a single server 1204b. Accordingly, FIG. 15G
illustrates the initiation of the screenshare in step 1578 (FIG.
15E) through the ending of the conference call in step 1590 (FIG.
15F) using the single server 1204b. As shown, steps 1581, 1582, and
1589 of FIGS. 15E and 15F are omitted, and other steps (e.g., steps
1585-1587) involve the server 1204b. It is understood that steps
similar or identical to steps 1581, 1582, and 1589 may still occur
internally within the architecture of the server 1204b.
[0102] Referring to FIGS. 16A and 16B, one embodiment of a sequence
diagram illustrates a portion of an alternate process 1600 by which
the control device 102 (representing a doctor) within the medical
environment of FIGS. 12A and 12B may communicate with client
devices 104 (representing a first patient) and 106 (representing a
second patient) in a conference call. In the present example, the
conference call is initiated directly with both client devices
rather than adding a participant to an active call as shown with
respect to FIGS. 15A-15G. Accordingly, only the initial steps are
shown in FIGS. 16A and 16B, as later steps are identical to those
in FIGS. 15A-15G.
[0103] Steps 1602-1624 are identical to steps 1502-1524 of FIGS.
15A and 15B and are not described in detail with respect to the
present example. In step 1626, rather than initiate a call only
with the client 104 as was done in step 1526 (FIG. 15B), the
control device 102 sends a message to the server 1204a to begin the
conference call with both client devices 104 and 106. In steps
1628, 1630, and 1632, respectively, the server 1204a sends
conference messages to the control device 102, client device 104,
and client device 106. The control device 102, client device 104,
and client device 106 send messages to the server 1204a to join the
conference call in steps 1634, 1636, and 1638, respectively. In
step 1640, the server 1204a connects to the server 1204b (similar
to step 1542 of FIG. 15B). The process 1600 then proceeds in the
same manner as the process 1500 in steps 1544-1597 of FIGS.
15C-15G.
[0104] Referring to FIGS. 17A-17F, one embodiment of a sequence
diagram illustrates a process 1700 by which the control device 102
(representing a doctor) within the medical environment of FIGS. 12A
and 12B may communicate with client devices 104 (representing a
first patient) and 106 (representing a second patient) in a
conference call. The present example is similar to the process 1500
of FIGS. 15A-15F with the addition of recording some or all
portions of the conference call. It is understood that some steps
(such as file sharing or screen sharing) may occur at different
times than those illustrated in FIGS. 17A-17F (or not at all), and
that such steps are included to show how general message flows for
various features may be incorporated. Although not shown, it is
understood that the alternate conference call establishment of FIG.
15G may be incorporated into FIGS. 17A-17F as described with
respect to FIGS. 15A-15F.
[0105] Referring specifically to FIG. 17A, in step 1702, the
control device 102 logs into a server 1204a, which is the server
1204a of FIGS. 12C and 12D in the present example. In steps 1704
and 1706, respectively, the control device 102 sends an invitation
to the client devices 104 and 106 via email, SMS, or other
communication channels. In steps 1708 and 1710, respectively, the
client devices 104 and 106 join the doctor's waiting room (which
may be specific to the doctor or may be a general waiting room in
the medical facility). In step 1712, the server 1204a notifies the
control device 102 of the available status of the client devices
104 and 106. In steps 1714 and 1716, respectively, the server 1204a
notifies the client devices 104 and 106 of the available status of
the control device 102. In steps 1718 and 1720, respectively, the
client devices 104 and 106 notify the control device 102 that they
are in the waiting room.
[0106] Referring specifically to FIG. 17B, in step 1722, the
control device 102 sends patient education material to the client
device 106, which the client device 106 displays to its user in
step 1724. As this process has been described in previous
embodiments, it is not further detailed here. In step 1726, the
control device 102 starts a session (e.g., an audio or video call)
with the client device 104. In step 1728, the control device 102
notifies the server 1204a that it is on a call, and the server
1204a updates the control device's status to the client device 106
in step 1730. At this time, there is an active call between the
control device 102 and the client device 104 as indicated by step
1732.
[0107] In step 1734, the control device 102 invites the client
device 106 to join the ongoing call of step 1732. However, the
control device 102 does not host the conference call itself and so
sends a message to the server 1204a that it is joining a conference
call. The message may include the identify of the other
participants (e.g., the client devices 104 and 106) and may serve
as an instruction to the server 1204a to set up the call. In steps
1740 and 1742, respectively, the client devices 104 and 106 send
messages to the server 1204a to join the conference call. In step
1742, the server 1204a connects to a server 1204b, which is the
server 1204b of FIGS. 12C and 12D in the present example.
[0108] Referring specifically to FIG. 17C, in steps 1744, 1746, and
1748, the server 1204b joins the control device 102, client device
104, and client device 106 in a conference call. As the server
1204b manages the media for the conference call, media for the call
(e.g., audio and/or video) flows between the server 1204b and each
of the control device 102, client device 104, and client device 106
as shown by arrows 1750, 1752, and 1754, respectively.
[0109] In step 1756, the control device 102 notifies the server
1204a to begin recording. In step 1758, the server 1204a instructs
the server 1204b (which may be the same server as the server 1204a
or a different server) to setup a recording session for the control
device 102 and the client device 104. At this point, there is an
active recording session with the server 1204b sending conference
call data to the server 1204c for recording as shown by step
1760.
[0110] Referring specifically to FIG. 17D, patient education
material may be sent and viewed in steps 1762-1767 as described in
previous examples. In steps 1768-1770, chat messages may be
exchanged as described in previous examples. As indicated by arrow
1771, this information may be sent from the server 1204b to the
server 1204c for recording. It is understood that the recording may
occur in various manners, including the sending of batched data or
via streaming, and so may occur as media, files, chat messages,
and/other data are being received from and sent to the various
devices 102, 104, and 106. Accordingly, the active recording may be
an ongoing process and is not limited to the illustrated
arrows.
[0111] Referring specifically to FIG. 17E, a screenshare session
may be established in steps 1772-1780 using a fourth server 1204d
(which may be combined with the server 1204b) as previously
described. File sharing may occur in steps 1781-1783 as previously
described. As indicated by arrow 1784, this information may be sent
from the server 1204b to the server 1204c for recording.
[0112] Referring specifically to FIG. 17F, in step 1785, the
control device 102 sends a message to the server 1204b to stop
recording. In step 1786, the server 1204b instructs the server
1204c to stop recording. In step 1787, the control device instructs
the server 1204b to end the conference call. In step 1788, the
server 1204b sends a message to the server 1204d to end the
screensharing session. In step 1789, the server 1204b notifies the
server 1204a that the conference has ended.
[0113] In steps 1790-1792, respectively, the server 1204a notifies
the control device 102, the client device 104, and the client
device 106 that the conference is over. In step 1793, the control
device 102 indicates to the server 1204a that it is available. In
step 1794, the control device 102 instructs the server 1204a to
disconnect the client devices 104 and 106 from the waiting room.
The server 1204a notifies the client devices 104 and 106 of the
disconnection in steps 1795 and 1796, respectively. In step 1797,
the control device 102 may retrieve the recording from the server
1204b and play back the session in step 1798.
[0114] FIGS. 18-26 illustrate embodiments of a GUI that may be used
by the control device of FIGS. 1A, 1B, and 12A-12D. Although the
particular examples of FIGS. 18-26 may contain details relevant
specifically to the medical environment 1200 of FIGS. 12A and 12B,
various illustrated features may be applied to many different
environments. It is understood that the functionality needed to
provide the GUIs may be present on the control device 102 (e.g., as
an application) or may be present on a server that is accessible to
the control device, in which case the control device may use an
application or a browser (with or without client side downloads
such as applets) to access the functionality. It is understood that
the illustrated displays and functionality may vary with more or
fewer functions than shown, and placement, size, and other details
may also change depending on the particular implementation.
[0115] As described in previously incorporated U.S. Provisional
Application Ser. No. 63/176,419, filed on Apr. 19, 2021, and
entitled SYSTEM AND METHOD FOR HIGHLY SCALABLE BROWSER-BASED
AUDIO/VIDEO CONFERENCING, in some embodiments, audio/video data for
the control device and/or client device(s) for single and/or
conference calls may be displayed via a browser (e.g., using
Chrome, Safari, Internet Explorer, Brave, Opera, or a similar
browser) without the use of a client-side application or plug-in on
the device. It is understood that in other embodiments the present
disclosure may be applied to environments in which a device uses an
application or browser plug-in to communicate with the other device
or MCU 1204b, and the description of browser only communications is
not intended to be limiting.
[0116] By relying strictly on the browser's inherent capabilities
without the use of applications or plug-ins, the ability to join
and participate in the conference call is available to any device
with a browser. This simplifies joining a conference call, and
enables joining even if the device does not permit the download or
installation of applications or browser plug-ins. Furthermore, this
provides a level of security to the device, as there are no
downloads to be installed or authorized in order to access the
conference call. In addition, as many browsers are widely used and
frequently updated for security reasons, the user of the device
need not be concerned about potential application or plug-in flaws
that might compromise the device's security if not updated. In
addition, by relying only on the device's browser, there is less
chance of needing an update before joining a conference call, as
might happen if an application or a plug-in has not been used for a
while. This also enables even mobile devices to fully participate
in a conference call using only their built-in browser (or another
browser that is selected by the user).
[0117] To accomplish this browser focused conferencing, the
conference server 1204b may use the WebRTC framework to provide
complete conference functionality. The solution also supports the
Unified Plan for SDPs as supported by Safari, in addition to Plan-B
that is supported by Chrome and other platforms. Further support
may be provided using a cross-platform JavaScript SDK that is fully
featured.
[0118] Referring to FIG. 18, one embodiment of a GUI 1800
illustrates a screen display of a control device 102 that enables a
user of the device to view a virtual waiting room, select client
devices with which to communicate, view call logs, and/or perform
other operations. In the present example, the GUI 1800 provides
various functions and information. Actuators (e.g., buttons) are
present for different views, including a dashboard button 1802, a
conference button 1804, a call logs button 1806, and a sigh out
button 1808. A selector button 1810 enables the left panel to be
hidden/unhidden.
[0119] A waiting room 1812 provides a view of anyone waiting in the
virtual waiting room. In the present example, the waiting room is
currently empty. An invitation panel 1814 provides a link and a way
in which to invite users to the waiting room or a call. For
purposes of example, the invitation panel 1814 includes a link 1816
that may be sent to invitees, where clicking on the link will place
them in the waiting room. For example, sending the link result in
step 1304 of FIG. 13A. A copy button 1818 enables the link to be
copied for convenience. An invite button 1820 allows the selection
of a communication channel for the invitation, which is either
email or SMS as shown by a popup window 1822.
[0120] Referring to FIG. 19, another embodiment of the GUI 1800
illustrates a single user identified as "Tim" now present in the
waiting room 1812. A pane 1902 indicates the name and status (e.g.,
waiting) of the client and an options selector 1904 may be
present.
[0121] Referring to FIG. 20, another embodiment of the GUI 1800
illustrates a second user identified as "Siva" now present in the
waiting room 1812. A pane 2002 indicates the name and status (e.g.,
waiting) of the client and an options selector 2004 may be
present.
[0122] Referring to FIG. 21, another embodiment of the GUI 1800
illustrates a window 2102 that appears when the selector 1904 is
actuated. In the present example, the window 2102 is a popup
window, but it is understood that other view options may be
provided. The window 2102 may include a picture 2104 of the client
along with their name and status. A status indicator 2106 may be
provided to visually indicate their status using color and/or other
representations. For example, a green dot may indicate ready and
waiting, a yellow dot may indicate the client has placed the call
on hold, has a poor connection, or is not ready (e.g., an
appointment time has not been reached), and a red dot may indicate
lack of a connection.
[0123] Various buttons or other actuators may provide
functionality, including a chat button 2108 to initiate a chat
message, a file share button 2110 to initiate a file share, an
audio call button 2112 to initiate an audio call, and a video call
button 2114 to initiate a video call. A patient education button
2116 may be used to allow the selection of patient education
material to send to the client. A disconnect button 2118 enables
disconnection of the current client, which may disconnect them from
the waiting room.
[0124] Referring to FIGS. 22A-22C, another embodiment of the GUI
1800 illustrates a window 2202 that appears when the selector 2116
is actuated. Referring specifically to FIG. 22A, in the present
example, the window 2202 includes patient education material in the
form of videos that may be selected and sent to a user (in this
came, the user Tim). It is understood that the materials may be
presented and/or selected in different ways, including checkboxes,
radio buttons, and/or drop down lists. Furthermore, it may be
possible to select multiple videos. A close button 2206 enables the
window 2202 to be closed, and a play button 2208 enables the
selected content to be played (e.g., sent to the user for display).
FIG. 22B illustrates an example where the patient education
material is in the form of text/images. FIG. 22C illustrates an
example where the patient education material is in the form of both
video and text/images.
[0125] Referring to FIG. 23, one embodiment of the GUI 1800
illustrates a video call display for the control device 102 when a
video call is established with user Tim. A photo button 2302
enables a photo to be captured by the control device 102. A
conference call button 2304 enables one or more participants to be
added to the conference call. A screenshare button 2306 enables the
user of the control device 102 to share the screen (e.g., step 1366
of FIG. 13D). A file transfer (or share) button 2308 enables a file
to be sent (e.g., step 1344 of FIG. 13C).
[0126] A waiting room view 2310 provides a view of users in the
virtual waiting room, with Siva currently waiting. An options
selector 2312 may provide options (e.g., some or all of the options
of the window 2102 of FIG. 21) for interacting with Siva. A view
window 2314 provides a self-view (e.g., a view using a camera of
the control device 102). A view window 2316 provides a view of Tim
(e.g., using the camera of Tim's client device). A selector button
2318 enables the left panel to be hidden/unhidden.
[0127] Various buttons may be used to provide additional functions
and/or control the call. A chat button 2320 may be provided to
enable text messaging. A video button 2322 may be provided to
enable and disable video. A pause button 2324 may be used to place
the call on hold, and may be replaced by a play button (not shown)
to remove the call from hold. A mic button 2326 may be used to mute
and unmute audio. A disconnect button 2328 may be used to end the
call.
[0128] Referring to FIG. 24, one embodiment of the GUI 1800
illustrates a window 2402 that enables a conference call to be
established. In the present example, the window 2302 enables one or
both of the users in the waiting room to be invited to the
call.
[0129] Referring to FIG. 25, one embodiment of the GUI 1800
illustrates a conference call display for the control device 102
when a conference call is established with two users Siva and Tim.
A participants button 2502 enables a participant view. An add
participants button 2504 enables one or more participants to be
added to the conference call. A screenshare button 2506 enables the
user of the control device 102 to share the screen. A file share
button 2508 enables a file to be sent.
[0130] Referring to FIG. 26, one embodiment of the GUI 1800
illustrates call logs that are displayed after the call logs button
1806 is selected. As shown, call logs may be selected, sorted, and
viewed based on a number of criteria.
[0131] Referring to FIG. 27, one embodiment of a GUI 2700
illustrates a screen display of a client device (e.g., one of the
client devices 104, 106, and 108) that enables a user of the device
to join a virtual waiting room and participate in a call via the
waiting room. For purposes of example, the client device 104 is a
smart phone and the display uses a browser with controls 2702, but
other devices and/or applications may be used. The GUI 2700 may
include a viewing portion 2704 showing a web page that provides the
user with the ability to join the waiting room by entering a name
in a name field 2706 and selecting a button 2708.
[0132] Referring to FIG. 28, one embodiment of the GUI 2700 shows
the window 2704 after check in. A status indicator 2802 for the
doctor may visually indicate availability. A window 2804 may
display patient education material (e.g., such as that sent in step
1336 of FIG. 13B). It is understood that controls for the
material's display may be provided and may vary based on the type
of material (e.g., text or video).
[0133] Referring to FIG. 29, one embodiment of the GUI 2700 shows
the window 2704 while waiting, but without patient education
material.
[0134] Referring to FIG. 30, one embodiment of a computer system
3000 is illustrated. The computer system 3000 is one possible
example of a system component or computing device such as a
communication device, a document server, an endpoint, and/or an
access server. The computer system 3000 may include a controller
(e.g., a central processing unit ("CPU")) 3002, a memory unit 3004,
an input/output ("I/O") device 3006, and a network interface 3008.
The components 3002, 3004, 3006, and 3008 are interconnected by a
transport system (e.g., a bus) 3010. A power supply (PS) 3012 may
provide power to components of the computer system 3000, such as
the CPU 3002 and memory unit 3004. It is understood that the
computer system 3000 may be differently configured and that each of
the listed components may actually represent several different
components. For example, the CPU 3002 may actually represent a
multi-processor or a distributed processing system; the memory unit
3004 may include different levels of cache memory, main memory,
hard disks, and remote storage locations; the I/O device 3006 may
include monitors, keyboards, and the like; and the network
interface 3008 may include one or more network cards providing one
or more wired and/or wireless connections to a network. Therefore,
a wide range of flexibility is anticipated in the configuration of
the computer system 3000.
[0135] The computer system 3000 may use any operating system (or
multiple operating systems), including various versions of
operating systems provided by Microsoft (such as WINDOWS), Apple
(such as Mac OS X), UNIX, and LINUX, and may include operating
systems specifically developed for handheld devices, personal
computers, and servers depending on the use of the computer system
3000. The operating system, as well as other instructions (e.g.,
for the processes and message sequences described herein), may be
stored in the memory unit 3004 and executed by the processor 3002.
For example, if the computer system 3000 is the control device 102
or a client device 104, 106, 108, the memory unit 3004 may include
instructions for performing some or all of the message sequences
and methods described with respect to such devices in the present
disclosure.
[0136] The network 3016 may be a single network or may represent
multiple networks, including networks of different types. For
example, the control device 102 or a client device 104, 106, 108
may be coupled to a network that includes a cellular link coupled
to a data packet network, or data packet link such as a wide local
area network (WLAN) coupled to a data packet network. Accordingly,
many different network types and configurations may be used to
establish communications between the control device 102, client
devices 104, 106, 108, servers, and/or other components described
herein.
[0137] Exemplary network, system, and connection types include the
internet, WiMax, local area networks (LANs) (e.g., IEEE 802.11a and
802.11g wi-fi networks), digital audio broadcasting systems (e.g.,
HD Radio, T-DMB and ISDB-TSB), terrestrial digital television
systems (e.g., DVB-T, DVB-H, T-DMB and ISDB-T), WiMax wireless
metropolitan area networks (MANs) (e.g., IEEE 802.16 networks),
Mobile Broadband Wireless Access (MBWA) networks (e.g., IEEE 802.20
networks), Ultra Mobile Broadband (UMB) systems, Flash-OFDM
cellular systems, and Ultra wideband (UWB) systems. Furthermore,
the present disclosure may be used with communications systems such
as Global System for Mobile communications (GSM) and/or code
division multiple access (CDMA) communications systems. Connections
to such networks may be wireless or may use a line (e.g., digital
subscriber lines (DSL), cable lines, and fiber optic lines).
[0138] Communication among the control device 102, client devices
104, 106, 108, servers, and/or other components described herein
may be accomplished using predefined and publicly available (i.e.,
non-proprietary) communication standards or protocols (e.g., those
defined by the Internet Engineering Task Force (IETF) or the
International Telecommunications Union-Telecommunications Standard
Sector (ITU-T)), and/or proprietary protocols. For example,
signaling communications (e.g., session setup, management, and
teardown) may use a protocol such as the Session Initiation
Protocol (SIP), while data traffic may be communicated using a
protocol such as the Real-time Transport Protocol (RTP), File
Transfer Protocol (FTP), and/or Hyper-Text Transfer Protocol
(HTTP). A sharing session and other communications as described
herein may be connection-based (e.g., using a protocol such as the
transmission control protocol/internet protocol (TCP/IP)) or
connection-less (e.g., using a protocol such as the user datagram
protocol (UDP)). It is understood that various types of
communications may occur simultaneously, including, but not limited
to, voice calls, instant messages, audio and video, emails,
document sharing, and any other type of resource transfer, where a
resource represents any digital data.
[0139] While the preceding description shows and describes one or
more embodiments, it will be understood by those skilled in the art
that various changes in form and detail may be made therein without
departing from the spirit and scope of the present disclosure. For
example, various steps illustrated within a particular sequence
diagram or flow chart may be combined or further divided. In
addition, steps described in one diagram or flow chart may be
incorporated into another diagram or flow chart. Furthermore, the
described functionality may be provided by hardware and/or
software, and may be distributed or combined into a single
platform. Additionally, functionality described in a particular
example may be achieved in a manner different than that
illustrated, but is still encompassed within the present
disclosure. Therefore, the claims should be interpreted in a broad
manner, consistent with the present disclosure.
* * * * *