U.S. patent application number 17/153366 was filed with the patent office on 2021-07-22 for system, apparatus and methods for providing an intent suggestion to a user in a text-based conversational experience with user feedback.
The applicant listed for this patent is Helpshift Inc.. Invention is credited to Amit Choudhary, Yashkumar Gandhi, Swaroop Chitlur Haridas, Christian Leipski, Shyam Hirachand Shinde.
Application Number | 20210224306 17/153366 |
Document ID | / |
Family ID | 1000005359775 |
Filed Date | 2021-07-22 |
United States Patent
Application |
20210224306 |
Kind Code |
A1 |
Choudhary; Amit ; et
al. |
July 22, 2021 |
System, Apparatus and Methods for Providing an Intent Suggestion to
a User in a Text-Based Conversational Experience with User
Feedback
Abstract
Systems, apparatuses, and methods for providing a more effective
customer service support system. A trained model may be created or
generated by a process that includes acquiring a corpus of
documents (such as customer generated messages), generating
keyphrases from each document in the corpus of documents,
generating word embeddings from each document in the corpus of
documents, performing a clustering operation on the word
embeddings, assigning each document in the corpus to a duster and
labeling each duster, and training a machine learning or text
classification model using the most relevant documents in a cluster
and the label assigned to that cluster. The methods include a
processing flow that provides multiple ways for a customer to
navigate and select their desired intent or purpose in seeking
customer assistance. As part of the processing flow, the trained
machine learning model and intent tree data are provided to a
customer seeking assistance.
Inventors: |
Choudhary; Amit; (Pune,
IN) ; Haridas; Swaroop Chitlur; (Dublin, CA) ;
Leipski; Christian; (Boulder Creek, CA) ; Gandhi;
Yashkumar; (Pune, IN) ; Shinde; Shyam Hirachand;
(Pune, IN) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Helpshift Inc. |
San Francisco |
CA |
US |
|
|
Family ID: |
1000005359775 |
Appl. No.: |
17/153366 |
Filed: |
January 20, 2021 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
62964548 |
Jan 22, 2020 |
|
|
|
62964539 |
Jan 22, 2020 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06F 40/284 20200101;
G06Q 30/0281 20130101; G06K 9/6278 20130101; G06N 20/00 20190101;
G06F 16/355 20190101 |
International
Class: |
G06F 16/35 20060101
G06F016/35; G06K 9/62 20060101 G06K009/62; G06Q 30/02 20060101
G06Q030/02; G06N 20/00 20060101 G06N020/00; G06F 40/284 20060101
G06F040/284 |
Claims
1. A method of providing assistance to a customer, comprising:
creating a trained text classification model by obtaining a set of
documents; generating a set of possible keyphrases from each
document in the set of documents; generating a set of word or token
embeddings from each document in the set of documents; performing a
clustering operation to generate a set of clusters from the set of
word or token embeddings, wherein each cluster of the set of
clusters contains one or more of the word or token embeddings in
the set of word or token embeddings; assigning each document in the
set of documents to one of the set of generated clusters; assigning
a label to each of the generated clusters; forming a set of
training data for a text classification model, wherein the training
data comprises a plurality of pairs of a document in the set of
documents and a label for the cluster to which the document is
assigned; and training the text classification model using the set
of training data; providing the trained text classification model
to the customer; generating a display on a device, the display
showing a set of labels with each label corresponding to one of the
clusters generated from the word or token embeddings; and receiving
a selection of one of the set of labels by the customer, and in
response providing the customer with access to an appropriate
process to provide them with assistance.
2. The method of claim 1, wherein the set of documents comprises a
plurality of text messages.
3. The method of claim 1, wherein generating the set of possible
keyphrases further comprises constructing a set of n-grams for each
document, wherein each n-gram is considered a possible
keyphrase.
4. The method of claim 1, wherein prior to the clustering
operation, each keyphrase is ranked using term
frequency-inverse-document-frequency (tf-idf) scoring and assigning
the label to each of the generated dusters depends at least in part
on the scores associated with the keyphrases in the cluster.
5. The method of claim 1, wherein the generated display is a
hierarchy of labels.
6. The method of claim 1, wherein if a selection of one of the set
of labels is not received from the customer, then the method
further comprises determining if text entered into a text entry
field by the customer corresponds to an existing label by using the
trained model to classify the entered text.
7. The method of claim 1, wherein the text classification model is
developed using the Naive Bayes algorithm.
8. The method of claim 1, further comprising pre-processing each
document in the set of documents, wherein the pre-processing
includes one or more of removing html, URLs, numbers, and emojis,
breaking the document into sentences, breaking each sentences into
words, removing single characters, and removing stop words.
9. The method of claim 6, wherein if the text entered by the
customer does not correspond to an existing label that the customer
confirms, then the method further comprises providing the entered
text or another set of text provided by the customer to a curator,
and using the entered text or other set of text and a label
provided by the curator as training data for the text
classification model.
10. The method of claim 6, wherein providing the customer with
access to an appropriate process to provide therm with assistance
further comprises providing a message from the user to a person or
a bot, launching an application, or directing the user to a webpage
to provide the user with assistance, wherein the person, bot,
application, or webpage are associated with the label selected by
the customer, a classification generated by the trained model, or a
label provided by a curator.
11. A system for assisting a user to obtain a service, comprising:
an application installed on a computing device separate from a
remote server and configured to receive a trained text
classification model from the remote server and to generate a user
interface for the user to interact with the trained model; and the
remote server, wherein the remote server is configured to create
the trained text classification model by obtaining a set of
documents; generating a set of possible keyphrases from each
document in the set of documents; generating a set of word or token
embeddings from each document in the set of documents; performing a
clustering operation to generate a set of clusters from the set of
word or token embeddings, wherein each cluster of the set of
clusters contains one or more of the word or token embeddings in
the set of word or token embeddings; assigning each document in the
set of documents to one of the set of generated clusters; assigning
a label to each of the generated clusters; forming a set of
training data for a text classification model, wherein the training
data comprises a plurality of pairs of a document in the set of
documents and a label for the cluster to which the document is
assigned; and training the text classification model using the set
of training data; provide the trained text classification model to
the customer; wherein the application is further configured to
generate a display on the device, the display showing a set of
labels with each label corresponding to one of the clusters
generated from the word or token embeddings and the remote server
is further configured to receive a selection of one of the set of
labels by the user, and in response provide the user with access to
an appropriate process to provide them with the service.
12. The system of claim 11, wherein the set of documents comprises
a plurality of text messages.
13. The system of claim 11, wherein generating the set of possible
keyphrases further comprises constructing a set of n-grams for each
document, wherein each n-gram is considered a possible
keyphrase.
14. The system of claim 11, wherein prior to the clustering
operation, each keyphrase is ranked using term
frequency-inverse-document-frequency (tf-idf) scoring and assigning
the label to each of the generated clusters depends at least in
part on the scores associated with the keyphrases in the
cluster.
15. The system of claim 11, wherein the generated display is a
hierarchy of labels.
16. The system of claim 11, wherein if a selection of one of the
set of labels is not received from the user, then the application
is configured to determine if text entered into a text entry field
by the user corresponds to an existing label by using the trained
model to classify the entered text.
17. The system of claim 16, wherein the remote server is configured
to provide the customer with access to an appropriate process to
provide them with the service by providing a message from the user
to a person or a bot, launching an application, or directing the
user to a webpage to provide the user with assistance, wherein the
person, bot, application, or webpage are associated with the label
selected by the customer, a classification generated by the trained
model, or a label provided by a curator.
18. One or more non-transitory computer-readable media comprising a
set of computer-executable instructions that when executed by one
or more programmed electronic processors, cause the processors to
provide assistance to a customer by: creating a trained text
classification model by obtaining a set of documents; generating a
set of possible keyphrases from each document in the set of
documents; generating a set of word or token embeddings from each
document in the set of documents; performing a clustering operation
to generate a set of dusters from the set of word or token
embeddings, wherein each duster of the set of dusters contains one
or more of the word or token embeddings in the set of word or token
embeddings; assigning each document in the set of documents to one
of the set of generated dusters; assigning a label to each of the
generated clusters; forming a set of training data for a text
classification model, wherein the training data comprises a
plurality of pairs of a document in the set of documents and a
label for the cluster to which the document is assigned; and
training the text classification model using the set of training
data; providing the trained text classification model to the
customer; generating a display on a device, the display showing a
set of labels with each label corresponding to one of the clusters
generated from the word or token embeddings; and receiving a
selection of one of the set of labels by the customer, and in
response providing the customer with access to an appropriate
process to provide them with assistance.
19. The one or more non-transitory computer-readable media of claim
18, wherein if a selection of one of the set of labels is not
received from the customer, then the one or more processors are
configured to determine if text entered into a text entry field by
the customer corresponds to an existing label by using the trained
model to classify the entered text.
20. The one or more non-transitory computer-readable media of claim
18, wherein the one or more processors are configured to provide
the customer with access to an appropriate process to provide them
with assistance by providing a message from the user to a person or
a bot, launching an application, or directing the user to a webpage
to provide the user with assistance, wherein the person, bot,
application, or webpage are associated with the label selected by
the customer, a classification generated by the trained model, or a
label provided by a curator.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit of U.S. Provisional
Application No. 62/964,548, entitled "System, Apparatus and Methods
for Providing an Intent Suggestion to a User in a Text-Based
Conversational Experience with User Feedback," filed Jan. 22, 2020,
the disclosure of which is incorporated, in its entirety (including
the Appendix), by this reference.
[0002] This application also claims the benefit of U.S. Provisional
Application No. 62/964,539, entitled "User Interface and Process
Flow for Providing an Intent Suggestion to a User in a Text-Based
Conversational Experience with User Feedback," filed Jan. 22, 2020,
the disclosure of which is incorporated, in its entirety (including
the Appendix), by this reference.
BACKGROUND
[0003] In conversation-based customer service systems, service
agents and customer support personnel are typically organized in
terms of the type, category or area in which they provide customers
with assistance. For example, in an eCommerce context, some agents
specialize in providing customer service or assistance with regards
to payments/billing questions or tasks (i.e., they understand
credit card processing, chargebacks, refunds, etc.) while other
agents specialize in providing customer assistance with regards to
fulfillment/shipping (i.e., they understand physical goods
tracking, courier agencies practices and schedule, weather impacts,
routing issues, etc.). Still other agents may be responsible for
answering questions and assisting customers with tasks related to
product information or warranties.
[0004] As a result of the segmentation of customer assistance
agents, when an end-user (such as a customer or vendor) sends a
message to customer service, it has to be routed to the department
or group of agents best suited to answering the user's question or
helping them to solve their problem. Traditionally, and in
conventional systems, this has been accomplished by a manual review
of a user's input and the determination by a person of the
appropriate agent or group of agents to route the user's request or
inquiry to. This means that someone is reading through every new
incoming service request ticket and assigning it to the right
department and then forwarding the ticket to that department.
[0005] However, manual performance of the evaluation and routing of
service request tickets has multiple disadvantages, among which are
at least the following: [0006] It is slow because a human is the
bottleneck in the process and having multiple humans coordinate
this work across a large set of service requests is another
inefficient aspect of the process; [0007] It is error-prone because
it is manual and based on human judgments. For example, there is
often some amount of ambiguity in a service request (e.g., it's not
always obvious whether a ticket should be classified as "billing"
or "refund"); and [0008] It is tedious and uninteresting work,
causing a relatively high churn rate among personnel in this
role--this can impact recruiting, training and retention costs and
staffing requirements. These problems and others related to the
conventional approach provide an incentive for companies to seek
ways of partially or fully automating the evaluation and routing
processes used for customer support requests. However, in many
cases routing cannot be automated effectively with a set of rules,
because as noted, there is often no clear pattern or consistency to
how a user may phrase their request for assistance, nor is there an
effective way to remove ambiguities that arise in interpreting user
requests. For example, "I want my money back" needs to be
classified as "refund" and there are no obvious keywords to
automate such a rule (such as are used in email filtering
rules).
[0009] Embodiments of the invention are directed toward solving
these and other problems associated with conventional approaches to
providing a conversational based customer service support system,
individually and collectively.
SUMMARY
[0010] The terms "invention," "the invention," "this invention,"
"the present invention," "the present disclosure," or "the
disclosure" as used herein are intended to refer broadly to all of
the subject matter described in this document, the drawings or
figures, and to the claims. Statements containing these terms
should be understood not to limit the subject matter described
herein or to limit the meaning or scope of the claims. Embodiments
of the invention covered by this disclosure are defined by the
claims and not by this summary. This summary is a high-level
overview of various aspects of the disclosure and introduces some
of the concepts that are further described in the Detailed
Description section below. This summary is not intended to identify
key, essential or required features of the claimed subject matter,
nor is it intended to be used in isolation to determine the scope
of the claimed subject matter. The subject matter should be
understood by reference to appropriate portions of the entire
specification, to any or all figures or drawings, and to each
claim.
[0011] Embodiments of the disclosure are directed to systems,
apparatuses, and methods for providing a more effective customer
service support system. In some embodiments, the systems,
apparatuses, and methods include use of a trained machine learning
or text classification model. The trained model may be created or
generated by a process that includes acquiring a corpus of
documents (such as customer generated messages) from a company,
generating keyphrases from each document in the corpus of
documents, generating word embeddings from each document in the
corpus of documents, performing a clustering operation on the word
embeddings, assigning each document in the corpus to a cluster and
labeling each cluster, and training a machine learning or text
classification model using the most relevant documents in a cluster
and the label assigned to that cluster.
[0012] In some embodiments, the systems, apparatuses, and methods
include a processing flow that provides multiple ways for a
user/customer to navigate and select (or identify) their desired
intent or purpose in seeking customer assistance. In one part of
the processing flow, a trained machine learning model and intent
tree data are provided to a customer seeking assistance. The model
and intent tree data may be provided in response to a customer
launching a previously supplied customer service application. In
some embodiments, the customer service application may be provided
in response to the customer contacting a company.
[0013] The application is executed on the customer's device (such
as a smartphone, tablet, laptop, desktop or other computing device)
and operates to perform one or more of (a) generate a display of
the intent tree data, (b) allow the customer to navigate, and if
desired, select an intent in the intent tree, (c) allow the
customer to provide text describing their reason for seeking
assistance and cause the trained model to attempt to classify the
input text with regards to its intent, (d) receive from the
customer a confirmation of a selected intent from the intent tree
or an identified intent from the model, and (e) provide the
customer's selected intent, intent identified by the model or their
entered text (in the situation where the customer does not confirm
an intent) to a server or platform for routing to an appropriate
"handler". In the case of a confirmed intent selected from the
intent tree or a confirmed intent identified by the trained model,
the message may be routed to a Bot, webpage URL, pre-defined task
or workflow, or a person. In the case of the customer not
confirming an intent, their message or text may be routed to a
person for evaluation, labeling and routing, and in some cases then
used as part of continued training of the model.
[0014] In some embodiments, the systems, apparatuses, and methods
include a user interface (UI) and process flow for implementing a
conversation-based customer service support system. The user
interface and associated processing enable a user/customer to
quickly and accurately select an intent, goal, or purpose of their
customer service request. In some embodiments, this is achieved by
a set of user interface displays or screens and an underlying logic
that provide a user with multiple ways of selecting, identifying,
or describing the intent, goal, or purpose of their request for
service or assistance. Further, the underlying logic operates to
improve the accuracy of the system over time based on obtaining
user feedback through their selection or the processing of entered
text and the construction of a data structure representing possible
user intents and the relationship between intents or categories of
intents.
[0015] Note that although some embodiments of the system and
methods will be described with reference to the use of a
conversational interface and underlying logic as part of a customer
service support system, embodiments may be used in other situations
and environments. These may include commerce, finance, or the
selection of other services or products. In some contexts, the
described user interface and underlying logic may assist a user to
submit a service request, discover a service or product of
interest, initiate or complete a transaction, or generally express
their goal or purpose in interacting with a system or service, as
expressed by their intent, etc.
[0016] Other objects and advantages of the present invention will
be apparent to one of ordinary skill in the art upon review of the
detailed description of the present invention and the included
figures.
BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Embodiments of the invention in accordance with the present
disclosure will be described with reference to the drawings, in
which:
[0018] FIG. 1 is a flow chart or flow diagram of an exemplary
computer-implemented method, operation, function or process for
processing a user message requesting a service to determine a
user's intent or the purpose of a user inquiry, in accordance with
some embodiments of the systems and methods described herein.
[0019] FIG. 2A is a flow chart or flow diagram of an exemplary
computer-implemented method, operation, function or process for
processing training data and training a machine learning model to
identify the intent or the purpose of a user inquiry, in accordance
with some embodiments of the systems and methods described
herein.
[0020] FIG. 2B is a flow chart or flow diagram of an exemplary
computer-implemented method, operation, function or process for the
client-side operations and event update flow for a process to
identify the intent or the purpose of a user inquiry, in accordance
with some embodiments of the systems and methods described
herein.
[0021] FIG. 3 is a diagram illustrating elements or components that
may be present in a computer device or system configured to
implement a method, process, function, or operation in accordance
with an embodiment of the invention;
[0022] FIG. 4 is an illustration of an example of a static user
interface display that an end-user may use to choose from a static
list of issue titles or subjects; and
[0023] FIGS. 5-7 are diagrams illustrating an architecture for a
multi-tenant or SaaS platform that may be used in implementing an
embodiment of the systems and methods described herein.
[0024] Note that the same numbers are used throughout the
disclosure and figures to reference like components and
features.
DETAILED DESCRIPTION
[0025] The subject matter of embodiments of the present disclosure
is described herein with specificity to meet statutory
requirements, but this description is not intended to limit the
scope of the claims. The claimed subject matter may be embodied in
other ways, may include different elements or steps, and may be
used in conjunction with other existing or later developed
technologies. This description should not be interpreted as
implying any required order or arrangement among or between various
steps or elements except when the order of individual steps or
arrangement of elements is explicitly noted as being required.
[0026] Embodiments of the invention will be described more fully
herein with reference to the accompanying drawings, which form a
part hereof, and which show, by way of illustration, exemplary
embodiments by which the invention may be practiced. The invention
may, however, be embodied in different forms and should not be
construed as limited to the embodiments set forth herein; rather,
these embodiments are provided so that this disclosure will satisfy
the statutory requirements and convey the scope of the invention to
those skilled in the art.
[0027] Among other things, the present invention may be embodied in
whole or in part as a system, as one or more methods, or as one or
more devices. Embodiments of the invention may take the form of a
hardware implemented embodiment, a software implemented embodiment,
or an embodiment combining software and hardware aspects. For
example, in some embodiments, one or more of the operations,
functions, processes, or methods described herein may be
implemented by one or more suitable processing elements (such as a
processor, microprocessor, CPU, GPU, TPU, controller, etc.) that is
part of a client device, server, network element, remote platform
(such as a SaaS platform), an "in the cloud" service, or other form
of computing or data processing system, device, or platform.
[0028] The processing element or elements may be programmed with a
set of executable instructions (e.g., software instructions), where
the instructions may be stored on (or in) a suitable non-transitory
data storage element. In some embodiments, one or more of the
operations, functions, processes, or methods described herein may
be implemented by a specialized form of hardware, such as a
programmable gate array, application specific integrated circuit
(ASIC), or the like. Note that an embodiment of the inventive
methods may be implemented in the form of an application, a
sub-routine that is part of a larger application, a "plug-in", an
extension to the functionality of a data processing system or
platform, or any other suitable form. The following detailed
description is, therefore, not to be taken in a limiting sense.
[0029] The service request routing problem (or more generally, the
interpretation and processing of a user's text message as provided
by a conversational interface, such as a chat window or other
format) is one to which Natural Language Processing (NLP) based
Machine Learning (ML) models can be applied. Such models can be
trained to learn implicit rules from actual text message data or
examples. This means that if a suitable set of training data and
labels/annotations are available, then a model may be trained to
assist in identifying or classifying incoming service requests.
Thus, if one has access to hundreds (or more) examples of messages
and the corresponding labels or annotations (e.g., the correct
group or department each message should be routed to), then a
machine learning (ML) classification algorithm can be applied to
create a model that will have a relatively high degree of accuracy
and be able to perform (or at least assist in) the routing
function.
[0030] While a trained Machine Learning model can be very helpful
in routing a service request from a customer to the appropriate
agent or subject matter expert, such models can be difficult for
many organizations to create. For example, the labels used to
annotate messages need to be clearly distinct from each other and
to accurately reflect the workflow within the organization for
handling service requests. In addition, the input data (message
examples) needs to be "clean" and not subject to misinterpretation.
This requires a company or organization using the model for routing
service requests to provide appropriate training data and they may
not understand how to do this effectively.
[0031] A second type of problem that may be encountered when
attempting to implement an automated or semi-automated service
request routing process is that workflows attached to such a system
are automatically invoked without confirmation from the end-user,
who may have had a different intention or goal of their request.
For example, "I want my money back" may need to be classified as
"refund", but "I got charged twice. I want my money back" may need
to be classified as a "billing" issue instead. This can lead to
frustration and dissatisfaction on the part of a customer/user and
increased organizational resources being needed to resolve the
problem and maintain customer goodwill.
[0032] Existing solutions to assist a customer/user to route their
request to the appropriate customer service agent, process, or
function are generally of two types:
[0033] (1) Static Menu
[0034] A static menu interface is one where an end-user is asked to
choose from a static list of issue titles or subjects. An example
of such a user interface display is shown in FIG. 4. While helpful
to some extent, the disadvantages of this approach include the
following: [0035] The end-user is expected to understand what each
title means and to interpret it as intended by the person who
assigned that title, e.g., a user may not understand the term
"venue" and may be looking for "location" or "where"; and [0036] A
company or brand is expected to pay attention to the major
categories of incoming service tickets and be able to process those
categories of messages by continually curating the list of issue
categories in the static menu as new categories are needed or as
currently available categories need to be redefined.
[0037] (2) NLP Based Routing
[0038] This type of solution is an NLP engine (e.g., Siri or
Alexa). In this solution, there is a Natural Language Processing
model that tries to understand the meaning of the sentence or
phrase entered by a user, and each meaning is attached to a
workflow. However, a problem with this approach is that human
languages are very expressive, and people can have the same intent
but express their request in different ways. This means that for a
routing engine to be effective and correctly handle a large
percentage of submitted requests, it would need to be trained for
all of the possible (or at least the most likely) ways of
expressing a request for a particular service. This places a burden
on the company or brand wanting to use a trained model for routing
service requests to have comprehensive training data, and even
then, there is a low confidence level in achieving a desired level
of success and often unclear expectations on the accuracy of the
routing engine.
[0039] As another example, the assignee of the present
disclosure/application has developed a customer service request
issue classification service/model referred to as "Predict."
Predict is a server-side text classification engine that is
intended to determine the most likely label (typically a category
or routing destination) for an incoming service request. However,
the results are not exposed to the user and are only used for
internal routing within the platform. The inputs used to train the
Predict model are multiple pairs of data, with each pair comprising
a label and a user's first service request message. The model is
trained on these pairs using a supervised ML algorithm. Once the
Predict model is trained, it operates to receive as an input a new
user's first message, and in response it provides as an output a
prediction of what the correct label for the message should be.
When the classification/prediction has a sufficiently
high-confidence level associated with it, the message is
automatically routed to the group or department corresponding to
the output label.
[0040] However, a limitation of this approach is that Predict
operates to predict the correct "label" for a request on the
server-side or service platform. An unfortunate consequence of this
architecture is that if there is a workflow attached to the label,
then the end-user is immediately taken to a bot or routine (or
other form of handler) to execute the workflow. This can be a
serious problem for the user in the situation where the
classification or label was incorrect, as it can be both
frustrating and inefficient. Further, as with the other approaches
described, there are one or more possible problems that may occur
as a result of the Predict processing flow: [0041] 1. Predict is
expected to have a high degree of confidence in its predictions, so
that once the confidence level for classifying a new message
crosses a pre-set threshold, the system automatically applies the
output of the model to the request/ticket. This means that the
labels used to classify and route messages need to accurately
reflect an organization's desired processing of requests. It also
means that the labels (and hence routing possibilities) need to be
regularly reviewed and corrected or updated to reflect how the
organization is actually processing requests; and [0042] 2.
Workflows attached to Predict are automatically invoked without
confirmation or other feedback from the end-user, who may have had
a different intention or goal of the request. This can lead to
frustration and dissatisfaction on the part of a customer/user.
[0043] In some embodiments, the proposed solution and processing
described herein overcomes these disadvantages by providing a form
of compromise between, and improvement over, the approaches
represented by existing solutions. In some embodiments, a user's
request for assistance is associated with an "Intent", where an
intent is an issue topic, subject or category that reflects the
user's intention or purpose for seeking assistance (or in some
cases, a goal of their interaction with a system as expressed by
their intent). Typically, responding to an inquiry or assisting a
customer to achieve an intent has a workflow associated with it,
where the workflow may be performed by a "handler" such as a Bot,
workflow, routine, or person. In the proposed solution described
herein, a hierarchy or arrangement of intents (or data structure)
that represents the relationships between different categories,
sub-categories or classifications of intents is termed an "intent
tree".
[0044] In some embodiments, the systems, apparatuses, and methods
described herein provide an end-user with one or more of a user
interface display, user-interface elements (such as a selectable
"intent", a hierarchical intent tree, and/or a text entry field)
and underlying data processing and user input processing logic, and
may include: [0045] A dynamic (i.e., one able to be updated or
changed) user interface (UI) which provides a user/customer with
multiple ways of navigating a set of issue/intent categories and
ways of helping them find the category that best matches their
problem or purpose in seeking assistance, or the task, function or
operation they desire to have performed, etc.; [0046] This is
beneficial, as it means an end-user is not expected or required to
understand what each category title means, or which title
represents a category or workflow that is most applicable to their
problem or issue; [0047] A dynamic UI where new intents/categories
can be displayed; [0048] This means that the list of issue
categories presented to a user/customer is not static and can
display a different or updated list of intents when the user opens
the conversational UI in a later session; [0049] In some
embodiments, the updated or revised list of intents or intent tree
structure may be constructed based on a customer's previous
interactions with customer service (such as the manner in which the
customer expresses their requests using idiom or slang, or the
manner in which prior requests were expressed and subsequently
classified), an updated intent tree model for a company, an updated
text message interpretation model for a company, etc.; [0050] In
some embodiments, the UI may ask a user/customer to explicitly
confirm their intent (i.e., that the assigned or selected category
is a correct one for addressing their issue or desired operation);
[0051] This means that an end-user will not be surprised to see the
workflow associated with that issue category; for example, the
workflow can be a bot that "walks" them through the solution steps
(e.g., forgot password) or a FAQ ("What is the shipping policy?"),
or the request can be assigned to a queue where a human agent
responds (e.g., reporting abuse); and [0052] If a user does not
identify/select their desired intent as a result of their
navigating through the intent tree display (or other form of
representing the set of possible intents, such as a list,
hierarchy, table, etc.), then the client application and/or remote
system platform may enable the user to enter text into a field or
user interface element and utilize the trained classification model
to attempt to find a "best" match to the text. The trained model
will operate to attempt to find a matching intent or intents to the
user's text and then present the user with this list or set of
intents for possible selection. Thus, in this aspect of some
embodiments, text entered by a user is provided as an input to the
trained model, with the output of the model being one or more
"intents" that the model has classified the input text as
representing. However, if the user's text does not sufficiently
"match" an existing intent label or the end-user chooses not to
select or confirm an intent suggested by the trained model, then
embodiments permit the user to enter a full-text question or phrase
specifying what they want or need; [0053] In some embodiments, a
match may depend on, for example, finding an exact match to text,
finding a sufficiently dose match to text, finding a match to
synonyms of (or other ways of expressing) the text, etc.; [0054]
The full-text question or phrase is processed on the server-side
(which in some embodiments, may include human interpretation of the
text) to attempt to identify the most relevant existing intent
label or topic. Under certain circumstances, the process may create
a new intent category or sub-category that the company or brand can
assign a title to and have added to the intent tree for use with
future requests for assistance from customers; [0055] In cases
where the server-side processing determines that the request should
be associated with an existing intent label or category, the
initial request from the customer and the existing (correct) label
may be used as additional training data for the classification
model used to process requests for that specific company.
[0056] As will be described in greater detail with reference to
FIG. 1, embodiments of the user interface and supporting processing
described herein provide 3 paths or sequences of navigation that
are available to a user/customer as part of enabling them to select
or specify their desired intent or purpose for requesting
assistance. Although an embodiment described herein represents an
example of using a conversational UI in the context of a user
seeking customer support assistance, it is noted that other use
cases and contexts exist in which the system and methods described
herein can be utilized to provide benefits to users and to the
organizations receiving the user requests for assistance. These use
cases and contexts include but are not limited to, commerce,
finance, and the selection of other services or products. In some
contexts, the described user interface and underlying logic may
assist a user to submit a service request, discover a service or
product of interest, initiate or complete a transaction, find
information, or select a workflow to accomplish a goal or purpose,
etc.
[0057] FIG. 1 is a flow chart or flow diagram of an exemplary
computer-implemented method, operation, function or process for
processing a user message requesting a service to determine a
user's intent or the purpose of a user inquiry, in accordance with
some embodiments of the systems and methods described herein. As
shown in the figure, an initial step is to collect or form a set of
training data to be used in training a machine learning model (as
suggested by step 120). In one embodiment, the training data set is
comprised of a set of the first messages or requests sent by a
plurality of users/customers along with an assigned label or
classification for each message. The label or classification
represents the "correct" category or routing of the message, as
curated by a person familiar with the routing categories and
classifications used by the brand or company whose customers
generated the messages.
[0058] Thus, in this embodiment, a machine learning model is
trained for use in processing incoming service requests from
end-users to a specific company, business, vendor, or organization
(referred to as the "company" herein). In this example, the company
may have its own titles or task categories that are used for
processing and routing incoming requests to specific work groups or
departments. These titles or task categories will be reflected by
the "correct" label(s) or annotations associated with a message or
request. In a typical embodiment, this acquisition of training data
and training of a machine learning model will occur on a server or
platform.
[0059] In some examples, the training of the machine learning model
may be performed or assisted by the entity providing the other
functionality of the processing flow illustrated in FIG. 1. For
example, the same entity that operates a system or services
platform to provide message processing and routing services may
provide tools to enable a company to upload a sample of customer
service request messages and to enable an employee of the company
to provide proper annotations or labels to those messages. The
uploaded messages and labels can be used by the service platform
provider to create the trained machine learning model for that
company. The trained model is provided to the company's customers
as described herein. In other examples, the company itself may
generate the trained model and provide it to the entity operating
the service platform.
[0060] Thus, in some embodiments, the entity operating the service
platform for processing customer service requests (or providing
that service through a multi-tenant platform operated by another
entity) may provide a set of services through each account they
administer on the platform. Each account may be provided with a set
of functions and services, as described with reference to FIGS.
5-7. The functions and services may include one or more of the
following: [0061] a process or service to acquire a corpus of
documents (such as customer generated messages) from a company or
organization; [0062] a in some examples, the corpus may be obtained
from a source of examples of customer service requests that were
(or might be expected to be) received by the company or
organization; [0063] a process or service to generate keyphrases
from each document in the corpus of documents; [0064] a process or
service to generate word or token embeddings from each document in
the corpus of documents; [0065] a process or service to perform a
clustering operation on the word or token embeddings; [0066] a
process or service to assign each document in the corpus to a
cluster and enable curation and labeling of each cluster; [0067] a
process or service to train a machine learning model using the most
relevant documents in a cluster and the label assigned to that
cluster; [0068] a process or service to provide the trained model
and a data structure representing an "intent tree" of possible
categories of requests for assistance to the company's customers
when they request assistance, send a message to a customer support
address, or launch a customer service application (or otherwise
indicate a need for assistance); [0069] a process or service to
provide an application (if it has not already been provided) to the
company's customers to display and enable customers to "navigate"
the intent tree, to select a desired intent or category of
assistance, to confirm a selected intent, to allow a customer to
insert text and have that text classified by the trained model, and
to confirm their acceptance of the classification produced by the
trained model; [0070] a process or service to allow a customer to
insert text, a phrase, a question, or a sentence (as examples) into
a field of a user interface and have the contents of the field
provided to the server/platform for further evaluation and
processing (typically by a person); and [0071] a process or service
to provide a mechanism (typically either through the application or
a service on the platform) to route a message or request to the
appropriate "handler", task, workflow, Bot, URL, person or
department of the company for resolution after determination of the
customer's intent or goal in requesting assistance.
[0072] The model is trained using the set of training data so that
it operates to receive as an input a text or other form of message
from a user/customer and in response operates to output a label or
classification of the input message, where the label or
classification represents the correct category of the request or
customer service topic and identifies where the request should be
routed. Thus, the trained model operates to receive a user's
service request or message and identify the intent or purpose of
the user's service request or message (step 122).
[0073] The trained model is made available to the company to
provide to customers requesting service and/or may be provided to
customers by the entity providing the other functionality of a
service platform, such as the message/request processing. In a
typical example, the company or system/platform receives a text or
other form of message requesting assistance from a customer (as
suggested by step or stage 123). In some embodiments, the text or
message may be sent by a customer to a server using the client-side
application. For instance, a customer may inform a brand or company
that they seek customer service assistance by using a text message
sent to a customer service number. In some embodiments, a customer
may instead send a snippet of text to a customer service email
address. In some embodiments, the text message or snippet of text
may be (re) directed to an account on the service platform. In some
embodiments, the company may receive the text message or snippet of
text and route it to the service platform.
[0074] In response to the customer's request, the system/platform
or company transfers the trained intent classification model to a
client device of the customer (step 124). If not already installed
on the client device of the customer, the system/platform or the
company may direct the customer to download and install a customer
service application on their device. The client device loads the
trained model and displays the intent tree to the customer by
generating a display or screen showing the intent groups or
categories and their relationships (typically in a visual and
hierarchical form, although others, such as a list having multiple
levels, are also possible to use) (step 126). In some embodiments,
when a customer launches a client-side application, the
system/platform may transfer the trained model and data
representing an intent tree for a company to the customer.
[0075] The user or customer is able to navigate the displayed
intent tree using user interface elements or controls on their
device and if desired, select a category or subject title/label
that best corresponds to the intent or purpose of their service
request or message (step 128). If the user has selected an intent
corresponding to their request for service, then the request is
routed to the appropriate department (as suggested by step 135),
either directly by the company or indirectly through the service
platform.
[0076] If the user/customer is unable or chooses not to select one
of the displayed intent groups or a specific intent type, then they
may enter text and have that text used as a basis for finding a
"match" to a label of the intent tree. In some embodiments, this
may be done by enabling the user to enter text into a text box or
field that is displayed. The entered text can be a partial or total
phrase (such as one or more words) that the user believes describes
the subject matter of their service request or the purpose of their
interaction (such as to initiate a transaction, search for a
product, find a specific piece of information about a product or
service, etc.). As the user is entering text (or after completion
of a word or phrase), the transferred classification model is
invoked and used to generate one or more potential matches to the
entered text, with that match or those matches displayed to the
user on the client device (step 130). In some embodiments, the
"match" processing described may be based on one or more of an
exact match between a label and the user's text, a sufficient
similarity to the user's text, and/or a match based on a synonym to
(or other way of expressing) the user's text.
[0077] In some embodiments, the trained model may instead be
executed on the server or platform side. In these embodiments, the
trained model does not need to be provided to the client. Instead,
the user's text would be transferred from the client to the server,
the matching to intents would be performed on the server-side, and
the results of the matching would be provided back to the client
for display on the device.
[0078] The user is provided with an opportunity to select and/or
confirm that one of the displayed intents corresponds to their
intent or purpose for the service request. If the user confirms an
intent corresponding to their request for service, then the request
is routed to the appropriate department (as suggested by step 135)
by the server or platform. Note that the confirmation stage or step
may be provided in response to the user/customer selecting a
displayed intent or category, or in response to the user entering
text which is classified by the trained model, where the model is
configured to provide one or more outputs representing the most
likely intent or category corresponding to the input text.
[0079] If the user is unable or unwilling to select/confirm one of
the displayed intents or one suggested by the trained model, then
they can enter a full text message into a displayed field (step
132). The full text message is transferred to the server or
platform hosting the classification model where it may be subject
to routing and/or human assessment for ultimately assisting the
user. Further, the transferred message may be used for purposes of
training the model, updating how an intent is characterized, adding
a new intent to the intent structure, processing using a clustering
or other machine learning model to determine if the user's message
should be associated with an existing group or subject, etc. (step
134). In some cases, the full text request may be evaluated by a
human "handler" or curator to determine if the request should be
associated with an existing category or label (and if desired, used
as training data for the model), or if a new category, sub-category
or label should be proposed for use with the customer's initial
message as part of training the model for the company.
[0080] In some embodiments, the entity operating the service
platform (such as the assignee of the present application) provides
the functionality or capability for (a) customer messaging, (b)
intent tree display and intent selection by a customer, and (c)
intent classification based on customer text entry using a trained
model to customers of the company. As noted, the data (previously
received customer messages) and data labels used to train the model
are provided by the company or with their assistance, with the
model training being performed by the entity operating the service
platform. The data labels are also used by the entity to build the
intent tree or other list or structure of the possible intents. In
some embodiments, the intent tree structure is built, curated and
then used as part of the labeling/annotation for the model training
process.
[0081] In some embodiments, the intent tree is displayed based on
intent tree data provided to the client by the server upon a
customer initiating a service request. In these embodiments, once a
customer begins entering text into a messaging field in the
application (or placing a cursor in a specific field or otherwise
indicating a need for assistance, such as by launching the customer
service application), the application may automatically generate a
display showing the intent tree structure (for example, a
hierarchical listing). Thus, in these embodiments, the downloaded
client application loads the intent tree and the trained model into
the customer's device. The application displays the intent tree in
a list format, although other formats may be used (e.g., a
graphical display). The user/customer may navigate the tree
structure and select an intent that best corresponds to their
intended request. In these embodiments, the trained model is not
utilized when the user is navigating the intent tree using one or
more user interface elements or controls.
[0082] If the customer does not select an intent from the
navigation of the tree and enters text, then the trained model is
invoked and used to attempt to find a match of the entered text to
an intent. In this situation, the customer's entered text is
treated as a model input and the trained model operates to generate
an output representing the label or classification for the input.
The label or classification corresponds to the model's "best"
estimate of the intent associated with the input text. The client
application displays the most relevant intents from the results of
the matching process on the client device. If the customer selects
an intent either from the navigation of the intent tree structure
or from the results of the matching process, then the user may be
asked to confirm their intent and the request is routed to the
appropriate handler by the server or platform (as suggested by step
or stage 135). If the user does not select and/or confirm an intent
(whether from the displayed intent tree structure or provided by
the output of the trained model), then the user provided message
may be used as feedback or additional training data for the
clustering algorithm.
[0083] In some embodiments, the intent tree contains the names of
the intents as well as information on their organization and layout
(e.g., their hierarchical structure and relationships). The trained
model is used when searching the intent tree based on user-provided
(typed) text or messages. For surfacing the most-likely intent(s)
for a given user text input, the model (for finding the best
matching intent), as well as the intent tree (for intent names and
hierarchical information), are used.
[0084] FIG. 2A is a flow chart or flow diagram of an exemplary
computer-implemented method, operation, function or process for
processing training data and training a machine learning model to
identify the intent or the purpose of a user inquiry, in accordance
with some embodiments of the systems and methods described herein.
Note that each "block" or module shown in the diagram and referred
to in the following description may represent a routine,
sub-routine, application, etc. that is implemented by the execution
of a set of software instructions by a programmed electronic
processor. In some embodiments, more than one programmed electronic
processor may implement the functions associated with an
illustrated block or module by executing the corresponding software
instructions in whole or in part. Thus, in some embodiments, a
plurality of electronic processors, with each being part of a
separate device, server, or system may be responsible for executing
all or a portion of the software instructions contained in an
illustrated block or module.
[0085] As shown in the figure, at block 102 the text of a set
document(s) is input to the processing flow for training a machine
learning model. In some embodiments, a document corresponds to a
message submitted in the context of a text-based conversational
experience. The collection of documents used for training a model
is referred to as the corpus. In a typical use of the systems and
methods described herein, the corpus is obtained from a specific
company or business and a model is being trained for use by
customers of that company or business.
[0086] In some embodiments, the corpus may be obtained from a
database of messages from one or more companies that have been
collected by the entity providing the trained model(s). In these
cases, a set of messages provided by multiple companies may be used
to increase the variety in types and wording of customer service
requests. For example, if a company does not have a sufficiently
large corpus of messages, then messages from other companies in a
similar industry may provide a suitable corpus for training a
model.
[0087] The text of the document(s) may be pre-processed, as
suggested by block 104. During pre-processing, one or more of the
following operations may be performed on each document:
[0088] 1. remove html, URLs, numbers, emojis;
[0089] 2. break document into sentences;
[0090] 3. break sentences into words;
[0091] 4. remove single characters; and
[0092] 5. remove stop words based on a language dependent reference
list.
Block 106 then utilizes the pre-processed data--which at this point
is a list of unigrams (single words)--to build/construct all
possible bigrams (combinations of two subsequent words, i.e., two
words that appear next to each other) and trigrams (combinations of
three subsequent words) on a document-by-document basis. All
possible n-grams across the corpus (where n is equal to 3 in this
example but may be a different number) are taken as keyphrase
candidates (i.e., possible keyphrases). Term
frequency-inverse-document-frequency (tf-idf) scoring is used to
rank all n-gram keyphrase candidates (although other scoring or
evaluation methods may be used). The top (e.g., 1,000 or another
sufficiently large number) keyphrases in this ranking are selected
for clustering. The appropriate or optimal number of selected
keyphrases may depend on the application as well as on the variety
of subject matters inherent to the corpus.
[0093] In general, the amount or number of keywords selected for
clustering may depend upon the use case and variety encountered in
the corpus. For example, based on the work of the inventors, 1000
key phrases proved to be a good compromise for the use case of
customer support. However, as the inventors learned, too few
keyphrases may limit the granularity that can be achieved (i.e.,
the ability to separate related but different intents). Similarly,
too many keyphrases can result in increased noise and reduced
quality of the final intent separation and document assignments to
intents.
[0094] Note that other methods for extracting or identifying
keyphrases from a corpus of documents are available and may also
(or instead) be used; in deciding upon the extraction method to
utilize, a model developer should consider that the appropriate
number of selected keyphrases depends on the application, as well
as on the variety of subject matters inherent to the corpus. As an
example, the list of keyphrases should provide coverage of all
primary subject matters in the corpus, while each keyphrase should
be relevant to one or more of these subject matters.
[0095] The approach described herein is based (at least in part) on
consideration of the results obtained and the context in which the
technique was being used, including practical implementation and
usage considerations. These practical considerations include a
relative ease of implementation, the scaling metric being
interpretable and the results being reproducible for multiple
languages considered relevant for the business application
(customer service) and the "intent surfacing" process.
[0096] In block 108 the process uses the pre-processed corpus of
documents to build word embeddings, which are dense numerical
representations of words. The word embeddings may be built using
Word2vec (a technique for natural language processing the Word2vec
algorithm uses a neural network model to learn word associations
from a large corpus of text), or a similar application which may be
used to accomplish the function. The Word2vec algorithm is applied
to all (pre-processed) documents of the input corpus. Where they
appear in the documents, the keyphrases selected in 106 are
considered single tokens during processing. As such, the Word2vec
algorithm will provide a separate embedding vector for each unique
token in the corpus, including the keyphrases (therefore identical
tokens in different documents will have the same embedding vector).
Note that a token is an individual unit of text, most commonly a
word, but can also represent a multi-word phrase. The final
embedding vectors of the keyphrases are then normalized and
subjected to the clustering operation. During that operation, the
keyphrases are separated into k clusters.
[0097] The Word2vec algorithm is trained on the (pre-processed)
input corpus. Block 110 performs a clustering operation on the
normalized word embeddings for the keyphrases selected in block 106
and clusters those into k clusters. In this disclosure, word
embeddings are used because they represent text in a way that
captures similarities between words. As such, words with similar or
related context or subject matter are closely aligned in the
embedding space. As a consequence, the clustering of word
embeddings can result in grouping keyphrases according to their
subject matter. Note that keyphrases are considered tokens in the
processing phase and the Word2vec algorithm provides one embedding
vector for each token in the corpus. Thus, after the
word-embeddings stage of the processing, there results a one-to-one
mapping of tokens (including keyphrases) to embedding vectors.
[0098] In some embodiments, the process uses the KMeans algorithm
to duster the keyphrases as expressed in the embedding space (i.e.,
the embedding vectors for each keyphrase). In one embodiment, the
standard KMeans clustering is modified to utilize cosine similarity
as the relevant metric. The output of the clustering process is a
unique assignment of each of the input keyphrases to a cluster.
[0099] Each document may then be assigned to a duster based on the
keyphrases present in the document. In one embodiment, for each
keyphrase present in the document, the process records the tf-idf
score and its cluster assignment. For a given document, a
cumulative tf-idf score is calculated for each cluster by summing
keyphrase tf-idf scores based on their cluster assignment. In one
example embodiment, the document is assigned to the cluster which
provides the highest cumulative tf-idf score of keyphrases within
the document. This means that the cluster to which a document is
assigned is the one containing the keyphrases/embeddings found to
be most similar to the keyphrases in the document.
[0100] In some embodiments, clustering may be performed in several
layers or hierarchies: a first clustering may be performed on the
entire keyphrase collection. In the case of additional layers, each
of the resulting clusters may again be clustered in a similar
fashion. However, the input to the clustering process is
constrained to the normalized word embeddings for the keyphrases
that were assigned to the respective parent duster during the first
clustering process.
[0101] Note that other techniques/algorithms to create "Intent
trees"/"intent hierarchies" may be used. For the general process
and goals described in this application, the particular
clustering/grouping algorithm used is not constrained to the
approach described. Another approach that delivers a grouping of
documents based on subject matter would also be applicable and
could be used. The approach described herein was selected because
it outperformed other approaches that were evaluated for the
particular use case and datasets of interest.
[0102] The output of the clustering process, in terms of depth
and/or width, depends on the properties of the input corpus, as
well as on the context of the application of the resulting trained
model. For example, overly granular clustering may result in
redundant clusters (i.e., several clusters sharing or representing
the same general subject matter) or may not be desired from a
business perspective (i.e., it may limit options for efficient
navigation by the end user). Overly coarse clustering may group
keyphrases with unrelated subject matter into the same duster and
be a source of confusion or ambiguity.
[0103] For each layer of clustering, the output or result of the
clustering is a unique assignment of the keyphrases and relevant
documents to a duster. An example of the intent tree or hierarchy
resulting from the clustering is shown in the box labeled "Intent
Tree" in the figure. As shown, the set of documents is assigned to
a hierarchical arrangement of Intents and Sub-Intents, with each
Intent and Sub-Intent being associated with one or more keyphrases
and documents. In block 112 the process uses this grouping of
keyphrases and documents for intent curation. For each duster,
manual (or in some embodiments, semi- or fully automated)
inspection of the keyphrases and documents assigned to that cluster
informs the curator (e.g., someone familiar with the operational
flow of the customer support process for the company that provided
the documents) of the general "Intent" of the cluster.
[0104] As mentioned, in some embodiments, the system and process
described herein may use the comprehension and inputs of a person
acting as a curator to assign succinct and descriptive titles to
the clusters. In some embodiments, this labeling or curation may be
performed in an automated or semi-automated fashion using a trained
machine learning model. In one embodiment, an automated or
semi-automated approach may include: [0105] labeling clusters by
their most representative keyphrases; [0106] labeling clusters by
using the most frequent words (or phrases) in the documents
assigned to the cluster; or [0107] applying text summarization
techniques on the documents assigned to the cluster to determine
cluster labels.
[0108] In one example, an Intent Tree or Intent Hierarchy may be
constructed using a bottom-up approach: first the clusters on the
most granular (bottom) layer of the clustering are named. If
appropriate, these Intents may be grouped into several groups or
"parent" categories. For the purpose of naming or surfacing, these
parent or higher-level categories of intents are considered an
aggregation of the curated Intents grouped within them.
[0109] Each duster now has an Intent or label assigned to it. Also,
each duster has a list of the most relevant documents corresponding
to that intent as a result of the document assignment clustering
performed in block 110. This information is used to build a labeled
training data set for each cluster. Block 114 takes the most
relevant documents for each cluster on the most granular (or
bottom) level and labels them using the appropriate Intent as
curated in block 112. These "Intent-document" pairs are used to
train a model using a supervised text classification algorithm
based on the Naive Bayes principle (although other classification
techniques may be used in addition to or instead). The result is a
trained text classification model or machine learning model that
operates to evaluate an input document/message and produce as an
output a classification of the document, where the classification
identifies the appropriate intent of the document (such as to seek
customer assistance with their account, etc.).
[0110] Note that although in some examples, the Naive Bayes
methodology is used, other supervised text classification
algorithms may instead (or also) be used. Further, alternative
supervised text classification algorithms may not utilize a
term-intent matrix as described here for the example of using the
Naive Bayes method. Instead, and depending on the specific
algorithm, they may utilize other ways to encode a prediction
model. In that case, differences may extend to the training,
prediction, update and other processes, as described herein and
with reference to the other figures.
[0111] The resulting trained text classification model is
represented by a term-Intent matrix and an intent hierarchy. Note
that in this context, "term" is typically equivalent to "token" or
"word" (where for some languages or phrases, each token may
represent something other than what is typically considered a word
in another language). A "word-intent matrix" is an aspect of the
Naive Bayes algorithm that encodes the relative importance of all
"words" in the model across all "intents" in the model. As such, it
is noted that the concept of a "word-intent matrix" may not apply
to other classification models or approaches that may be chosen to
perform a similar task. In some embodiments, the Naive Bayes
algorithm was chosen for implementation because it is relatively
easy to implement, relatively fast to execute and the model size
can be controlled. All these aspects were relevant for on-device
execution of the classification processing.
[0112] In some embodiments, the term-intent matrix and the Intent
hierarchy (as represented by a data structure such as a
hierarchical tree or list) are transferred to the customer's client
device in block 116 when a text-based conversational experience is
initiated on the client side. Text inputs on the client side are
evaluated by the trained model using the term-intent matrix, which
provides a probability distribution across all intents for the
input. A set of intents are then surfaced on the client in a visual
manner, with the set ranked in accordance with their probability
scores and serving as suggestions for the most likely intent
corresponding to the user's text input.
[0113] The user interface (UI) displays, and underlying processing
logic are described in U.S. Provisional Patent Application No.
62/964,539, entitled "User Interface and Process Flow for Providing
an Intent Suggestion to a User in a Text-Based Conversational
Experience with User Feedback", referred to previously in this
application. As mentioned, in some cases a user may identify or
select an intent from a displayed intent tree or other data
structure to indicate the goal of their inquiry. In other cases,
they may enter text into a text entry field in the user interface
and the trained model will be used to evaluate the entered text and
attempt to "match" it to an intent or intents as described.
[0114] FIG. 2B is a flow chart or flow diagram of an exemplary
computer-implemented method, operation, function or process for the
client-side operations and event update flow for a process to
identify the intent or the purpose of a user inquiry, in accordance
with some embodiments of the systems and methods described herein.
As shown in FIG. 2B, text inputs on the client side are evaluated
in block 202 using the term-intent matrix described with reference
to FIG. 2A, which provides a probability distribution across all
intents for the input text. The intents are then presented to a
user on the client device in a visual display or other suitable
manner, ranked in accordance with their probability scores. If one
of the surfaced intents matches the user's intent, then the user
may select that intent and enter a downstream workflow or
experience tailored to be responsive to the selected intent (block
204). The workflow may include, for example, direction of the user
to a FAQ page, initiation of a programmed chat bot process to
collect information or providing the message to a human service
agent.
[0115] The words (text fragments) that the user entered before
choosing an intent, together with the intent that was eventually
selected, may be recorded in block 206 on the server side. This
pairing of text fragment and selected intent can be considered a
labeled training example for updating the trained text classifier.
An update process for the model is performed in block 208 on a
regular basis in an automated fashion. The resulting, updated
term-intent matrix as well as the revised intent tree or other
intent hierarchy are transferred to the client in block 220 when a
text-based conversational experience is initiated on the client
side. In some embodiments, this doses the loop to block 202.
[0116] If the user chooses not to engage in a downstream workflow
or experience--possibly because the customer did not select one of
the displayed intents and no matching intents (or sufficiently
closely matching in the opinion of the user) were presented or
otherwise surfaced--then the user may submit a message to start a
text-based conversation (block 210). Such messages are recorded and
pooled on the server side (block 212). At regular time or event
count intervals, this pool of recorded messages (which do not have
an intent attached to them) may be used to discover new intents not
currently represented in the current intent tree (block 214). This
discovery may be performed in an automated, semi-automated, or
completely human driven process.
[0117] If a new intent is discovered, then the most relevant
messages assigned to the new intent from the pool of submitted
messages are used to define labeled training data for this intent
(block 216). In one embodiment, a Human's comprehension ability is
used as a curator to assign succinct and descriptive titles to the
new intents and to place them into the existing intent tree or
other form of hierarchy at the appropriate location and level.
Combining the new training data and labels with the existing model
(block 218) creates an updated model which includes the new
intent(s). The resulting, updated term-intent matrix as well as the
intent hierarchy are provided to the client in block 220 when a
text-based conversational experience is initiated on the client
side. In some embodiments, this closes the loop to block 202.
[0118] As noted, in some embodiments, the inventive system and
methods may be implemented in the form of an apparatus that
includes a processing element and a set of executable instructions.
The executable instructions may be part of a software application
and arranged into a software architecture. In general, an
embodiment of the invention may be implemented using a set of
software instructions that are designed to be executed by a
suitably programmed processing element (such as a GPU, CPU,
microprocessor, processor, controller, computing device, etc.). In
a complex application or system such instructions are typically
arranged into "modules" with each such module typically performing
a specific task, process, function, or operation. The entire set of
modules may be controlled or coordinated in their operation by an
operating system (OS) or other form of organizational platform.
[0119] Each application module or sub-module may correspond to a
particular function, method, process, or operation that is
implemented by the module or sub-module. Such function, method,
process, or operation may include those used to implement one or
more aspects of the inventive system and methods. The application
modules and/or sub-modules may include any suitable
computer-executable code or set of instructions (e.g., as would be
executed by a suitably programmed processor, microprocessor, or
CPU), such as computer-executable code corresponding to a
programming language. For example, programming language source code
may be compiled into computer-executable code. Alternatively, or in
addition, the programming language may be an interpreted
programming language such as a scripting language.
[0120] As described, the system, apparatus, methods, processes,
functions, and/or operations for implementing an embodiment of the
disclosure may be wholly or partially implemented in the form of a
set of instructions executed by one or more programmed computer
processors such as a central processing unit (CPU) or
microprocessor. Such processors may be incorporated in an
apparatus, server, client or other computing or data processing
device operated by, or in communication with, other components of
the system. As an example, FIG. 3 is a diagram illustrating
elements or components that may be present in a system, server,
platform or computing device 300 configured to implement a method,
process, function, or operation in accordance with an embodiment of
the invention.
[0121] The subsystems shown in FIG. 3 are interconnected via a
system bus 314. Additional subsystems include may include
input/output devices 322, communications elements 324, and
additional memory or data storage devices 326. The interconnection
via the system bus 314 allows one or more processors 330 to
communicate with each subsystem and to control the execution of
instructions that may be stored in a module 302 in memory 320, as
well as the exchange of information between subsystems. The system
memory 320 and/or the memory devices 326 may embody a tangible
computer-readable medium.
[0122] Modules 302 each may contain a set of computer-executable
instructions, which when executed by a programmed processor 330
will cause system, server, platform, or device 300 to perform one
or more operations or functions. As mentioned, typically modules
302 include an operating system 303 which performs functions
involved in accessing and transferring sets of instructions so that
the instructions may be executed.
[0123] Note that the functions performed by execution of the
instructions contained in modules 302 may be the result of the
execution of a set of instructions by an electronic processing
element located in a remote server or platform, a client device, or
both. Modules 302 may include Obtain Corpus of Text 304 (which
contains instructions which when executed perform some or all of
the operations associated with block 102 of FIG. 2A), Pre-Process
Each Document/Message in Corpus Module 305 (which contains
instructions which when executed perform some or all of the
operations associated with block 104 of FIG. 2A), Generate and Rank
Possible Keyphrases Module 306 (which contains instructions which
when executed perform some or all of the operations associated with
block 106 of FIG. 2A), Generate Word or Token Embeddings from
Corpus Module 308 (which contains instructions which when executed
perform some or all of the operations associated with block 108 of
FIG. 2A), Perform Clustering on Word or Token Embeddings Module 309
(which contains instructions which when executed perform some or
all of the operations associated with block 110 of FIG. 2A), Assign
Each Document/Message in Corpus to a Cluster and "Label" Each
Cluster Module 310 (which contains instructions which when executed
perform some or all of the operations associated with building the
Intent Tree and block 112 of FIG. 2A), and Use Most Relevant
Documents in Each Cluster and Cluster Label to Train Text
Classification Model Module 312 (which contains instructions which
when executed perform some or all of the operations associated with
block 114 of FIG. 2A).
[0124] As mentioned, each module may contain a set of
computer-executable instructions. The set of instructions may be
executed by a programmed processor contained in a server, client
device, network element, system, platform or other component. A
module may contain instructions that are executed by a processor
contained in more than one of a server, client device, network
element, system, platform or other component. Thus, in some
embodiments, a plurality of electronic processors, with each being
part of a separate device, server, or system may be responsible for
executing all or a portion of the software instructions contained
in an illustrated module.
[0125] In some embodiments, the functionality and services provided
by the system and methods described herein may be made available to
multiple users by accessing an account maintained by a server or
service platform. Such a server or service platform may be termed a
form of Software-as-a-Service (SaaS). FIG. 5 is a diagram
illustrating a SaaS system in which an embodiment of the
invention/disclosure may be implemented. FIG. 6 is a diagram
illustrating elements or components of an example operating
environment in which an embodiment of the invention/disclosure may
be implemented. FIG. 7 is a diagram illustrating additional details
of the elements or components of the multi-tenant distributed
computing service platform of FIG. 6, in which an embodiment of the
invention/disclosure may be implemented.
[0126] In some embodiments, the corpus processing and text
classification model training processing system or service(s)
described herein may be implemented as micro-services, processes,
workflows or functions performed in response to the submission of a
message or service request. The micro-services, processes,
workflows or functions may be performed by a server, data
processing element, platform, or system. In some embodiments, the
services may be provided by a service platform located "in the
cloud". In such embodiments, the platform is accessible through
APIs and SDKs. The described corpus processing and text
classification model training services may be provided as
micro-services within the platform for each of multiple users or
companies, with each of those companies having a specific trained
model and application available for download to their customers
seeking assistance. The interfaces to the micro-services may be
defined by REST and GraphQL endpoints. An administrative console
may allow users or an administrator to securely access the
underlying request and response data, manage accounts and access,
and in some cases, modify the processing workflow or
configuration.
[0127] Note that although FIGS. 5-7 illustrate a multi-tenant or
SaaS architecture that may be used for the delivery of
business-related or other applications and services to multiple
accounts/users, such an architecture may also be used to deliver
other types of data processing services and provide access to other
applications. For example, such an architecture may be used to
provide corpus processing and text classification model training
services to assist end-users to resolve requests for customer
support. Although in some embodiments, a platform or system of the
type illustrated in FIGS. 5-7 may be operated by a 3.sup.rd party
provider to provide a specific set of business-related
applications, in other embodiments, the platform may be operated by
a provider and a different business may provide the applications or
services for users through the platform. For example, some of the
functions and services described with reference to FIGS. 5-7 may be
provided by a 3.sup.rd party with the provider of the trained
models and client application maintaining an account on the
platform for each company or business using a trained model to
provide services to that company's customers.
[0128] FIG. 5 is a diagram illustrating a system 500 in which an
embodiment of the invention may be implemented or through which an
embodiment of the corpus processing and text classification model
training services described herein may be accessed. In accordance
with the advantages of an application service provider (ASP) hosted
business service system (such as a multi-tenant data processing
platform), users of the services described herein may comprise
individuals, businesses, stores, organizations, etc. A user may
access the services using any suitable client, including but not
limited to desktop computers, laptop computers, tablet computers,
scanners, smartphones, etc. In general, any client device having
access to the Internet may be used to provide a request or text
message requesting customer support services and to receive and
display an intent tree model. Users interface with the service
platform across the Internet 513 or another suitable communications
network or combination of networks. Examples of suitable client
devices include desktop computers 503, smartphones 504, tablet
computers 505, or laptop computers 506.
[0129] Corpus Processing and Text Classification Model Training
system 510, which may be hosted by a third party, may include a set
of Corpus Processing and Text Classification Model Training
services 512 and a web interface server 514, coupled as shown in
FIG. 5. It is to be appreciated that either or both of the Corpus
Processing and Text Classification Model Training services 512 and
the web interface server 514 may be implemented on one or more
different hardware systems and components, even though represented
as singular units in FIG. 5. Corpus Processing and Text
Classification Model Training services 512 may include one or more
functions or operations for the processing of a received corpus of
text to generate a trained text classification model for use in
providing customer support services for a specific company or
organization.
[0130] In some embodiments, the set of applications available to a
company may include one or more that perform the functions and
methods described herein for receiving a corpus of
documents/messages from a company or organization, pre-processing
each document/message in the corpus, generating and ranking
possible keyphrases), generating word embeddings from the corpus,
performing clustering on the word embeddings, assigning each
document message in the corpus to a cluster and labeling each
cluster, and using the most relevant documents in each cluster and
the cluster label to train a text classification model for the
company or organization. As discussed, these functions or
processing workflows may be used to provide a customer with a more
efficient and effective response to their inquiry or request for
customer support.
[0131] As examples, in some embodiments, the set of applications,
functions, operations or services made available through the
platform or system 510 may include: [0132] account management
services 516, such as [0133] a process or service to authenticate a
customer, company or organization wishing to utilize the services
and functionality of the platform; [0134] a process or service to
receive a request from the company or organization to provide the
platform with a corpus of documents/messages for use in generating
a trained text classification/machine learning model to be used in
responding to customer service requests; [0135] an optional process
or service to generate a price for the requested services or a
charge against a service contract; [0136] a process or service to
generate a container or instantiation of the corpus processing and
model training processes for the requester, where the instantiation
may be customized for a particular company's workflow or processing
needs (e.g., by removing certain information from documents, using
specific formats for labels, etc.); and [0137] other forms of
account management services. [0138] a process or service to
generate a trained machine learning model for a company or business
518, such as [0139] a process or service to acquire the corpus of
documents (such as customer generated messages) from the company;
[0140] a process or service to generate keyphrases from each
document in the corpus of documents; [0141] a process or service to
generate word or token embeddings from each document in the corpus
of documents; [0142] a process or service to perform a clustering
operation on the word or token embeddings; [0143] a process or
service to assign each document in the corpus to a duster and
enable curation and labeling of each cluster; and [0144] a process
or service to train a machine learning model using the most
relevant documents in a cluster and the label assigned to that
cluster; [0145] after training of the text classification model for
a specific company, the platform may respond to a customer's
service request or launching of a client device application by a
process or service for accessing an appropriate message
classification model and providing it to the customer's client
device 520, such as [0146] a process or service to identify the
correct trained model to provide to the customer; [0147] in this
case the "correct" model is the one corresponding to the company
from whom the customer is requesting assistance (and may be
indicated by the application the customer used); [0148] a process
or service to provide the intent tree data and the appropriate
trained model to the requester's client device, along with a
supporting application (if needed and not already provided) to
enable display and interaction with an intent tree data structure;
[0149] a process or service for processing the customer selection,
model output or customer text input 522, such as [0150] a process
or service that receives an intent selected by a user or a
prediction or classification from the client application or trained
model that represents the "predicted" or most likely to be correct
intent of the requester based on text entered into a text entry
field in the user interface; [0151] a process or service that
receives and processes a partial or full text input from the
requester in the situation where the requester is unable to select
an intent from the intent tree or to determine a best fit to their
intent using the trained model; [0152] based on the determined
intent (as a result of the customer selection, operation of the
trained model or processing of the partial or full text provided by
a requester by a person or process), request routing processes or
services 524, such as [0153] processes or services that function to
route the request to the appropriate handler, such as a department,
Bot, task or workflow, application, URL, or person based on the
customer's selection, the intent determined/predicted by the model
or further processing of the customer's text input (typically by a
person acting as an evaluator); [0154] a process or service to
provide additional data to the system/platform for use in further
training and/or evaluation of possible improvements to the trained
model, 526, such as [0155] a process or service to provide the
customer's initial message (and if applicable, their more detailed
text message) and the determined intent/label to a process for
training and updating the intent tree and trained model; and [0156]
administrative services 528, such as [0157] a process or services
to enable the provider of the customer service request processing
and routing services and/or the platform to administer and
configure the processes and services provided to requesters. Note
that some of the functions or services described with reference to
FIG. 5 (such as services or functions 522 and/or 524) may be
performed in whole or in part by the application downloaded to the
customer's device.
[0158] The platform or system shown in FIG. 5 may be hosted on a
distributed computing system made up of at least one, but likely
multiple, "servers." A server is a physical computer dedicated to
providing data storage and an execution environment for one or more
software applications or services intended to serve the needs of
the users of other computers that are in data communication with
the server, for instance via a public network such as the Internet.
The server, and the services it provides, may be referred to as the
"host" and the remote computers, and the software applications
running on the remote computers being served may be referred to as
"clients." Depending on the computing service(s) that a server
offers it could be referred to as a database server, data storage
server, file server, mail server, print server, web server, etc. A
web server is a most often a combination of hardware and the
software that helps deliver content, commonly by hosting a website,
to client web browsers that access the web server via the
Internet.
[0159] FIG. 6 is a diagram illustrating elements or components of
an example operating environment 600 in which an embodiment of the
invention may be implemented. As shown, a variety of clients 602
incorporating and/or incorporated into a variety of computing
devices may communicate with a multi-tenant service platform 608
through one or more networks 614. For example, a client may
incorporate and/or be incorporated into a client application (e.g.,
software) implemented at least in part by one or more of the
computing devices. Examples of suitable computing devices include
personal computers, server computers 604, desktop computers 606,
laptop computers 607, notebook computers, tablet computers or
personal digital assistants (PDAs) 610, smart phones 612, cell
phones, and consumer electronic devices incorporating one or more
computing device components, such as one or more electronic
processors, microprocessors, central processing units (CPU), or
controllers. Examples of suitable networks 614 include networks
utilizing wired and/or wireless communication technologies and
networks operating in accordance with any suitable networking
and/or communication protocol (e.g., the Internet).
[0160] The distributed computing service/platform (which may also
be referred to as a multi-tenant data processing platform) 608 may
include multiple processing tiers, including a user interface tier
616, an application server tier 620, and a data storage tier 624.
The user interface tier 616 may maintain multiple user interfaces
617, including graphical user interfaces and/or web-based
interfaces. The user interfaces may include a default user
interface for the service to provide access to applications and
data for a user or "tenant" of the service (depicted as "Service
UI" in the figure), as well as one or more user interfaces that
have been specialized/customized in accordance with user specific
requirements (e.g., represented by "Tenant A UI", "Tenant Z UI" in
the figure, and which may be accessed via one or more APIs).
[0161] The default user interface may include user interface
components enabling a tenant to administer the tenant's access to
and use of the functions and capabilities provided by the service
platform. This may include accessing tenant data, launching an
instantiation of a specific application, causing the execution of
specific data processing operations, etc. Each application server
or processing tier 622 shown in the figure may be implemented with
a set of computers and/or components including computer servers and
processors, and may perform various functions, methods, processes,
or operations as determined by the execution of a software
application or set of instructions. The data storage tier 624 may
include one or more data stores, which may include a Service Data
store 625 and one or more Tenant Data stores 626. Data stores may
be implemented with any suitable data storage technology, including
structured query language (SQL) based relational database
management systems (RDBMS).
[0162] Service Platform 608 may be multi-tenant and may be operated
by an entity in order to provide multiple tenants with a set of
business-related or other data processing applications, data
storage, and functionality. For example, the applications and
functionality may include providing web-based access to the
functionality used by a business to provide services to end-users,
thereby allowing a user with a browser and an Internet or intranet
connection to view, enter, process, or modify certain types of
information. Such functions or applications are typically
implemented by one or more modules of software code/instructions
that are maintained on and executed by one or more servers 622 that
are part of the platform's Application Server Tier 620. As noted
with regards to FIG. 5, the platform system shown in FIG. 6 may be
hosted on a distributed computing system made up of at least one,
but typically multiple, "servers."
[0163] As mentioned, rather than build and maintain such a platform
or system themselves, a business may utilize systems provided by a
third party. A third party may implement a business system/platform
as described above in the context of a multi-tenant platform, where
individual instantiations of a business' data processing workflow
(such as the message/request processing and routing described
herein) are provided to users, with each business representing a
tenant of the platform. One advantage to such multi-tenant
platforms is the ability for each tenant to customize their
instantiation of the data processing workflow to that tenant's
specific business needs or operational methods. Each tenant may be
a business or entity that uses the multi-tenant platform to provide
business services and functionality to multiple users.
[0164] FIG. 7 is a diagram illustrating additional details of the
elements or components of the multi-tenant distributed computing
service platform of FIG. 6, in which an embodiment of the invention
may be implemented. The software architecture shown in FIG. 7
represents an example of an architecture which may be used to
implement an embodiment of the invention. In general, an embodiment
of the invention may be implemented using a set of software
instructions that are designed to be executed by a suitably
programmed processing element (such as a CPU, microprocessor,
processor, controller, computing device, etc.). In a complex system
such instructions are typically arranged into "modules" with each
such module performing a specific task, process, function, or
operation. The entire set of modules may be controlled or
coordinated in their operation by an operating system (OS) or other
form of organizational platform.
[0165] As noted, FIG. 7 is a diagram illustrating additional
details of the elements or components 700 of a multi-tenant
distributed computing service platform, in which an embodiment of
the invention may be implemented. The example architecture includes
a user interface layer or tier 702 having one or more user
interfaces 703. Examples of such user interfaces include graphical
user interfaces and application programming interfaces (APIs). Each
user interface may include one or more interface elements 704. For
example, users may interact with interface elements in order to
access functionality and/or data provided by application and/or
data storage layers of the example architecture. Examples of
graphical user interface elements include buttons, menus,
checkboxes, drop-down lists, scrollbars, sliders, spinners, text
boxes, icons, labels, progress bars, status bars, toolbars,
windows, hyperlinks and dialog boxes. Application programming
interfaces may be local or remote and may include interface
elements such as parameterized procedure calls, programmatic
objects and messaging protocols.
[0166] The application layer 710 may include one or more
application modules 711, each having one or more sub-modules 712.
Each application module 711 or sub-module 712 may correspond to a
function, method, process, or operation that is implemented by the
module or sub-module (e.g., a function or process related to
providing business related data processing and services to a user
of the platform). Such function, method, process, or operation may
include those used to implement one or more aspects of the
inventive system and methods, such as for one or more of the
processes or functions described with reference to FIGS. 1, 2A, 2B,
3 and 5: [0167] a process or service to acquire a corpus of
documents (such as customer generated messages) from a company or
organization; [0168] in some examples, the corpus may be obtained
from a source of examples of customer service requests that were
(or might be expected to be) received by the company or
organization; [0169] a process or service to generate keyphrases
from each document in the corpus of documents; [0170] a process or
service to generate word or token embeddings from each document in
the corpus of documents; [0171] a process or service to perform a
clustering operation on the word or token embeddings; [0172] a
process or service to assign each document in the corpus to a
cluster and enable curation and labeling of each cluster; [0173] a
process or service to train a machine learning model using the most
relevant documents in a cluster and the label assigned to that
cluster; [0174] a process or service to provide the trained model
and a data structure representing an "intent tree" of possible
categories of requests for assistance to the company's customers
when they request assistance, send a message to a customer support
address, or launch a customer service application (or otherwise
indicate a need for assistance); [0175] a process or service to
provide an application (if it has not already been provided) to the
company's customers to display and enable customers to "navigate"
the intent tree, to select a desired intent or category of
assistance, to confirm a selected intent, to allow a customer to
insert text and have that text classified by the trained model, and
to confirm their acceptance of the classification produced by the
trained model; [0176] where the application is configured to
generate a display or user interface including the intent tree and
user interface elements to enable the selection, text input,
confirmation, and other functions; [0177] a process or service to
allow a customer to insert text, a phrase, a question, or a
sentence (as examples) into a field of a user interface and have
the contents of the field provided to the server/platform for
further evaluation and processing (typically by a person); [0178] a
process or service to provide the customer's initial message (and
if applicable, their more detailed text message) and the determined
intent/label to a process for training and updating the intent tree
and trained model; and [0179] a process or service to provide a
mechanism (typically either through the application or a service on
the platform) to provide a customer's message or request to the
appropriate "handler", task, workflow, Bot, URL, person or
department of the company for resolution after determination of the
customer's intent or goal in requesting assistance.
[0180] The application modules and/or sub-modules may include any
suitable computer-executable code or set of instructions (e.g., as
would be executed by a suitably programmed processor,
microprocessor, or CPU), such as computer-executable code
corresponding to a programming language. For example, programming
language source code may be compiled into computer-executable code.
Alternatively, or in addition, the programming language may be an
interpreted programming language such as a scripting language. Each
application server (e.g., as represented by element 622 of FIG. 6)
may include each application module. Alternatively, different
application servers may include different sets of application
modules. Such sets may be disjoint or overlapping.
[0181] The data storage layer 720 may include one or more data
objects 722 each having one or more data object components 721,
such as attributes and/or behaviors. For example, the data objects
may correspond to tables of a relational database, and the data
object components may correspond to columns or fields of such
tables. Alternatively, or in addition, the data objects may
correspond to data records having fields and associated services.
Alternatively, or in addition, the data objects may correspond to
persistent instances of programmatic data objects, such as
structures and classes. Each data store in the data storage layer
may include each data object. Alternatively, different data stores
may include different sets of data objects. Such sets may be
disjoint or overlapping.
[0182] Note that the example computing environments depicted in
FIGS. 5-7 are not intended to be limiting examples. Further
environments in which an embodiment of the invention may be
implemented in whole or in part include devices (including mobile
devices), software applications, systems, apparatuses, networks,
SaaS platforms, IaaS (infrastructure-as-a-service) platforms, or
other configurable components that may be used by multiple users
for data entry, data processing, application execution, or data
review.
[0183] It should be understood that the present invention as
described above can be implemented in the form of control logic
using computer software in a modular or integrated manner. Based on
the disclosure and teachings provided herein, a person of ordinary
skill in the art will know and appreciate other ways and/or methods
to implement the present invention using hardware and a combination
of hardware and software.
[0184] In some embodiments, certain of the methods, models or
functions described herein may be embodied in the form of a trained
neural network, where the network is implemented by the execution
of a set of computer-executable instructions or representation of a
data structure. The instructions may be stored in (or on) a
non-transitory computer-readable medium and executed by a
programmed processor or processing element. A trained neural
network, trained machine learning model, or other form of decision
or classification process may be used to implement one or more of
the methods, functions, processes or operations described herein.
Note that a neural network or deep learning model may be
characterized in the form of a data structure in which are stored
data representing a set of layers containing nodes, and connections
between nodes in different layers are created (or formed) that
operate on an input to provide a decision or value as an
output.
[0185] In general terms, a neural network may be viewed as a system
of interconnected artificial "neurons" that exchange messages
between each other. The connections have numeric weights that are
"tuned" during a training process, so that a properly trained
network will respond correctly when presented with an image or
pattern to recognize (for example). In this characterization, the
network consists of multiple layers of feature-detecting "neurons";
each layer has neurons that respond to different combinations of
inputs from the previous layers. Training of a network is performed
using a "labeled" dataset of inputs in a wide assortment of
representative input patterns that are associated with their
intended output response. Training uses general-purpose methods to
iteratively determine the weights for intermediate and final
feature neurons. In terms of a computational model, each neuron
calculates the dot product of inputs and weights, adds the bias,
and applies a non-linear trigger or activation function (for
example, using a sigmoid response function).
[0186] Machine learning (ML) is being used more and more to enable
the analysis of data and assist in making decisions in multiple
industries. In order to benefit from using machine learning, a
machine learning algorithm is applied to a set of training data and
labels to generate a "model" which represents what the application
of the algorithm has "learned" from the training data. Each element
(or example, in the form of one or more parameters, variables,
characteristics or "features") of the set of training data is
associated with a label or annotation that defines how the element
should be classified by the trained model. A machine learning model
is a set of layers of connected neurons that operate to make a
decision (such as a classification) regarding a sample of input
data. When trained (i.e., the weights connecting neurons have
converged and become stable or within an acceptable amount of
variation), the model will operate on a new element of input data
to generate the correct label or classification as an output.
[0187] The present disclosure includes the following numbered
clauses: [0188] Clause 1. A method of providing assistance to a
customer, comprising:
[0189] creating a trained text classification model by [0190]
obtaining a set of documents; [0191] generating a set of possible
keyphrases from each document in the set of documents; [0192]
generating a set of word or token embeddings from each document in
the set of documents; [0193] performing a clustering operation to
generate a set of clusters from the set of word or token
embeddings, wherein each cluster of the set of clusters contains
one or more of the word or token embeddings in the set of word or
token embeddings; [0194] assigning each document in the set of
documents to one of the set of generated clusters; [0195] assigning
a label to each of the generated clusters; [0196] forming a set of
training data for a text classification model, wherein the training
data comprises a plurality of pairs of a document in the set of
documents and a label for the cluster to which the document is
assigned; and [0197] training the text classification model using
the set of training data;
[0198] providing the trained text classification model to the
customer;
[0199] generating a display on a device, the display showing a set
of labels with each label corresponding to one of the clusters
generated from the word or token embeddings; and
[0200] receiving a selection of one of the set of labels by the
customer, and in response providing the customer with access to an
appropriate process to provide them with assistance. [0201] Clause
2. The method of clause 1, wherein the set of documents comprises a
plurality of text messages, [0202] Clause 3. The method of clause
1, wherein generating the set of possible keyphrases further
comprises constructing a set of n-grams for each document, wherein
each n-gram is considered a possible keyphrase. [0203] Clause 4.
The method of clause 1, wherein prior to the clustering operation,
each keyphrase is ranked using term
frequency-inverse-document-frequency (tf-idf) scoring and assigning
the label to each of the generated dusters depends at least in part
on the scores associated with the keyphrases in the cluster. [0204]
Clause 5. The method of clause 1, wherein the generated display is
a hierarchy of labels. [0205] Clause 6. The method of clause 1,
wherein if a selection of one of the set of labels is not received
from the customer, then the method further comprises determining if
text entered into a text entry field by the customer corresponds to
an existing label by using the trained model to classify the
entered text. [0206] Clause 7. The method of clause 1, wherein the
text classification model is developed using the Naive Bayes
algorithm. [0207] Clause 8. The method of clause 1, further
comprising pre-processing each document in the set of documents,
wherein the pre-processing includes one or more of removing html,
URLs, numbers, and emojis, breaking the document into sentences,
breaking each sentences into words, removing single characters, and
removing stop words. [0208] Clause 9. The method of clause 6,
wherein if the text entered by the customer does not correspond to
an existing label that the customer confirms, then the method
further comprises providing the entered text or another set of text
provided by the customer to a curator, and using the entered text
or other set of text and a label provided by the curator as
training data for the text classification model. [0209] Clause 10.
The method of clause 6, wherein providing the customer with access
to an appropriate process to provide them with assistance further
comprises providing a message from the user to a person or a bot,
launching an application, or directing the user to a webpage to
provide the user with assistance, wherein the person, bot,
application, or webpage are associated with the label selected by
the customer, a classification generated by the trained model, or a
label provided by a curator. [0210] Clause 11. A system for
assisting a user to obtain a service, comprising: [0211] an
application installed on a computing device separate from a remote
server and configured to receive a trained text classification
model from the remote server and to generate a user interface for
the user to interact with the trained model; and [0212] the remote
server, wherein the remote server is configured to
[0213] create the trained text classification model by [0214]
obtaining a set of documents; [0215] generating a set of possible
keyphrases from each document in the set of documents; [0216]
generating a set of word or token embeddings from each document in
the set of documents; [0217] performing a clustering operation to
generate a set of clusters from the set of word or token
embeddings, wherein each cluster of the set of clusters contains
one or more of the word or token embeddings in the set of word or
token embeddings; [0218] assigning each document in the set of
documents to one of the set of generated clusters; [0219] assigning
a label to each of the generated clusters; [0220] forming a set of
training data for a text classification model, wherein the training
data comprises a plurality of pairs of a document in the set of
documents and a label for the cluster to which the document is
assigned; and [0221] training the text classification model using
the set of training data;
[0222] provide the trained text classification model to the
customer; [0223] wherein the application is further configured to
generate a display on the device, the display showing a set of
labels with each label corresponding to one of the dusters
generated from the word or token embeddings and the remote server
is further configured to receive a selection of one of the set of
labels by the user, and in response provide the user with access to
an appropriate process to provide them with the service. [0224]
Clause 12. The system of clause 11, wherein the set of documents
comprises a plurality of text messages. [0225] Clause 13. The
system of clause 11, wherein generating the set of possible
keyphrases further comprises constructing a set of n-grams for each
document, wherein each n-gram is considered a possible keyphrase.
[0226] Clause 14. The system of clause 11, wherein prior to the
clustering operation, each keyphrase is ranked using term
frequency-inverse-document-frequency (tf-idf) scoring and assigning
the label to each of the generated clusters depends at least in
part on the scores associated with the keyphrases in the duster.
[0227] Clause 15. The system of clause 11, wherein the generated
display is a hierarchy of labels. [0228] Clause 16. The system of
clause 11, wherein if a selection of one of the set of labels is
not received from the user, then the application is configured to
determine if text entered into a text entry field by the user
corresponds to an existing label by using the trained model to
classify the entered text. [0229] Clause 17. The system of clause
16, wherein the remote server is configured to provide the customer
with access to an appropriate process to provide them with the
service by providing a message from the user to a person or a bot,
launching an application, or directing the user to a webpage to
provide the user with assistance, wherein the person, hot,
application, or webpage are associated with the label selected by
the customer, a classification generated by the trained model, or a
label provided by a curator. [0230] Clause 18. One or more
non-transitory computer-readable media comprising a set of
computer-executable instructions that when executed by one or more
programmed electronic processors, cause the processors to provide
assistance to a customer by:
[0231] creating a trained text classification model by [0232]
obtaining a set of documents; [0233] generating a set of possible
keyphrases from each document in the set of documents; [0234]
generating a set of word or token embeddings from each document in
the set of documents; [0235] performing a clustering operation to
generate a set of clusters from the set of word or token
embeddings, wherein each cluster of the set of clusters contains
one or more of the word or token embeddings in the set of word or
token embeddings; [0236] assigning each document in the set of
documents to one of the set of generated clusters; [0237] assigning
a label to each of the generated clusters; [0238] forming a set of
training data for a text classification model, wherein the training
data comprises a plurality of pairs of a document in the set of
documents and a label for the cluster to which the document is
assigned; and [0239] training the text classification model using
the set of training data;
[0240] providing the trained text classification model to the
customer;
[0241] generating a display on a device, the display showing a set
of labels with each label corresponding to one of the clusters
generated from the word or token embeddings; and
[0242] receiving a selection of one of the set of labels by the
customer, and in response providing the customer with access to an
appropriate process to provide them with assistance. [0243] Clause
19. The one or more non-transitory computer-readable media of
clause 18, wherein if a selection of one of the set of labels is
not received from the customer, then the one or more processors are
configured to determine if text entered into a text entry field by
the customer corresponds to an existing label by using the trained
model to classify the entered text. [0244] Clause 20. The one or
more non-transitory computer-readable media of clause 18, wherein
the one or more processors are configured to provide the customer
with access to an appropriate process to provide them with
assistance by providing a message from the user to a person or a
hot, launching an application, or directing the user to a webpage
to provide the user with assistance, wherein the person, hot,
application, or webpage are associated with the label selected by
the customer, a classification generated by the trained model, or a
label provided by a curator. [0245] Clause 21. A method of
providing assistance to a customer, comprising:
[0246] receiving a message from the customer requesting assistance;
[0247] in response to receiving the message, providing a trained
text classification model to the customer, the trained model
configured to classify text input to the model, wherein the
classification represents a category corresponding to the message;
[0248] generating a display on a device of a set of categories,
with each category corresponding to a possible type of
assistance;
[0249] receiving a selection of one of the set of displayed
categories by the customer, and in response providing the customer
with access to an appropriate process to provide them with
assistance; and
[0250] if a selection of one of the set of categories is not
received from the user, then the method further comprises
determining if text entered into a text entry field by the customer
corresponds to a category by using the trained model to classify
the entered text. [0251] Clause 22. The method of clause 21,
wherein providing the customer with access to an appropriate
process to provide them with assistance further comprises routing a
message generated by the customer to a process, a bot, an
application or a person associated with the category selected by
the customer, a classification generated by the trained model, or a
category provided by a curator. [0252] Clause 23. The method of
clause 21, further comprising providing the customer with an
application configured to generate the display on the device.
[0253] Clause 24. The method of clause 21, further comprising
providing the customer with a set of data that is used to generate
the display. [0254] Clause 25. The method of clause 21, wherein the
display of the set of categories is in the form of a list or a
hierarchical display.
[0255] Any of the software components, processes or functions
described in this application may be implemented as software code
to be executed by a processor using any suitable computer language
such as Python, Java, JavaScript, C++ or Perl using conventional or
object-oriented techniques. The software code may be stored as a
series of instructions, or commands in (or on) a non-transitory
computer-readable medium, such as a random-access memory (RAM), a
read only memory (ROM), a magnetic medium such as a hard-drive or a
floppy disk, or an optical medium such as a CD-ROM. In this
context, a non-transitory computer-readable medium is almost any
medium suitable for the storage of data or an instruction set aside
from a transitory waveform. Any such computer readable medium may
reside on or within a single computational apparatus and may be
present on or within different computational apparatuses within a
system or network.
[0256] According to one example implementation, the term processing
element or processor, as used herein, may be a central processing
unit (CPU), or conceptualized as a CPU (such as a virtual machine).
In this example implementation, the CPU or a device in which the
CPU is incorporated may be coupled, connected, and/or in
communication with one or more peripheral devices, such as display.
In another example implementation, the processing element or
processor may be incorporated into a mobile computing device, such
as a smartphone or tablet computer.
[0257] The non-transitory computer-readable storage medium referred
to herein may include a number of physical drive units, such as a
redundant array of independent disks (RAID), a floppy disk drive, a
flash memory, a USB flash drive, an external hard disk drive, thumb
drive, pen drive, key drive, a High-Density Digital Versatile Disc
(HD-DV D) optical disc drive, an internal hard disk drive, a
Blu-Ray optical disc drive, or a Holographic Digital Data Storage
(HDDS) optical disc drive, synchronous dynamic random access memory
(SDRAM), or similar devices or other forms of memories based on
similar technologies. Such computer-readable storage media allow
the processing element or processor to access computer-executable
process steps, application programs and the like, stored on
removable and non-removable memory media, to off-load data from a
device or to upload data to a device. As mentioned, with regards to
the embodiments described herein, a non-transitory
computer-readable medium may include almost any structure,
technology or method apart from a transitory waveform or similar
medium.
[0258] Certain implementations of the disclosed technology are
described herein with reference to block diagrams of systems,
and/or to flowcharts or flow diagrams of functions, operations,
processes, or methods. It will be understood that one or more
blocks of the block diagrams, or one or more stages or steps of the
flowcharts or flow diagrams, and combinations of blocks in the
block diagrams and stages or steps of the flowcharts or flow
diagrams, respectively, can be implemented by computer-executable
program instructions. Note that in some embodiments, one or more of
the blocks, or stages or steps may not necessarily need to be
performed in the order presented or may not necessarily need to be
performed at all.
[0259] These computer-executable program instructions may be loaded
onto a general-purpose computer, a special purpose computer, a
processor, or other programmable data processing apparatus to
produce a specific example of a machine, such that the instructions
that are executed by the computer, processor, or other programmable
data processing apparatus create means for implementing one or more
of the functions, operations, processes, or methods described
herein. These computer program instructions may also be stored in a
computer-readable memory that can direct a computer or other
programmable data processing apparatus to function in a specific
manner, such that the instructions stored in the computer-readable
memory produce an article of manufacture including instruction
means that implement one or more of the functions, operations,
processes, or methods described herein.
[0260] While certain implementations of the disclosed technology
have been described in connection with what is presently considered
to be the most practical and various implementations, it is to be
understood that the disclosed technology is not to be limited to
the disclosed implementations. Instead, the disclosed
implementations are intended to cover various modifications and
equivalent arrangements included within the scope of the appended
claims. Although specific terms are employed herein, they are used
in a generic and descriptive sense only and not for purposes of
limitation.
[0261] This written description uses examples to disclose certain
implementations of the disclosed technology, and also to enable any
person skilled in the art to practice certain implementations of
the disclosed technology, including making and using any devices or
systems and performing any incorporated methods. The patentable
scope of certain implementations of the disclosed technology is
defined in the claims, and may include other examples that occur to
those skilled in the art. Such other examples are intended to be
within the scope of the claims if they have structural and/or
functional elements that do not differ from the literal language of
the claims, or if they include structural and/or functional
elements with insubstantial differences from the literal language
of the claims.
[0262] All references, including publications, patent applications,
and patents, cited herein are hereby incorporated by reference to
the same extent as if each reference were individually and
specifically indicated to be incorporated by reference and/or were
set forth in its entirety herein.
[0263] The use of the terms "a" and "an" and "the" and similar
referents in the specification and in the following claims are to
be construed to cover both the singular and the plural, unless
otherwise indicated herein or clearly contradicted by context. The
terms "having," "including," "containing" and similar referents in
the specification and in the following claims are to be construed
as open-ended terms (e.g., meaning "including, but not limited
to,") unless otherwise noted. Recitation of ranges of values herein
are merely indented to serve as a shorthand method of referring
individually to each separate value inclusively falling within the
range, unless otherwise indicated herein, and each separate value
is incorporated into the specification as if it were individually
recited herein. All methods described herein can be performed in
any suitable order unless otherwise indicated herein or clearly
contradicted by context. The use of any and all examples, or
exemplary language (e.g., "such as") provided herein, is intended
merely to better illuminate embodiments of the invention and does
not pose a limitation to the scope of the invention unless
otherwise claimed. No language in the specification should be
construed as indicating any non-claimed element as essential to
each embodiment of the present invention.
[0264] Different arrangements of the components depicted in the
drawings or described above, as well as components and steps not
shown or described are possible. Similarly, some features and
sub-combinations are useful and may be employed without reference
to other features and sub-combinations. Embodiments of the
invention have been described for illustrative and not restrictive
purposes, and alternative embodiments will become apparent to
readers of this patent. Accordingly, the present invention is not
limited to the embodiments described above or depicted in the
drawings, and various embodiments and modifications can be made
without departing from the scope of the claims below.
* * * * *