U.S. patent application number 17/245435 was filed with the patent office on 2021-11-04 for method for training information retrieval model based on weak-supervision and method for providing search result using such model.
The applicant listed for this patent is Hoseo University Academic Cooperation Foundation, Livin Al Inc.. Invention is credited to Geun Jin AHN, Suehyun CHANG, Sung Bum PARK.
Application Number | 20210342718 17/245435 |
Document ID | / |
Family ID | 1000005586481 |
Filed Date | 2021-11-04 |
United States Patent
Application |
20210342718 |
Kind Code |
A1 |
PARK; Sung Bum ; et
al. |
November 4, 2021 |
METHOD FOR TRAINING INFORMATION RETRIEVAL MODEL BASED ON
WEAK-SUPERVISION AND METHOD FOR PROVIDING SEARCH RESULT USING SUCH
MODEL
Abstract
A search method using an artificial intelligence based
information retrieval model and a method for training the
artificial intelligence based information retrieval model used for
the method are provided. In the method, even if there is no labeled
data and only a corpus exists, the artificial intelligence based
information retrieval model can be trained using the
weak-supervision methodology. Search can be performed by dividing
documents into passages having short lengths. Compared to an
information retrieval model based on unsupervised learning,
improved search results are provided.
Inventors: |
PARK; Sung Bum; (Yongin-si,
KR) ; CHANG; Suehyun; (Seoul, KR) ; AHN; Geun
Jin; (Bucheon-si, KR) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Hoseo University Academic Cooperation Foundation
Livin Al Inc. |
Asan-si
Cheonan-si |
|
KR
KR |
|
|
Family ID: |
1000005586481 |
Appl. No.: |
17/245435 |
Filed: |
April 30, 2021 |
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06N 5/04 20130101; G06F
16/93 20190101; G06N 20/00 20190101 |
International
Class: |
G06N 5/04 20060101
G06N005/04; G06N 20/00 20060101 G06N020/00; G06F 16/93 20060101
G06F016/93 |
Foreign Application Data
Date |
Code |
Application Number |
May 1, 2020 |
KR |
10-2020-0053018 |
Claims
1. A computer-implemented method for providing a user with a search
result corresponding to a query input by the user from a corpus
including a plurality of passages, comprising the steps of: (a)
retrieving, by an information retrieval model based on an
unsupervised learning, N passages corresponding to the query from
the corpus; (b) re-ranking, by an artificial intelligence based
information retrieval model trained by a weak-supervision
methodology, the N passages retrieved in step (a) based on the
query; and, (c) outputting a list of search results corresponding
to the passages re-ranked in step (b), wherein each passage of the
corpus is a part of each document of the document corpus including
a plurality of documents to be searched, and wherein the artificial
intelligence based information retrieval model trained by the
weak-supervision methodology is trained based on pseudo-labels
which are generated by using the plurality of passages included in
the corpus and a pseudo-query generated from each passage, said
pseudo-query generated by an information retrieval model based on
unsupervised learning including the information retrieval model
that is used in the step (a).
2. The method according to claim 1, wherein each document in the
document corpus has a title, and wherein each passage in the corpus
includes the title of the document in which the passage is included
as part of.
3. The method according to claim 1, wherein, in the step (c), the
search result list is a list in which documents including the
re-ranked passages are sorted so as to correspond to the sorting
order of the re-ranked passages in step (b).
4. The method according to claim 3, wherein, when the re-ranked
result in step (b) includes more than two passages extracted from
one document, the sorting order of the document is determined to
correspond to the order of the most relevant passage.
5. An apparatus for providing a user with a search result
corresponding to a query input by the user from a corpus including
a plurality of passages, comprising: at least one processor; and at
least one memory for storing computer-executable instructions, the
computer-executable instructions stored in the at least one memory
that are operable, when executed by the at least one processor, to
perform operations including: (a) retrieving, by an information
retrieval model based on an unsupervised learning, N passages
corresponding to the query from the corpus; (b) re-ranking, by an
artificial intelligence based information retrieval model trained
by a weak-supervision methodology, the N passages retrieved in step
(a) based on the query; and, (c) outputting a list of search
results corresponding to the passages re-ranked in step (b),
wherein each passage of the corpus is a part of each document of
the document corpus including a plurality of documents to be
searched, and wherein the artificial intelligence based information
retrieval model trained by the weak-supervision methodology is
trained based on pseudo-labels which are generated by using the
plurality of passages included in the corpus and a pseudo-query
generated from each passage, said pseudo-query generated by an
information retrieval model based on unsupervised learning
including the information retrieval model that is used in the step
(a).
6. The apparatus according to claim 5, wherein each document in the
document corpus has a title, and wherein each passage in the corpus
includes the title of the document in which the passage is included
as part of.
7. The apparatus according to claim 5, wherein, in the step (c),
the search result list is a list in which documents including the
re-ranked passages are sorted so as to correspond to the sorting
order of the re-ranked passages in step (b).
8. The apparatus according to claim 7, wherein, when the re-ranked
result in step (b) includes more than two passages extracted from
one document, the sorting order of the document is determined to
correspond to the order of the most relevant passage.
Description
BACKGROUND OF THE INVENTION
1. Field of the Invention
[0001] The present invention relates to a method for training an
information retrieval model based on the weak supervision and a
method for providing a search result using the information
retrieval model (hereinafter, "IR model") trained by this method,
and more specifically, to a method for generating pseudo-labels and
training an IR model by the pseudo-labels when labels for
supervised learning of an artificial intelligence based IR model do
not exist and also to a method for providing search results for the
query entered by the user by using the IR model.
2. Description of the Related Art
[0002] Information retrieval has been continuously developed since
Google introduced the PageRank based search technology based on
graph theory. This search technique was based on unsupervised
learning, and it was possible to search if only a documents corpus
was given. BM25 is a representative IR technology based on
unsupervised learning, and it shows very improved performance when
used with a query expansion technique called RM3. As an open
source, Anserini is widely used in the academic and the practical
fields.
[0003] In the field of natural language processing, various IR
models have been proposed according to research in the academic
field to apply artificial intelligence techniques. For example,
deep learning based IR models such as DRMM, KNRM, and PACRR have
been proposed. BERT, released by Google in 2018, has shown good
performance in various natural language processing fields, and
studies have been conducted to use BERT as an IR model based on a
transformer or a language model.
[0004] In the Ad-Hoc Information Retrieval section of Paper With
Code, a website that introduces artificial intelligence models
having open source in each field, it is possible to identify the
SOTA (State-of-the-Art) IR model, that is, the artificial
intelligence based IR model that exhibits the best performance at
the present time, including Anserini, a search model based on
unsupervised learning.
[0005] According to a researcher at the University of Waterloo,
Canada, named Jimmy Lin, the performance of deep learning IR models
before BERT, i.e., DRMM, KNRM, and PACRR, are similar or worse
compared to Anserini, a search model based on unsupervised
learning, the models proposed after BERT are said to have improved
performance over Anserini (Lin, Jimmy. "The Neural Hype, Justified!
A Recantation."). Such matters can also be checked on the leader
board of the Ad-Hoc Information Retrieval section of Paper With
Code described above. From these academic research results, it can
be seen that the accuracy of the search results can be improved by
the IR model based on artificial intelligence.
[0006] However, there are some limitations to the artificial
intelligence based IR model. In order to use an artificial
intelligence based IR model for inference, it must first be
trained, and such learning requires a large amount of labeled data.
Labeled data is basically processed and provided by humans, but it
is uneconomical because the cost for labeling is too high when
considering the amount of data required for learning. As another
problem, IR models based on unsupervised learning generally do not
matter even if the length of the document is long, but most of the
IR models based on artificial intelligence have limitations in the
length of documents that can be processed. For example, in the case
of BERT, the maximum number of tokens that can be processed is
limited to 512. Therefore, there is no problem in the case of
searching for a corpus consisting of short texts, but it is
difficult to apply in the case of searching for long documents such
as patents and papers.
PRIOR ARTS
[0007] [1]
https://paperswithcode.com/task/ad-hoc-information-retrieval [0008]
[2] MacAvaney, Sean, et al. "CEDR: Contextualized embeddings for
document ranking." Proceedings of the 42nd International ACM SIGIR
Conference on Research and Development in Information Retrieval.
2019. [0009] [3] Dai, Zhuyun, and Jamie Callan. "Deeper text
understanding for IR with contextual neural language modeling."
Proceedings of the 42nd International ACM SIGIR Conference on
Research and Development in Information Retrieval. 2019.
SUMMARY OF THE INVENTION
[0010] The present invention is to solve the above-described
problem and an object of the present invention is to provide a
method for training an artificial intelligence based IR model even
when there is no labeled data and only a corpus is given.
[0011] Another object of the present invention is to provide a
search method using an artificial intelligence based search model
capable of performing a search even for a corpus including
documents having long lengths.
[0012] According to an aspect of the present invention,
[0013] According to other aspect of the present invention,
[0014] According to another aspect of the present invention,
[0015] According to the present invention,
[0016] According to the present invention, there is provided a
method for training an artificial intelligence based IR model even
when the labeled data does not exist and only the corpus is
given.
[0017] In addition, according to the present invention, there is
provided a search method using an artificial intelligence based IR
model capable of performing a search even on a corpus including
documents having long lengths.
BRIEF DESCRIPTION OF THE DRAWINGS
[0018] FIG. 1 is a flow chart showing a method for training an
information retrieval model based on artificial intelligence
according to the present invention.
[0019] FIG. 2 is a flowchart showing an information retrieval
method using an artificial intelligence based information retrieval
model according to the present invention.
[0020] FIG. 3 is a diagram schematically showing an information
retrieval model according to an example of the present
invention.
[0021] FIG. 4 is a diagram schematically showing an apparatus for
performing an information retrieval method using an artificial
intelligence based information retrieval model according to the
present invention.
[0022] FIG. 5 is a diagram for comparing a search result according
to an embodiment of the present invention with a search result of
another model.
[0023] FIG. 6 is a view comparing a search result according to an
embodiment of the present invention with a search result of still
another model.
[0024] FIG. 7 shows a search result integrating the search results
each of which were independently performed based on queries for
each part of claim 1 of a patent, that is, a preamble, a first
component, and a second component of the claim 1.
DETAILED DESCRIPTION OF THE INVENTION
[0025] Hereinafter, exemplary embodiments of the present invention
will be described in detail with reference to the accompanying
drawings. The same or similar components are assigned the same or
similar reference numerals, and redundant descriptions thereof will
be omitted. In describing the embodiments disclosed in the present
specification, when it is considered that a detailed description of
related known technologies may obscure the subject matter of the
embodiments of the present specification, the detailed description
thereof will be omitted. The accompanying drawings are only for
making it easier to understand the embodiments disclosed in the
present specification, and the technical idea disclosed in the
present specification is not limited by the accompanying drawings.
It is to be understood as including all changes, equivalents, or
substitutions included in the spirit and scope of the present
invention.
[0026] Terms including ordinal numbers, such as first and second,
can be used to describe various components, but these terms are
used only for the purpose of distinguishing one component from
another, and the corresponding components are not limited by these
terms. Singular expressions include plural expressions, unless the
context clearly indicates otherwise.
[0027] As used herein, terms such as "comprises", "includes" or
"have" should be understood as limiting the existence of features,
steps, elements, or combinations thereof described in the
specification and it is not intended to exclude the possibility of
the presence or addition of one or more features, steps,
components, or combinations thereof.
[0028] FIG. 1 shows a method of training an artificial intelligence
based IR model according to the present invention, and FIG. 2 shows
a step of performing inference using the artificial intelligence
based IR model according to the present invention. FIG. 3
schematically shows an example of an IR model according to the
present invention. Hereinafter, the present invention will be
described with reference to these drawings.
[0029] [Training Stage of an IR Model Based on Artificial
Intelligence]
[0030] Information Retrieval Model
[0031] In this specification, the information retrieval model is
divided into `a retrieval model based on unsupervised learning` and
`a retrieval model based on artificial intelligence`. The former
refers to an information retrieval model based on statistical or
other unsupervised learning methodologies such as BM25 and QL
(Query Likelihood), and the latter is an information retrieval
model prepared by training, including deep learning series such as
DRMM, KNRM, and PACRR, and language model series such as BERT
etc.
[0032] In the present invention, the information retrieval model
based on the former is used to prepare a pseudo-label in the
training stage and is used to retrieve passages from the corpus in
the inference stage. The latter is used to re-rank the retrieved
passages in the inference stage.
[0033] Information retrieval models with open sources are disclosed
in the Ad-Hoc Information Retrieval section of Paper With Code.
[0034] IR Model Based on Unsupervised Learning
[0035] As an IR model based on an unsupervised learning, BM25 and
QL (Query Likelihood) are known, and Anserini is widely used as an
open source package. Anserini is an IR model that uses RM3, a query
expansion methodology, along with BM25. According to the results of
the study, the performance of Anserini is similar to that of deep
learning IR models such as DRMM, KNRM, and PACRR, but it is known
to be inferior to the IR models based on language models such as
BERT.
[0036] The IR model based on the unsupervised learning does
determines the similarity between the query and the document based
on statistical theories or other methodology.
[0037] Artificial Intelligence Based IR Model
[0038] As artificial intelligence based IR models, IR models based
on deep learning such as DRMM, KNRM, and PACRR and IR models based
on a transformer or language model such as BERT are known.
According to Paper With Code, as of the filing date of this
application, the SOTA (State-of-the-Art) IR model is the CEDR,
which has a structure that combines the BERT based on a language
model and an IR model based on deep learning.
[0039] The artificial intelligence based IR model is completed by
training the similarity between a query and a document based on the
training data. In academic fields, AI-based IR models are trained
using datasets such as TREC, SQuAD, and MS Marco that provide
query-document relationships.
[0040] Language model IR engines such as BERT use self-supervision
learning in the pre-training stage but, in the fine-tuning stage
for information retrieval, supervised learning based on the dataset
of the query-document relationship is required like the AI-based IR
model.
[0041] Weak Supervision Methodology
[0042] Query-document relationship datasets such as TREC, SQuAD,
and MS Marco, which are used in the academic field, correspond to
ground truths extracted based on actual user search activity, or
using the title of a document as a query. However, in the practical
field, only document corpus exists, and there is no query input by
the user to search for the document. Or, even if the query exists,
it is very insufficient to train an artificial intelligence based
IR model. In the present invention, when only a document corpus
exists, a pseudo-query is generated from each document, a pseudo
query-document relationship is formed using the generated
pseudo-query, and a pseudo-label is generated. In addition, a
methodology for training an artificial intelligence based IR model
using the pseudo-label is provided. The weak supervision
methodology for an artificial intelligence based IR model according
to the present invention includes the following steps: 1)
generating a pseudo-query from each document in a document corpus,
2) using the pseudo-query, creating a pseudo query-document
relationship and generating a pseudo-label based on the generated
pseudo query-document relationship, 3) training an artificial
intelligence based search model using the generated pseudo-label.
Hereinafter, each of the above-described steps will be described in
details.
[0043] 1) Step of Creating a Pseudo-Query
[0044] One or more keywords are extracted from each document in the
document corpus, and this is determined as a pseudo-query.
Previously known keyword extraction technique is used for
extracting keywords from documents. Keyword extraction algorithms
can be largely divided into techniques based on unsupervised
learning and techniques based on supervised learning, and there are
open sources for a number of techniques.
[0045] Most of the techniques are methods of extracting a plurality
of keywords or phrases from a document, but recently, a technique
called Doc2Query, which creates a natural language sentence-type
query using BERT as a supervised learning based technique, has also
been released as open source.
[0046] 2) Step of Creating a Pseudo-Label
[0047] The pseudo-query extracted from each document is used as an
input of a search model based on an unsupervised learning
methodology such as BM25, and M documents are retrieved from the
document corpus. In this case, the document from which the
pseudo-query is extracted is likely to be included in the upper
rank among the M documents, but may not be included.
[0048] Among the retrieved M documents, the top m (<M) documents
are labeled as positive training data, and at least some of the
remaining M documents are labeled as negative training data.
Depending on the AI-based search model, additional parts can be
labeled as neutral training data. In general, positive data and
negative data are necessary, but neutral data is not.
[0049] Here, the number of documents to be retrieved M, the number
of positive training data m, the number of negative training data,
and the number of neutral training data are integers, and may be
determined differently according to the characteristics of the
document corpus as a kind of hyper parameter. Developers can
increase the accuracy of artificial intelligence based IR models by
adjusting these hyperparameters in an experimental or theoretical
approach according to the characteristics of the document
corpus.
[0050] 3) Step of Training an Artificial Intelligence Based IR
Model
[0051] Since a query-document relationship is required to train an
artificial intelligence based IR model, the training data is
provided in the form of a relationship between a query used for
retrieving and a document retrieved by the query. For example, it
may be given in the form of a pair of two data such as
(pseudo-query, positive document), or three data may be provided in
a related form such as (pseudo-query, positive document, negative
document). Of course, up to a neutral document can also be
associated with four pieces of data.
[0052] The amount of data required for training is not known
exactly, and is mainly confirmed by experimental methods. However,
the amount of data required for training increases according to the
number of parameters included in the model. BERT type language
model based IR models generally have pre-learned models based on,
i.e., Wikipedia data, and a small amount of data is required for
fine-tuning pre-trained models according to specific tasks such as
information retrieval compared with pre-learning.
[0053] In academic research, the amount of training data is
determined according to the available dataset, but in the practical
field, the training data can be determined according to the amount
of documents included in the document corpus. If the amount of
documents included in the document corpus is insufficient for
training, another document of a similar field can be added to the
document corpus, or the amount of data can be increased by using a
data augmentation technique.
[0054] In the case of training an artificial intelligence based
model, it is generally desirable to validate the performance by
dividing the data into training data and validation data. For
example, 80% of the data can be used for training, and the
remaining 20% of the data can be used for validation.
[0055] [Inference Stage]
[0056] In the present invention, the inference step refers to a
process of providing a search result in response to a query input
by a user. The inference step can be broadly divided into the
following steps: a retrieving step, a re-ranking step, and an
output step. Hereinafter, each step will be described in
detail.
[0057] Retrieving Step
[0058] This is a step of extracting N documents from a document
corpus to be searched by an IR model based on an unsupervised
learning. In general, the IR model based on artificial intelligence
has a considerably longer time required for inference compared to
the IR model based on the unsupervised learning. Therefore, when an
artificial intelligence based IR model is applied to all documents
included in the document corpus, it takes an excessively long time,
and the user's convenience is degraded. Therefore, it is common to
retrieve a number of documents by an IR model based on an
unsupervised learning which is relatively fast first, and then an
artificial intelligence based IR model is used only to the
retrieved documents. Research to skip the retrieving step by
improving the inference speed of the artificial intelligence based
IR model is continuously being conducted, but it is understood that
it has not yet reached a sufficiently fast inference speed in
consideration of user convenience.
[0059] At this step, although accuracy is important, recall is more
important. On the other hand, in the second re-ranking step,
accuracy is more important than recall. One of the ways to increase
the recall is to retrieve the number of documents several times
larger than the number of documents to be provided in the search
result. Another solution is to increase the recall of the IR model
based on the unsupervised learning. By using a technique called
DeepCT Index based on BERT, it is possible to increase the recall
while using an IR model based on the same unsupervised learning.
The time required for retrieving increases in proportion to the
number of documents. Using the DeepCT Index, the time required for
retrieval can be shortened by reducing the number of documents to
be retrieved and the same level of recall as the case of retrieving
several times larger number of documents can be realized.
[0060] As an IR model based on the unsupervised learning, the same
IR model as used in the training step may be used, or a different
IR model may be used. It is important to select an IR model with a
high recall in the field. In academic research, Anserini (BM25+RM3)
is widely used as an open source based IR model.
[0061] Re-Ranking Step
[0062] The relevance between the retrieved N documents and the
query entered by the user is evaluated by an IR model based on the
unsupervised learning and arranged according to the degree of
relevance. In this step, the relationship between the retrieved N
documents and the query entered by the user is re-evaluated by the
artificial intelligence based IR model, and then re-ordered, that
is, re-ranked. This step is performed by the artificial
intelligence based IR model trained in the above-described learning
step.
[0063] Search Results Output Step
[0064] The N documents re-ranked by the artificial intelligence
based IR model are sorted according to the order of the degree of
relevance and output as a search result.
[0065] [Regarding Special Cases]
[0066] Regarding the Length of the Document
[0067] In General, an IR Model Based on Unsupervised Learning does
not have any problem in execution even if the length of a document
is long, but the IR model based on artificial intelligence often
has a limit on the maximum length of a document that can be
processed. In particular, AI-based search models introduced after
BERT are limited in the number of maximum tokens that can be
processed. For example, in the case of BERT, the maximum number of
tokens that can be processed is limited to 512.
[0068] When there is a limit on the maximum length of documents
that can be processed by the artificial intelligence based IR
model, each document in the document corpus can be divided into
passages less than the limited length and used as a search target.
In this specification, a corpus of passages made of short texts is
expressed as simply a `corpus` in order to distinguish it from a
`document corpus` including long documents.
[0069] In the training stage, it is the same as described above,
except that the passages included in the corpus replace documents.
That is, a pseudo-query is extracted from a passage, a pseudo-label
is generated using an IR model based on an unsupervised learning,
and then an artificial intelligence based IR model is trained. The
artificial intelligence based IR model is trained based on the
relationship between the pseudo-query and the passage, but not on
the relationship between the pseudo-query and the entire
document.
[0070] In the inference stage, it is necessary to pre-process each
document of the document corpus into a corpus by dividing it into
passages, and storing the relationship between the passage and the
document in a form that can be referenced. In the retrieving step,
N passages are retrieved from the corpus based on a query input by
the user, and in the re-ranking step, the retrieved N passages are
reranked.
[0071] In the search results, documents other than passages must be
provided in the order of relevance. To this end, documents are
sorted so as to correspond to the sorting order of the re-ranked
passages by referring to the relationship between the passage and
the document, and provided as a search result. Since multiple
passages are separated from one document, a plurality of passages
extracted from one document may be included in the reranked result.
In this case, the sorting order of a document may correspond to the
order of the most relevant passage among said a plurality of
passages.
[0072] Document Expansion
[0073] In the case of a long document, it contains a number of
paragraphs, but the content of each paragraph does not necessarily
match the subject matter of the document. Therefore, when each
paragraph is divided into passages and stored as a corpus, there is
a possibility that the degree of relevance is evaluated low even
though the paragraph is separated from the relevant document.
[0074] To prevent this problem, the title of the document or
equivalent text can be added to each passage separated from the
document. In other words, the same phrase is added to each separate
passage in a document. In this specification, the addition of a
phrase having a relevance to the passage is expressed as a term
"document extension".
[0075] The phrase corresponding to the title of a document can be
generated using, for example, a text summarization technique.
Document summarization techniques are largely divided into
extractive summarization and abstract summarization, and both can
be applied. Techniques for generating additional phrases for
document expansion are not necessarily limited to the text
summarization technique, and any technique capable of compressively
expressing the subject of the document may be used.
[0076] For example, a query entered by a user includes a plurality
of keywords, some of which are included in the original passage,
but other keywords may be included in a phrase added by document
expansion. Since the phrase added by document expansion includes
the subject of the entire document, there is a possibility that the
message contains the content intended to be searched by the user. A
passage that would not have been retrieved if the phrase was not
added by document expansion can be retrieved by applying document
expansion. This can increase the recall of the retrieving step. As
described above, it is important to have a high recall in the
retrieving step.
[0077] If the length of the phrase added to the passage by document
expansion is too long, the length of the passage that is increased
by document expansion may exceed the length that can be processed
by the artificial intelligence based IR model. In order to minimize
the side effects resulting from this, it is desirable that the
phrase added by document expansion be placed in the front part of
the passage. This is because even if the trailing end of a passage
that has been lengthened by document expansion is cut off, if the
keyword included in the query entered by the user is not included
in the trailing end, the passage will still be extracted in the
retrieving step.
[0078] Domain Adaptiveness
[0079] When an artificial intelligence based IR model is trained
using documents of a specific domain, it can be used as an IR model
specialized for that domain. For example, in the case of
pre-training BERT first by a universal document corpus such as
Wikipedia and then fine-tuning it by documents of a specific
domain, general relationship between the vocabularies is trained in
the pre-learning stage, and the vocabularies specialized for the
domain is trained in fine tuning stage.
[0080] An IR model adapted for a specific domain may have
relatively poor performance for other domains but have improved
performance for that domain.
[0081] Ensemble Model
[0082] In the above method, the results re-ranked by the artificial
intelligence based IR model were used as the final search results.
However, the search results may be provided using an ensemble of an
IR model based on unsupervised learning and an IR model based on
artificial intelligence. In this case, the final evaluation may be
expressed as in Equation 1.
(Final evaluation)=a*(Evaluation of IR model based on unsupervised
learning)+(1-a)*(Evaluation of artificial intelligence based IR
model) [Equation 1]
[0083] In Equation 1, a is a value between 0 and 1, and may be
adjusted to provide the best search result as a hyper
parameter.
[0084] The ensemble model does not necessarily guarantee that the
performance is improved compared to the single model, and it can be
considered whether or not it is adopted depending on the
domain.
[0085] FIG. 4 shows a computer device for performing the search
method according to the present invention.
[0086] Since the search method and the training method according to
the present invention have been described in detail with reference
to FIGS. 1 to 3, an apparatus 100 for performing such a search
method will be briefly described with reference to FIG. 4.
[0087] Referring to FIG. 4, the computer device 100 includes a
processor 110, a nonvolatile storage unit 120 for storing programs
and data, a volatile memory 130 for storing programs being
executed, and an input/output unit 140 for inputting and outputting
information between a user, and a bus which is an internal
communication path between these units. As a running program, there
may be an operating system and various applications. Although not
shown, it includes a power supply unit.
[0088] In the training stage, an artificial intelligence based IR
model is trained in the memory 130 by using the training data
stored in the storage unit 120. In the inference stage, the
unsupervised learning based IR model 210 and the artificial
intelligence based IR model 220 stored in the storage unit 120 are
executed in the memory 130. The corpus is stored in the storage
unit 120 and the search method is performed based on a query input
through the input/output unit 140.
Example
[0089] Hereinafter, an embodiment in which the method for providing
a search result according to the present invention is applied to
the patent documents owned by the Korea Institute of Industrial
Technology will be described.
[0090] Artificial Intelligence Based IR Model
[0091] As an artificial intelligence based IR model, the open
source CEDR-KNRM model was used. In brief, the CEDR-KNRM model has
an architecture that processes BERT and KNRM in parallel, and
details can be checked from related papers and open source codes.
According to the Paper With Code, it was confirmed that the model
had the best performance at the time of implementing the
embodiment, and there is no change until the filing date of this
application.
[0092] Corpus
[0093] The number of patent documents to be searched was
approximately 3,000, and since the length of all documents exceeded
512 tokens that could be processed by BERT, a corpus was formed by
dividing each document into paragraphs. The corpus contains
approximately 120,000 passages.
[0094] The relationship between each passage and the patent
document from which said each passage was extracted is stored so
that it can be referred to at a later stage of inference.
[0095] Pseudo-Query Generation
[0096] At least one keyword or phrase was extracted as a
pseudo-query by applying the keyword extraction technique to the
passage formed by each paragraph of the patent document. The
technique used here is RAKE whose paper and source codes are open
to the public.
[0097] If same pseudo-queries was removed, approximately 40,000
pseudo-queries were generated.
[0098] Query-Document Relationship Generation
[0099] Each pseudo-query was input and 300 passages were extracted
from the corpus using BM25, an IR model based on unsupervised
learning. Since there were approximately 40,000 pseudo-queries, the
total number of extracted passages was 12,000,000. This may be a
sufficient amount of data to train the CEDR-KNRM model.
[0100] Pseudo-Labeling
[0101] Among the 300 passages extracted by each pseudo-query, the
upper m passages were classified as positive training data, and the
lower p passages were classified as negative training data. Since
the CEDR-KNRM model also uses neutral training data, the remaining
passages excluding the upper m passages and the lower p passages
were classified as neutral training data.
[0102] Hyperparameters such as m and p were adjusted while
verifying the performance of the IR model after training by the
inventor reviewing the search results. Since it is a dataset
without ground truth, some of the prepared training data can be
used as verification data, but since it is a domain that can be
verified by the inventor, the performance was verified
manually.
[0103] Inference Stage
[0104] When a user inputs a query composed of a plurality of
keywords, 300 passages are first retrieved from the corpus using
the BM25. The retrieved 300 passages are then provided to the
trained CEDR-KNRM model along with a query input by the user. The
CEDR-KNRM model reranks 300 passages. The order of the patent
documents is arranged to correspond to the order of the passages by
inquiring the patent documents corresponding to the 300 re-ranked
passages. In the case of a plurality of passages which belong to a
same patent document are retrieved, the corresponding patent
document are sorted according to the order of the highest-ranked
passage, and the subsequent ones are ignored when sorting the
corresponding patent document. Therefore, the search results are
less than or equal to 300.
[0105] Compare Search Results
[0106] In FIGS. 5 and 6, the results by the search method according
to the present embodiment and the results by another search method
are described in comparison.
[0107] FIG. 5 is a search result among patents held by the Korea
Institute of Industrial Technology by entering a query including
two Korean keywords "(wheelchair)" and "(walking aid)". The left is
the search result by the search method according to this embodiment
and the right is the search result on KIPRIS (Korea Intellectual
Property Rights Information Service). As shown, in the search
method according to the present embodiment, documents including
words that exactly match the keywords "(wheelchair)" and "(walking
aid)" as well as other devices that can be used by people with
disabilities. While patents are searched in KIPRIS, only documents
containing words that exactly match the keywords "(wheelchair)" and
"(walking aid)" are searched.
[0108] FIG. 6 is a search result among patents held by the Korea
Institute of Industrial Technology by entering a query including
six Korean keywords such as "(scanning sonar)", "(safety
monitoring)", "(marine operation equipment)", "(sea monitoring)",
and "(aquaculture monitoring)". The upper side is the search result
by the search method according to the present embodiment, and the
lower side is the search result by Google Patent Search. Since, in
the case of the Google Patent Search, the search was not performed
when the number of keywords included in the query exceeded five
(5), the search was performed excluding the keyword "(aquaculture
monitoring)". As shown, the search method according to the present
embodiment searches for documents containing words that are
semantically similar even though they do not exactly match the
keywords included in the query, whereas the Google Patent Search
provided no search result. These results are the same for KIPRIS
(not shown).
[0109] As can be seen from FIGS. 5 and 6, the search method
according to the present invention using an artificial intelligence
based IR model does not include keywords included in the input
query, but can additionally search documents that are semantically
related.
[0110] [In Case the Concept to be Searched is Spread Across
Multiple Paragraphs]
[0111] In the above, when the length of the document is long, a
method for dividing a document into multiple passages such as
paragraphs and targeting them for search has been described. This
method is effective when the target to be searched is concentrated
on one passage in a long document, but may not be effective when
the target to be searched is distributed over several parts of the
document. For example, in the case of a patent, there is a
difference in the content to be described in the prior art, the
detailed description of the invention, and the claims, and in the
detailed description of the invention, paragraphs are often divided
for each component of the corresponding invention. In the case of
claims, the contents are divided into independent claims and
dependent claims. In the case of papers in the field of artificial
intelligence, it is divided into introduction, background, dataset,
model, experiment, discussions and conclusion. In some cases, the
search results may not be valid.
[0112] In this case, a plurality of concepts to be searched may be
searched as independent queries, and then a valid search result may
be obtained by integrating each of the search results.
[0113] FIG. 7 shows, in relation to Korean Patent Application
2010-0028166, after independently performing a search based on a
query for each part of claim 1, that is, a preamble, a first
component, and a second component. The result of integrating each
search result performed is shown. The result of simply adding all
the sorting order of each search result, the evaluation result of
the artificial intelligence search model (AI score), the evaluation
result of the search model based on unsupervised learning (BM25
score), and the evaluation result of the ensemble model. As shown,
the search target Korean Patent Application No. 2010-0028166
appears fourth in order, the AI score is 5th, the BM25 score is
1st, and the ensemble is 4th. Although the contents of other
patents in the search result list have not been reviewed, it is
considered that the contents would be similar to claim 1 of Korean
Patent Application 2010-0028166.
[0114] The above detailed description should not be construed as
limiting in any respect, but should be considered as illustrative.
The scope of the present invention should be determined by rational
interpretation of the appended claims, and all changes within the
equivalent scope of the present invention are included in the scope
of the present invention.
* * * * *
References