U.S. patent application number 17/018027 was filed with the patent office on 2021-01-07 for standardized form recognition method, associated computer program product, processing and learning systems.
The applicant listed for this patent is Ivalua SAS. Invention is credited to Christopher BOUREZ, Aurelien COQUARD.
Application Number | 20210004578 17/018027 |
Document ID | / |
Family ID | |
Filed Date | 2021-01-07 |
![](/patent/app/20210004578/US20210004578A1-20210107-D00000.png)
![](/patent/app/20210004578/US20210004578A1-20210107-D00001.png)
![](/patent/app/20210004578/US20210004578A1-20210107-D00002.png)
![](/patent/app/20210004578/US20210004578A1-20210107-D00003.png)
![](/patent/app/20210004578/US20210004578A1-20210107-D00004.png)
![](/patent/app/20210004578/US20210004578A1-20210107-D00005.png)
![](/patent/app/20210004578/US20210004578A1-20210107-M00001.png)
![](/patent/app/20210004578/US20210004578A1-20210107-M00002.png)
![](/patent/app/20210004578/US20210004578A1-20210107-M00003.png)
United States Patent
Application |
20210004578 |
Kind Code |
A1 |
COQUARD; Aurelien ; et
al. |
January 7, 2021 |
STANDARDIZED FORM RECOGNITION METHOD, ASSOCIATED COMPUTER PROGRAM
PRODUCT, PROCESSING AND LEARNING SYSTEMS
Abstract
The present invention concerns a method for transforming an
unstructured set of data representing a standardized form to a
structured set of data. The method comprises a processing phase
comprising the steps of determining a plurality of data blocks in
the unstructured set of data using learning parameters determined
using a plurality of samples, each data block corresponding to a
visual pattern on the standardized form and being categorized to a
known class, of processing data in each data block and of forming a
structured set of data using the processed data from each data
block, according to the class of this block.
Inventors: |
COQUARD; Aurelien; (Orsay,
FR) ; BOUREZ; Christopher; (Orsay, FR) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Ivalua SAS |
Orsay |
|
FR |
|
|
Appl. No.: |
17/018027 |
Filed: |
September 11, 2020 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
PCT/EP2019/056330 |
Mar 13, 2019 |
|
|
|
17018027 |
|
|
|
|
Current U.S.
Class: |
1/1 |
International
Class: |
G06K 9/00 20060101
G06K009/00; G06K 9/62 20060101 G06K009/62; G06Q 10/10 20060101
G06Q010/10; G06Q 40/00 20060101 G06Q040/00; G06F 16/25 20060101
G06F016/25; G06N 3/08 20060101 G06N003/08; G06N 3/04 20060101
G06N003/04 |
Foreign Application Data
Date |
Code |
Application Number |
Mar 13, 2018 |
EP |
18305268.7 |
Claims
1. Method for transforming an unstructured set of data representing
an invoice to a structured set of data; the method comprising a
processing phase; the processing phase comprising the following
steps: determining a plurality of data blocks in the unstructured
set of data using machine learning parameters determined using
visual patterns on the invoice and being categorized to a known
class; processing data in each data block; and forming a structured
set of data using the processed data from each data block,
according to the class of this block.
2. Method according to claim 1, wherein the machine learning
parameters were further determined using a convolutional neural
network configured to detect segmentations.
3. Method according to claim 1, wherein the step of processing data
from each data block comprises a sub-step of optical recognition of
characters comprised in this block.
4. Method according to claim 1, wherein the step of determining a
plurality of data blocks in the unstructured set of data comprises
the following sub-steps: forming an input layer using the
unstructured set of data; propagating the input layer through at
least one hidden layer; and forming an output layer using data
obtained after propagation through the last hidden layer, the
output layer being composed with a plurality of data blocks; each
layer comprising a plurality of neurons processing at least some
items of the corresponding data; at least some neurons of each
layer being connected to at least some neurons of its neighboring
layer(s) by a propagation function depending on a plurality of
weights; the weight being defined by the machine learning
parameters.
5. Method according to claim 4, wherein the sub-step of propagating
the input layer comprises propagating the input layer through at
least two hidden consecutive convolutional layers.
6. Method according to claim 4, wherein the sub-step of propagating
the input layer comprises propagating the input layer through at
least three consecutive groups of hidden layers, called
respectively early layers group, middle layers group and last
layers group; the neurons of the layers of the early layers group
being configured to detect characters, lines and textures; the
neurons of the layers of the middle layers group being configured
to detect groups of visual attributes and higher level semantics;
and the neurons of the layers of the last layers group being
configured to detect segmentations and to link elements detected by
the neurons of layers of the early layers group or of the middle
layers group.
7. Method according to claim 1, wherein the visual patterns on the
invoice being recognized using a plurality of samples; each sample
corresponds to an annotated invoice in which each visual pattern is
annotated as a data block, called annotated data block, and is
categorized to a known class.
8. Method according to claim 7, further comprising a learning phase
comprising the following steps: processing the plurality of
samples; and generating the machine learning parameters using the
samples.
9. Method according to claim 8, wherein the step of processing
(110) the plurality of samples comprises the following sub-steps,
implemented for each sample: forming an input layer using the
sample; propagating the input layer through at least one hidden
layer; and forming an output layer using data obtained after
propagation through the last hidden layer, the output layer being
composed with a plurality of test data blocks; each layer
comprising a plurality of neurons processing at least some items of
the corresponding data; and at least some neurons of each layer
being connected to at least some neurons of its neighboring
layer(s) by a propagation function depending on a current set of
weights.
10. Method according to claim 9, wherein the step of processing a
plurality of samples comprises further the following sub-steps,
implemented for each sample: determining a cost function
representing differences between the test data blocks and the
corresponding annotated data blocks; and determining a new set of
weights by optimizing the cost function.
11. Method according to claim 1, wherein the unstructured set of
data corresponds to an image or PDF file.
12. Method according to claim 1, wherein each class of data block
is chosen from the group comprising: title; supplier
identification; supplier logo; client identification; bank
coordinates; items; invoice data; total without taxes; taxes; total
with taxes; tax rate; remaining; payment terms; period; stamp;
currency; barcode; useful information.
13. Method according to claim 1, wherein each data block comprises
linked data items.
14. A computer memory comprising software instructions which, when
executed by a processor, implement a method according to claim
1.
15. Processing system configured to implement the processing phase
of the method according to claim 1.
16. Learning system configured to implement the learning phase of
the method according to claim 8.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation of International
Application No. PCT/EP2019/056330, filed Mar. 13, 2019, which
claims priority to EP Application No. 18305268.7, filed Mar. 13,
2018, under 35 U.S.C. .sctn. 119(a). Each of the above-referenced
patent applications is incorporated by reference in its
entirety.
BACKGROUND OF THE INVENTION
Field of the Invention
[0002] The present invention concerns a standardized form
recognition method. Particularly, it concerns a method for
transforming an unstructured set of data representing a
standardized form to a structured set of data.
[0003] The present invention concerns also a computer program
product, a processing system and a learning system, associated to
this method.
Description of the Related Technology
[0004] In the meaning of the invention, an unstructured set of data
is understood as computer data which can be visualized by a user
for example on a screen of an electronic device or printed from it,
but cannot be read and processed directly by such an electronic
device. An example of an unstructured set of data is an image file
of JPG type or a PDF file.
[0005] Moreover, in the meaning of the invention, a standardized
form is understood as a document having at least some parts which
are filled according to some predetermined or standardized formats
and eventually, some graphic elements. Thus, a supplier invoice, an
administrative form, a curriculum vitae and a specific type of
contract are some examples of such a standardized form.
[0006] In this context, an unstructured set of data representing a
standardized form corresponds for example to an image representing
a scanned version of a supplier invoice.
[0007] Finally, in the meaning of the invention, a structured set
of data associated to a standardized form, corresponds to an
electronic version of this form. Data contained in a structured set
is readable and usable directly by a computer system. A structured
set of data presents for example a computer file of cXML, xCBL, CSV
or XML type.
[0008] FIG. 1 illustrates an example of a standardized form which
is a supplier invoice in this example.
[0009] Thus, as it is known in the art and as it is illustrated on
this figure, a supplier invoice comprises a title 1, supplier
identification (such as supplier's name 2, address 3, bank
coordinates 4), recipient or client identification 5 (such as
client's name and address), service and product fields 6 presenting
items of the invoice, payable amounts 7 (such as prices, taxes,
total amount), payment terms 8 and other useful information 9.
These types of data are usually common for all invoices. Some
additional data may be presented in the invoice and may depend on
the supplier.
[0010] Upon receipt of an invoice for example in a paper or scanned
version, i.e. in a form of an unstructured set of data, a user has
to enter all data contained in this invoice, into a computer system
in order to generate the corresponding payment order.
[0011] Thus, according to a known process of manual invoice
filling, upon receipt of the invoice illustrated for example on
FIG. 1, the user has to create firstly an invoice form in the
corresponding computer system. Then, the user has to fill the
fields of this form corresponding to the supplier's name and the
invoice number and date. In the example of FIG. 1, this data is
situated in the left upper corner of the invoice.
[0012] Then, the user has to fill the invoice total amount and the
tax amount or the applicable tax rate. In the example of FIG. 1,
this data is referenced by reference "7".
[0013] If more than one tax rate are applicable, the user can
create multiple tax lines and enter manually the amount that
corresponds to each tax rate.
[0014] Then, if necessary, the user has to enter all of the items
mentioned in the "Description" field as well as the corresponding
amounts. Particularly, in the example of FIG. 1, the items "Product
1", "Product 2" and "Service 1" referenced by reference "6" have to
be entered into the corresponding fields of the invoice form.
[0015] Then, if the supplier is not yet registered in the
corresponding computer system, the user has to enter its bank
coordinates which are referenced by reference "4" in the example of
FIG. 1.
[0016] Finally, the user has to check all of the fields and if the
filling is correct and no modification is needed, proceed to
generation of a payment order.
[0017] The complexity of this process can vary according to
computer system's interface used by the user to enter the data but
one can conceive that it presents always a relatively long and
laborious process. The task may be even more complex for other
types of standardized forms.
[0018] Thus, numerous methods in the art propose already at least
semi-automatic processing of the unstructured sets of data
representing standardized forms.
[0019] The most common method consists in using of optical
character recognition routines, known as OCR routines.
[0020] The OCR routines make it possible to recognize the
characters in an image or PDF file so as these characters can be
captured and inserted manually into the corresponding fields of the
computer system.
[0021] Some methods in the art propose further to analyze the
captured characters in order to compose words, phrases or linked
numbers and to fill automatically the corresponding fields. In this
case, usually, the user has to validate the filled data.
[0022] Some more sophisticated methods propose to compare the
analyzed characters with data bases containing verified
information. Such data bases are relative for example to each
supplier.
[0023] However, the existing methods are not completely
satisfying.
[0024] In particular, using the OCR routines and basic
automatically filling methods does not simplify considerably the
user's task of manual control. Using some advanced methods
providing for example a database for each supplier, implies
creation of such data bases which could be also a long and
laborious process. Moreover, the supplier's data can change
regularly or a new supplier can appear so as regular updating of
the corresponding databases is usually needed.
SUMMARY
[0025] The present invention aims to resolve these issues and to
propose an efficient method for transforming an unstructured set of
data representing a standardized form to a structured set of data,
which does not depend on the issuing part. In other words, the
present invention aims to propose a method for capturing data from
an unstructured set of data.
[0026] For this purpose, the present invention proposes a method
for transforming an unstructured set of data representing a
standardized form to a structured set of data; the method
comprising a processing phase; the processing phase comprising the
following steps: determining a plurality of data blocks in the
unstructured set of data using learning parameters determined using
a plurality of samples, each data block corresponding to a visual
pattern on the standardized form and being categorized to a known
class; processing data in each data block; and forming a structured
set of data using the processed data from each data block,
according to the class of this block.
[0027] In other embodiments, the method may comprise one or several
of the following optional features, taken individually or in any
technically feasible combination: [0028] wherein the step of
processing data from each data block comprises a sub-step of
optical recognition of characters comprised in this block; [0029]
the step of determining a plurality of data blocks in the
unstructured set of data comprises the following sub-steps: [0030]
forming an input layer using the unstructured set of data; [0031]
propagating the input layer through at least one hidden layer;
[0032] forming an output layer using data obtained after
propagation through the last hidden layer, the output layer being
composed with a plurality of data blocks; each layer comprising a
plurality of neurons processing at least some items of the
corresponding data; [0033] at least some neurons of each layer
being connected to at least some neurons of its neighboring
layer(s) by a propagation function depending on a plurality of
weights; [0034] the weight being defined by the learning
parameters; [0035] the sub-step of propagating the input layer
comprises propagating the input layer through at least two hidden
consecutive convolutional layers; [0036] the sub-step of
propagating the input layer comprises propagating the input layer
through at least three consecutive groups of hidden layers, called
respectively early layers group, middle layers group and last
layers group; [0037] the neurons of the layers of the early layers
group being configured to detect characters, lines and textures;
[0038] the neurons of the layers of the middle layers group being
configured to detect groups of visual attributes and higher level
semantics; the neurons of the layers of the last layers group being
configured to detect segmentations and to link elements detected by
the neurons of layers of the early layers group or of the middle
layers group; [0039] each sample corresponds to an annotated
standardized form in which each visual pattern is annotated as a
data block, called annotated data block, and is categorized to a
known class; [0040] it further comprising a learning phase
comprising the following steps: [0041] processing the plurality of
samples; [0042] generating the learning parameters using the
samples; [0043] the step of processing the plurality of samples
comprises the following sub-steps, implemented for each sample:
[0044] forming an input layer using the sample; [0045] propagating
the input layer through at least one hidden layer; [0046] forming
an output layer using data obtained after propagation through the
last hidden layer, the output layer being composed with a plurality
of test data blocks; [0047] each layer comprising a plurality of
neurons processing at least some items of the corresponding data;
[0048] at least some neurons of each layer being connected to at
least some neurons of its neighboring layer(s) by a propagation
function depending on a current set of weights. [0049] the step of
processing a plurality of samples comprises further the following
sub steps, implemented for each sample: [0050] determining a cost
function representing differences between the test data blocks and
the corresponding annotated data blocks; [0051] determining a new
set of weights by optimizing the cost function; [0052] the
unstructured set of data corresponds to an image or PDF file;
[0053] the standardized form is an invoice document; [0054] each
class of data block is chosen from the group comprising: title,
supplier identification, supplier logo, client identification, bank
coordinates, items, invoice data, total without taxes, taxes, total
with taxes, tax rate, remaining, payment terms, period, stamp,
currency, barcode, useful information; [0055] each data block
comprises linked data items.
[0056] The invention also relates to a computer program product
comprising software instructions which, when executed by a
processor, implement a method defined above.
[0057] The invention also relates to a processing system configured
to implement the processing phase of the method defined above.
[0058] The invention also relates to a learning system configured
to implement the learning phase of the method defined above.
[0059] The method defined above can also be considered as a method
for transforming an unstructured set of data representing an
invoice document to a structured set of data;
[0060] the method comprising a processing phase;
[0061] the processing phase comprising the following steps: [0062]
determining a plurality of data blocks in the unstructured set of
data using learning parameters determined using a plurality of
samples and using a machine learning technique, each sample
corresponding to an annotated invoice document in which each visual
pattern is annotated as a data block, called annotated data block,
and is categorized to a known class, each data block corresponding
to a visual pattern on the invoice document and being categorized
to a known class; [0063] processing data in each data block; [0064]
forming a structured set of data using the processed data from each
data block, according to the class of this block.
BRIEF DESCRIPTION OF THE DRAWINGS
[0065] The invention and its advantages will be better understood
upon reading the following description, which is given solely by
way of non-limiting example and which is made with reference to the
appended drawings, in which:
[0066] FIG. 1 is a schematic view of an unstructured set of data
representing a supplier invoice;
[0067] FIG. 2 is a schematic view of a processing system and a
learning system, according to a first embodiment of the
invention;
[0068] FIG. 3 is a flowchart of a method for transforming an
unstructured set of data according to the first embodiment of the
invention, the method being implemented by the systems of FIG.
2;
[0069] FIGS. 4 to 6 are a schematic views illustrating some of the
steps of the method of FIG. 3;
[0070] FIG. 7 is a schematic view of a processing system and a
learning system, according to a second embodiment of the invention;
and
[0071] FIG. 8 is a schematic view of a processing system, according
to a third embodiment of the invention.
DETAILED DESCRIPTION OF CERTAIN INVENTIVE EMBODIMENTS
First Embodiment of the Invention
[0072] The processing system 10 of FIG. 2 is able to transform an
unstructured set of data representing a standardized form to a
structured set of data, using learning parameters.
[0073] Particularly, the processing system 10 is able to implement
a processing phase PP of the method according to the first
embodiment of the invention, as it will be explained in more detail
thereafter.
[0074] As it is mentioned above, an unstructured set of data
corresponds, for example, to an image or PDF file corresponding
notably to a scanned version of the associated standardized
form.
[0075] A structured set of data associated to a standardized form
corresponds, for example, to an XML file comprising readable data
by an external computer system.
[0076] In reference to FIG. 2, the processing system 10 comprises
an input module 12, a computing module 14 and an output module
16.
[0077] According to one example of the first embodiment of the
invention, the processing system 10 is a computer or a computer
cluster comprising a memory able to store software items and at
least one processor able to process at least some software items
stored in the memory. In this case, the modules 12, 14 and 16
present such software items.
[0078] According to another example of the first embodiment of the
invention, the processing system 10 presents a more complex
computing structure wherein the modules 12, 14 and 16 present at
least partially programmable logic devices able to execute at least
some computing instructions. An example of such a programmable
logic device is an FPGA (Field-programmable gate array).
[0079] The input module 12 is able to acquire data issued from a
first database 21 and a second database 22.
[0080] The computing module 14 is able to process the data acquired
by the input module 12.
[0081] The output module 16 is able to transmit the data processed
by the processing module 14 to a third database 23.
[0082] The third database 23 is able to store data issued from the
system 10, i.e. structured sets of data representing standardized
forms. This database 23 is able notably to transmit the stored data
to an external system and according to one example of the first
embodiment of the invention, presents a simple buffer connecting
the processing system 10 to this external system.
[0083] The first database 21 is able to store a plurality of
unstructured sets of data, each unstructured set of data
representing a standardized form. These sets of data are issued
from known data transmission or generating means, as for example
supplier mails or emails.
[0084] According to one example of the first embodiment of the
invention, the first database 21 presents a simple buffer
connecting for example an external data transmission system to the
processing system 10.
[0085] The second database 22 is able to store a plurality of
learning parameters generated by a learning system 30 according to
the invention.
[0086] Particularly, the learning system 30 is able to implement a
learning phase LP of the method according to the first embodiment
of the invention, as it will be explained in more detail
thereafter.
[0087] The structure of the learning system 30 is similar to the
structure of the processing system 10.
[0088] Hence, like in the previous case, the learning system 30 is
a computer or a computer cluster comprising an input module 32, a
computing module 34 and an output module 35, presenting software
items.
[0089] According to another example of the first embodiment of the
invention, the learning system 30 presents a more complex computing
structure wherein the modules 32, 34 and 36 present at least
partially programmable logic devices able to execute at least some
computing instructions.
[0090] The input module 32 is able to acquire data issued from a
fourth database 44.
[0091] The computing module 34 is able to process the data acquired
by the input module 32.
[0092] The output module 36 is able to transmit the data processed
by the computing module 14 to the second database 22.
[0093] The forth database 22 is able to store a plurality of
samples, each sample corresponding to an annotated standardized
form in which each visual pattern is annotated as a data block and
is categorized to a known class.
[0094] For example, when the corresponding standardized form
presents a supplier invoice, each known class is chosen from the
group comprising notably: [0095] title; [0096] supplier
identification; [0097] supplier logo; [0098] client identification;
[0099] bank coordinates; [0100] items; [0101] invoice data; [0102]
total without taxes; [0103] taxes; [0104] total with taxes; [0105]
tax rate; [0106] remaining; [0107] payment terms; [0108] period;
[0109] stamp; [0110] currency; [0111] barcode; [0112] useful
information.
[0113] A data block corresponding to an annotated visual pattern
will be called hereinafter annotated data block.
[0114] In the meaning of the present invention, each visual pattern
on a standardized form corresponds to a group of elements on this
form that can be visually distinguished from another group of
elements.
[0115] Thus, for example, each visual pattern comprises for example
a specific style of characters, a particular foreground or
background color, the presence of delimiting lines (horizontal or
vertical) or any other graphical elements.
[0116] In the example of FIG. 1, the title 1 and the company name 2
are printed using character styles different from the character
style of the other characters of the invoice. So, each one of these
items can be associated to an independent visual pattern.
[0117] Moreover, in the same example, the client identification 5
is delimited by a box and thus, can be associated to an independent
visual pattern. Admitting that the items description box is filled
with a specific background color (not visible on the black and
white reproduction of FIG. 1), this box can be also associated to
an independent visual pattern.
[0118] At least some visual patterns may be comprised in one or
several other visual patterns.
[0119] So, in the example of FIG. 1, each line ("Product 1",
"Product 2", "Service 1", "Invoice Total") of the items description
box forms a visual pattern comprised in more general visual pattern
of the items description box.
[0120] At least some visual patterns may contain linked data items.
Particularly, by "linked data items", it is understood data items
having at least one semantic and/or visual link. A semantic link is
a link that can be established using a known rule, as for example a
grammar rule, a mathematical rule, a physical rule, a logical rule,
etc.
[0121] Thus, in the example of FIG. 1, the visual pattern
corresponding to the items description box comprises the linked
data items "Total", "Invoice Total" and "520 ".
[0122] Indeed, the data item "520 " is situated in the intersection
position of a vertical line passing through the data item "Total"
and a horizontal line passing through the data item "Invoice
Total". In this case, these data items are linked visually.
[0123] In the same example, the visual pattern corresponding to the
line "Invoice Total" comprises three linked data items: "500 ", "20
" and "520 ". Indeed, in this case, the total amount "520 "
corresponds to the sum of the prices "500 " and taxes "20 ". In
this case, these data items are linked semantically by a
mathematical rule.
[0124] Method
[0125] The method for transforming an unstructured set of data
representing a standardized form to a structured set of data,
according to the first embodiment of the invention will now be
explained in view of FIG. 3 presenting a flowchart of its
steps.
[0126] As mentioned before, the method comprises a processing phase
PP implemented by the processing system 10 and a learning phase LP
implemented by the learning system 30.
[0127] The learning phase LP is executed at least once before
executing the processing phase PP. It is understood that the
invention is not limited to this particular example. The leaning
phase LP and the processing phase PP can be implemented by
different systems or a same system, as it will be explained in
relation with the further embodiments. In addition, multiple
processing phases PP can be implemented after a learning phase
LP.
[0128] Initially, the second and the third databases 22, 23 are
empty. The first database 21 contains at least one unstructured set
of data to be transformed to a structured set. The fourth database
44 contains a plurality of samples as defined above. In particular,
each sample defines a plurality of annotated data blocks and a
class for each data block. For example, such annotation has been
performed manually by an operator or a user for the corresponding
type of the standardized form, using an annotation tool known in
the art.
[0129] Such an annotation tool is chosen for example from the group
comprising: Faster RCNN, Yolo, SSD, Feature Pyramid Pooling,
DeepMask, SharpMask and Mask RCNN.
[0130] Particularly, each one of these annotation tools is able to
detect visual patterns using one of the known computer vision
techniques.
[0131] The operator or the user can thus validate or modify the
detected visual pattern by the tool.
[0132] Moreover, the operator or the user has the possibility to
associate to each detected or modified visual pattern a class
depending on the type of the corresponding standardized form.
Step 110
[0133] During the first step 110 of the learning phase LP, the
learning system 30 acquires the samples contained in the fourth
database 44.
[0134] Then, the learning system 30 processes these samples in
order to generate a plurality of learning parameters, using a
machine learning technique.
[0135] In particular, during this step 110, the learning system 30
processes each sample via a convolutional neural network, known
also as CNN.
[0136] The CNN used by the learning system 30 contains an input
layer, a plurality of hidden layers and an output layer. At least
some of the hidden layers are convolutional layers.
[0137] As it is known in the art, each one of the layers of such a
network comprises a plurality of neurons processing at least some
items of the data processed by this layer. Moreover, as it is known
in the art, at least some neurons of each layer are connected to at
least some neurons of its neighboring layer(s) by a propagation
function depending on a plurality of weights.
[0138] Each layer presents a size which corresponds to the number
of neurons used by the layer. Moreover, each layer presents a
dimension which is determined by the arrangement of the neurons in
the layer.
[0139] Initially the weights are defined for example by random
coefficients.
[0140] Each neuron in a given layer is defined by an activation
function which computes a neuron output from a neuron input which
is determined by neuron outputs of the predecessor neurons and by
the corresponding propagation functions.
[0141] By expression "predecessor neuron" in relation with a given
neuron, it is understood a neuron from the layer previous to the
layer of the given neuron and connected to this given neuron.
[0142] Generally, the same activation function is used for all of
the neurons of the same layer.
[0143] Different examples of the activation functions can be
chosen.
[0144] Thus, for example, it is possible to choose an activation
function called "ReLU" which signifies "rectified linear unit".
This function is defined as positive part of its argument, i.e. of
the neuron input. In other words, by denoting this function as f,
it can be defined using the following expression:
f(x)=max(0,x),
[0145] where x is the neuron input.
[0146] It is also possible to choose a ReLU function extended to
include Gaussian noise. Such a function is called "Noisy ReLU" and
is determined using the following expression:
f ( x ) = e x e x - 1 , ##EQU00001##
[0147] where Y is a random variable following the Gaussian
distribution.
[0148] It is also possible to choose an activation function called
"sigmoid" corresponding to a sigmoid curve. By denoting this
function as f, it can be defined using the following
expression:
f ( x ) = e x e x - 1 , ##EQU00002##
[0149] where x is the neuron input.
[0150] It is also possible to choose an activation function called
"softmax" which gives a vector output from several preceding neuron
outputs. It can be defined using the following expression:
f j ( x ) = e x i k e x k . ##EQU00003##
[0151] where:
[0152] f.sub.j is a j component of the vector output, and
[0153] z is a vector input composed with z.sub.j components
corresponding to several neuron outputs from the previous
layer.
[0154] Of course, many other examples of activation functions are
still possible.
[0155] Different examples of the hidden layers as well as their
dimension, size and activation functions are also possible.
[0156] Thus, as indicated below, at the CNN implemented by the
learning system 30, at least some of the hidden layers are
convolutional layers.
[0157] In particular, the activation function of each of the
neurons of a convolutional layer presents a convolutional operation
depending on the outputs of a group of neurons of the previous
layer which is called "receptive field".
[0158] So, using of the convolutional layers reduces the number of
the weights used by the corresponding propagation functions.
[0159] The CNN implemented by the learning system 30 may also
comprise some other types of hidden layers, as for example a
pooling layer.
[0160] In particular, each neuron of a pooling layer combines the
outputs of a group of neurons of the previous layer. Such a neuron
computes its activation function as function of these outputs.
[0161] For example, each of the neurons of a pooling layer called
"max pooling" computes its activation function as the maximum value
from the outputs of the associated group. Each of the neurons of a
pooling layer called "average pooling" computes its activation
function as the average value from the outputs of the associated
group.
[0162] According to a particular example of the first embodiment of
the invention, the CNN constructed by the learning system 30
comprises at least four hidden layers. Among these layers, three
layers are convolutional two-dimensional layers using the ReLU
function as activation function and one layer is a max pooling
layer.
[0163] Such a CNN can be constructed using for example Keras
library for Python.
[0164] Returning to the description of the step 110 in more detail,
during this step, the learning system 30 implements a
backpropagation technique via the constructed CNN for the samples
contained in the fourth database 44.
[0165] The implementation of this technique applies execution of
the following five sub steps, implemented consecutively for each
sample.
[0166] In particular, during a first sub-step, the learning system
30 forms an input layer using the corresponding sample.
[0167] During a second sub-step, the learning system 30 propagates
the input layer through the hidden layers using notably the
corresponding propagation functions and the current set of
weights.
[0168] During a third sub-step, the learning system 30 forms an
output layer so as it is composed with a plurality of data blocks
which are considered by the CNN to correspond to the visual
patterns of the corresponding sample. These data blocks are called
hereinafter test data blocks.
[0169] During a forth sub-step, the learning system 30 determines a
cost function representing differences between the test data blocks
and the annotated data blocks on the corresponding sample. The
differences concern for example distances, dimensions, forms, block
classes, and other features of the corresponding blocks.
[0170] Such a cost function is determined using an appropriate
metric known in the art.
[0171] During a fifth sub-step, the learning system 30 determines a
new set of weights using the cost function and an optimization
technique, known in the art. This technique could consist for
example in minimization of the cost function.
[0172] Optionally, for at least one the samples, the step 110
further comprises a sixth sub-step wherein the learning system 30
introduces random variations or noises to the corresponding
sample.
[0173] Then, the learning system 30 executes once again the
previous sub-steps for the modified sample.
[0174] In an alternative embodiment, the fourth database 44
comprises already one or several samples corresponding to different
variations of another sample comprised in this database 44.
[0175] According to one example of the first embodiment of the
invention, the step 110 comprises a preliminary sub-step executed
for at least one sample. During this preliminary sub-step, the
corresponding sample is pre-processed before its processing by the
CNN. For example, such pre-processing may comprise applying of a
known dimension reduction technique.
[0176] After executing these sub-steps for all of the samples
contained in the fourth database 44, the learning system executes
the step 120 consisting in determining a plurality of learning
parameters.
Step 120
[0177] In particular, during this step 120, the learning system 30
defines these learning parameters by the current set of
weights.
[0178] At the end of the step, the learning system 30 stores the
learning parameters in the second database 22 eventually with the
CNN's structure using one or several appropriate frameworks.
[0179] Such a framework corresponds for example to one of the known
frameworks used by the following machine learning libraries or
tools: Labelme, CNTK FasterRCNN/VoTT and Detectron/Caffe2.
[0180] At the end of the learning phase LP, the CNN with the
learning parameters present a structure which is configured to
detect data blocks corresponding to visual patterns on a
standardized form.
[0181] In particular, the hidden layers of such a structure can be
regrouped into three consecutive groups of layers called
respectively early layers group, middle layers group and last
layers group.
[0182] The neurons of the early layers group are configured to
detect characters, lines and textures in the data processed by the
layers of this group.
[0183] This configuration is illustrated on FIG. 4 wherein the
characters "z", "e", "a", "2", "1", "T", "o", "t", "a", "I" as well
as a line and an arrow, have been detected by a layer 45 of the
early layers group, from the input layer 46.
[0184] The neurons of the middle layers group are configured to
detect groups of visual attributes and higher level semantics in
the data processed by the layers of this group.
[0185] For example, the visual attributes comprise logos, barcodes,
stamps or other graphical illustrations.
[0186] The higher level semantics comprise words, numerics, dates,
expressions, etc.
[0187] In the example illustrated on FIG. 5, the layers of the
middle layers group are able to detect for example the logo 47A,
the stamp 47B, the barcode 47C, the word "total" 48A, the word
"amounts" 48C and the number "970,59".
[0188] The neurons of the last layers group are configured to
detect segmentations and to link at least some data blocks detected
previously one to another so as to form a new data block or to
categorize at least one of these blocks.
[0189] Thus, in the example of FIG. 5, the layers of the last
layers group are for example able to detect the segmentations 49A
and 49B corresponding respectively to "products" and
"services".
[0190] Further, in the example of the same figure, the layers of
the last layers group are for example able to link the data blocks
48A, 48B and 48C one to another so as to categorize the data block
48B. Indeed, it clear that the number 48B placed in the
intersection position of a horizontal line passing through the data
block "total" 48A and a vertical line passing through the data
block 48C, corresponds to the total amount of the invoice
illustrated on this figure.
Step 150
[0191] During the first step 150 of the processing phase PP, the
processing system 10 acquires an unstructured set of data to be
transformed from the first database 21 as well as the learning
parameters and eventually the CNN's structure from the second
database 22.
Step 160
[0192] During the next step 160, the processing system 10
determines a plurality of data blocks in the unstructured set of
data using the learning parameters. Each data block corresponds to
a visual pattern on the corresponding standardized form and is
categorized to a known class.
[0193] This step 160 is performed using the same CNN used during
the learning phase LP with the weights corresponding to the
learning parameters.
[0194] Consequently, this step 160 comprises several sub-steps
which are similar to the first, second and third sub-steps of the
step 110 of the learning phases LP detailed before.
[0195] Particularly, during a first sub-step of the step 160, the
processing system 10 forms an input layer using the unstructured
set of data.
[0196] During a second sub-step, the processing system 10
propagates the input layer through the hidden layer explained
above, using the same propagation functions as well as the weighs
issued from the second database 22.
[0197] During a third sub-step, the processing system 10 forms an
output layer using data obtained after propagation through the last
hidden layer. This output layer is composed with a plurality of
data blocks, each data block being characterized to a known
class.
[0198] An example of an output layer obtained during the execution
of this sub-step for a particular unstructured set of data
representing an invoice, is illustrated on FIG. 6.
[0199] Thus, in reference to this figure, during this sub-step, the
processing system 10 determines blocks 51 to 57 which are
categorized respectfully as a title block (block 51), a stamp block
(block 52), a supplier logo block (block 53), a client
identification block (block 54), a bank coordinates block (block
55), a period block (block 56) and an invoice data block (block
57).
[0200] As it can be seen on this figure, each one of these blocks
corresponds to a specific visual pattern.
[0201] Indeed, the block 51 contains delimiting lines. The block 52
contains some irregular graphical elements which correspond to a
date stamp and a processed stamp. The block 53 contains a regular
graphical element which corresponds to a suppler logo. The blocks
54 and 55 contain characters regrouped in a specific manner.
Additionally, the block 55 may be delimited by a specific
background color (not visible on the black and white reproduction
of the figure). Similarly, the blocks 56 and 57 contain characters
regrouped in a specific manner and delimited by separation lines
and boxes.
[0202] Moreover, at least some of these blocks contain sub-blocks.
Thus, the invoice data block 57 contains a first item block 58 and
a second item block 59. Of course, each one of these sub-blocks can
contain other sub-blocks related for example to an amount, a total
amount, taxes, etc.
Step 170
[0203] During the next step 170, the processing system 10 processes
data in each data block.
[0204] This step is for example performed by capturing data
characters in the corresponding data block. In this case, a
sub-step of optical recognition of characters comprised in the
corresponding data block, is executed.
[0205] The step 170 can also be performed without data capturing,
using only the information about the corresponding block. Thus, for
example, if the block's class corresponds to the class "stamp" or
to the class "supplier logo", this data block can be simply
ignored.
Step 180
[0206] During the next step 180, the processing system 10 forms a
structured set of data using the processed data from each data
block, according to the class of this block.
[0207] Thus, for example, for a data block categorized to the class
"supplier identification", the characters captured in this block
will be automatically associated to the supplier's name and address
in the corresponding field of the structured set of data.
[0208] According to one example of the first embodiment of the
invention, this step 180 further comprises analyzing the data
blocks using an analysis by regular expressions.
[0209] In particular, each regular expression presents a generic
predetermined rule, also called a pattern.
[0210] For example, it is possible to associate to each field of
the structured set of data, a regular expression that matches a
data block if the data issued from this data block corresponds to
this field.
[0211] Thus, it is possible to implement an additional test of the
class associated to each data blocks.
[0212] One can conceive that the invention according to the first
embodiment presents a certain number of advantages.
[0213] First of all, the invention proposes to recognize visual
patterns (or data blocks in the meaning of the invention) in a
standardized form and not the characters as it is proposed by the
art. Each visual pattern is associated to a known class.
[0214] According to the invention, the visual patterns are
recognized using a machine learning technique using predetermined
learning parameters.
[0215] These learning parameters are determined using a plurality
of annotated samples and for example a backpropagation
technique.
[0216] The samples can be annotated manually depending on the type
of the considered standardized form.
[0217] After recognizing data blocks corresponding to the visual
patterns of the standardized form, data contained in each block can
be processed using, among others, an optical character recognition
routine. Given the information about the block's class, the
recognized characters can be more simply associated to the
corresponding field in the structured set of data.
[0218] For example, for a data block categorized to the class "bank
coordinates", a sequence of numeric characters corresponds more
likely to the supplier's IBAN than to the supplier's telephone
number.
[0219] Moreover, an analysis by regular expressions can be applied
in order to check the captured data validity. Thus, for example, it
is possible to check the number captured as an IBAN by a regular
expression corresponding to a generic form of an IBAN number.
[0220] Thus, using the invention, there is no need to constitute a
separate database for each supplier or more generally, issuing
part. A generic database containing the learning parameters for
each specific type of standardized form is sufficient to process
any standardized form issued even from a new supplier.
Second Embodiment of the Invention
[0221] FIG. 7 illustrates a processing system 210 and a learning
system 230 according to a second embodiment of the invention.
[0222] According to the second embodiment of the invention, the
processing system 210 and the learning system 230 are integrated
into a unique computing structure 220 further comprising an input
module 212 and an output module 216.
[0223] The unique computing structure 220 is for example a computer
or a computer cluster comprising a memory able to store software
items and at least one processor able to process at least some
software items stored in the memory. In this case, the modules 212
and 216, and the systems 210 and 230 present such software
items.
[0224] According to another example of the second embodiment of the
invention, the unique computing structure 220 presents a more
complex computing structure wherein the modules 212 and 216, and
the systems 210 and 230 present at least partially programmable
logic devices able to execute at least some computing instructions.
An example of such a programmable logic device is an FPGA
(Field-programmable gate array).
[0225] According to the second embodiment of the invention, the
input module 212 is able to acquire data issued from a first
database 221 similar to the first database 21 explained above and
from a fourth database 244 similar to the fourth database 44
explained above.
[0226] The processing system 210 is able to implement a processing
phase PP of the method according to the second embodiment of the
invention using data acquired by the input module 212 as well as
data issued from a second database 222 which, according to the
example illustrated on FIG. 8, is integrated into the unique
computing structure 220.
[0227] Further, by implementing the processing phase PP, the
processing system 210 is able to produce data.
[0228] The learning system 230 is able to implement a learning
phase LP of the method according to the second embodiment of the
invention using data acquired by the input module 212.
[0229] Further, by implementing the learning phase LP, the learning
system 210 is able to produce data and to store the produced data
in the second database 222.
[0230] Finally, the output module 216 is able to acquire the data
produced by the processing system 210 and to store this data in the
third database.
[0231] According to a non-illustrated example of the second
embodiment of the invention, the second database is an exterior
database from the unique computing structure 220. In this case, the
input module 212 is further able to acquire data stored in this
second database 222 and the output module 216 is further able to
store data produced by the learning module 230 in the second
database 222. Additionally, in this case, the processing system 210
is able to acquire the data issued from the second database 222 via
the input module 212.
[0232] The method according to the second embodiment of the
invention is similar to the method according to the first
embodiment of the invention, and will not be explained in further
detail.
[0233] The invention according to the second embodiment presents
the same advantages as the invention according to the first
embodiment.
Third Embodiment of the Invention
[0234] FIG. 8 illustrates a processing system 310 according to a
third embodiment of the invention.
[0235] According to the third embodiment of the invention, the
processing system 310 is similar to the processing system 10
according to the first embodiment of the invention.
[0236] Particularly, according to the third embodiment of the
invention, the processing system comprising an input module 312, a
computing module 314 and an output module 316 which are similar
respectfully to the modules 12, 14 and 16 explained before.
[0237] The input module 312 is able to acquire data issued from a
first database 321 and a second database 322.
[0238] The computing module 314 is able to process the data
acquired by the input module 312.
[0239] The output module 316 is able to store the data processed by
the computing module 314 in a third database 323.
[0240] The first database 321 and the third database 323 are
similar respectively to the first database 21 and the third
database 23 explained above.
[0241] According to the third embodiment of the invention, the
second database 322 is able to store learning parameters generated
by an external system.
[0242] Thus, for example, these parameters are generated by the
external using a method similar to the learning phase LP explained
before.
[0243] The method according to the third embodiment of the
invention differs from the method according to the first embodiment
of the invention only in that it does not comprise a specific
learning phase.
[0244] In this case, the learning parameters used during the
processing phase PP are issued directly from the second database
322.
[0245] Moreover, it is clear that if the processing phase PP of the
method according to the third embodiment of the invention uses a
CNN, its structure should be similar to the structure of a CNN used
by the external system to generate the learning parameters.
[0246] The invention according to the third embodiment presents the
same advantages as the invention according to the first and the
second embodiments.
* * * * *