U.S. patent application number 11/802171 was filed with the patent office on 2008-11-27 for method and apparatus for performing semantically informed text operations.
This patent application is currently assigned to JustSystems Evans Research, Inc.. Invention is credited to Jeffrey K. Bennett, Hua Cheng, David A. Evans, Ilya M. Goldin, David A. Hull, Jesse A. Montgomery, Yan Qu, Carol L. Tenny.
Application Number | 20080295013 11/802171 |
Document ID | / |
Family ID | 40073559 |
Filed Date | 2008-11-27 |
United States Patent
Application |
20080295013 |
Kind Code |
A1 |
Evans; David A. ; et
al. |
November 27, 2008 |
Method and apparatus for performing semantically informed text
operations
Abstract
One example of a semantically informed text operation comprises
selecting a source region of a document and determining if the
source region has a surface region bi-directionally coupled to a
semantic object. The coupled semantic object is identified as are
the presentation(s) associated with the semantic object. A target
region of the same or anther document is selected. Any of the
presentations that cannot be expressed in the target region are
eliminated to identify a set of remaining presentations. A set of
semantic choices based on the remaining presentations is
determined. One of the semantic choices is selected and executed in
the target region.
Inventors: |
Evans; David A.;
(Pittsburgh, PA) ; Bennett; Jeffrey K.;
(Pittsburgh, PA) ; Hull; David A.; (Pittsburgh,
PA) ; Cheng; Hua; (Bridgeville, PA) ; Qu;
Yan; (Cupertino, CA) ; Tenny; Carol L.;
(Pittsburgh, PA) ; Montgomery; Jesse A.; (Garland,
TX) ; Goldin; Ilya M.; (Pittsburgh, PA) |
Correspondence
Address: |
JONES DAY
222 EAST 41ST ST
NEW YORK
NY
10017
US
|
Assignee: |
JustSystems Evans Research,
Inc.
Pittsburgh
PA
|
Family ID: |
40073559 |
Appl. No.: |
11/802171 |
Filed: |
May 21, 2007 |
Current U.S.
Class: |
715/770 |
Current CPC
Class: |
G06F 40/30 20200101;
G06F 40/166 20200101 |
Class at
Publication: |
715/770 |
International
Class: |
G06F 3/048 20060101
G06F003/048 |
Claims
1. A semantically informed text operation, comprising: selecting a
source region of a document; determining if said source region has
surface regions bi-directionally coupled to semantic objects;
identifying the coupled semantic objects and the presentations
associated with the semantic objects; selecting a target region of
a document; eliminating any of the presentations that cannot be
expressed in said target region to identify a set of remaining
presentations; determining a set of semantic choices based on said
remaining presentations; selecting one of said semantic choices;
and executing the selected semantic choice in said target
region.
2. The method of claim 1 wherein said selecting a source region is
performed one of manually or automatically.
3. The method of claim 1 wherein said eliminating comprises
determining a set of presentations that are referentially
compatible with the identified semantic objects.
4. The method of claim 1 additionally comprising displaying said
set of semantic choices, including displaying defaults.
5. The method of claim 1 wherein one of said semantic choices is to
semantically paste said selected source region into the target
region.
6. The method of claim 5 additionally comprising updating the
information that bi-directionally couples said surface region to
said semantic object to reflect said surface region's new
position.
7. The method of claim 5 additionally comprising cutting said
source region from said document.
8. A semantically informed text operation, comprising: selecting a
source region of a document; identifying a surface region in said
source region that is linked to a remotely stored semantic object
and wherein the remotely stored semantic object is associated with
the surface region which links to said semantic object; and
performing an operation on the text in the source region of the
document consistent with the semantic object and presentation
constraints associated with said source region.
9. A semantically informed text operation, comprising: selecting a
source region of a document; identifying a surface region in said
source region that is linked to a remotely stored semantic object
and wherein the remotely stored semantic object is associated with
the surface region which links to said semantic object; selecting a
target region in a document; and performing an operation on the
text in the target region of the document consistent with the
semantic object and presentation constraints associated with said
target region.
10. The method of claim 8 wherein said performing an operation
includes selecting a subset of the semantic objects identified in
the source region.
11. The method of claim 10 wherein said selecting a subset is based
on one of the attributes of the semantic objects or the types of
semantic objects.
12. The method of claim 8 wherein said performing an operation
includes one of removing the surface text in the selected source
region or removing any links between semantic objects and the
surface text in the selected source region.
13. The method of claim 8 wherein said performing an operation
includes temporarily storing one of the identified semantic objects
or links to said objects.
14. The method of claim 8 wherein said performing an operation
includes selecting a presentation format consistent with a semantic
object identified in the source region.
15. The method of claim 9 wherein said identifying is followed by
selecting a subset of the semantic objects identified in the source
region.
16. The method of claim 15 wherein said selecting is based on one
of the attributes of the semantic objects or the types of semantic
objects.
17. The method of claim 9 wherein said identifying is followed by
one of removing the surface text in the selected source region or
removing any links between semantic objects and the surface text in
the selected source region.
18. The method of claim 9 wherein said identifying is followed by
temporarily storing one of the identified semantic objects or links
to said objects.
19. The method of claim 9 wherein said identifying is followed by
selecting a presentation format consistent with a semantic object
identified in the source region.
20. The method of claim 9 wherein said performing an operation
includes performing a paste operation consisting of inserting a
copy of the source region into the target region.
21. The method of claim 20 wherein said paste operation includes
updating local and remote resources to maintain the links between
surface regions and their remotely stored semantic objects in the
pasted surface text.
22. The method of claim 20 wherein said paste operation includes
selecting a presentation format for the linked surface regions
being inserted in the target region.
23. The method of claim 9 where said performing an operation
includes one of removing the surface text in the selected target
region or removing any links between semantic objects and the
surface text in the selected target region.
24. A computer readable medium storing a set of instructions for
performing a semantically informed text operation, comprising:
determining if a selected source region of a document has surface
regions bi-directionally coupled to semantic objects; identifying
the coupled semantic objects and the presentations associated with
the semantic objects; for a selected target region of a document,
eliminating any of the presentations that cannot be expressed in
said target region to identify a set of remaining presentations;
determining a set of semantic choices based on said remaining
presentations; and executing a selected one of said semantic
choices in said target region.
25. A computer readable medium storing a set of instructions for
performing a semantically informed text operation, comprising:
identifying a surface region in a selected source region of a
document that is linked to a remotely stored semantic object and
wherein the remotely stored semantic object is associated with the
surface region which links to said semantic object; and performing
an operation on the text in the source region of the document
consistent with the semantic object and presentation constraints
associated with said source region.
26. A computer readable medium storing a set of instructions for
performing a semantically informed text operation, comprising:
identifying a surface region in a selected source region of a
document that is linked to a remotely stored semantic object and
wherein the remotely stored semantic object is associated with the
surface region which links to said semantic object; and performing
an operation on the text in a selected target region of the
document consistent with the semantic object and presentation
constraints associated with said target region.
Description
BACKGROUND
[0001] The present disclosure is directed generally to information
technology and, more particularly, to the use of semantic
information for processing expressions found in documents, images,
etc.
[0002] Since the advent of word processors and (especially What You
See Is What You Get WYSIWYG) electronic document editors (going
back to the Xerox Bravo Editor and its precursors), there have been
a number of attempts to systematize (formalize) the relationship
between the underlying data in an electronically-held information
object and its intended surface expression (projection or
presentation) as viewed by a user. Some past efforts include the
work on TeX, SGML, HTML, XML and its extensions, and, more
recently, RDF. Today, information coded in SGML, HTML, or XML can
be found commonly used in many systems, including on the
Internet.
[0003] While electronic information formats and the scope (and
power) of annotations have grown more sophisticated, the ability of
systems to process information objects based on the intended
meaning or interpretation of their contents has remained limited.
This aspect of information processing--being able to understand the
information presented, its semantics--is not addressed by
formatting or data-encoding conventions. Annotations in a document,
for example, may tell us that "John Smith" is a person's name or
the surface label of a link to a web page, but they cannot provide
the information required to interpret all references to the
individual, John Smith, and to associate with that individual all
the attributes and values that may be asserted in the document (and
other documents) as presented by the system. Such power of
interpretation is left to the user.
[0004] It would be especially valuable for a document or
information management system to maintain a persistent, coherent,
and correct representation of the important elements in an
information object and make them available, automatically, for use
whenever a document is being used or whenever various
document-transformation operations are being performed.
[0005] One common document transformation operation is "copy and
paste." In current systems, copying an item or a region of a
document involves a literal copy of the surface contents, including
the hidden formatting information. Pasting an item or region
involves reproducing the source in a virtually identical form
(copy). Some variations on this basic operation do exist, but they
represent little more than choices among formatting controls.
[0006] In one example variation, the copied contents may "lose"
some formatting when pasted into the new area. In the system
PowerPoint, when one copies slides from one presentation into
another presentation with different basic formatting (e.g.,
background colors, headers and footers, slide-layout formats,
etc.), the copied material takes on some of the characteristics of
the slides/file into which it is pasted. The "text content" may be
reproduced, but in a different font or color; the old background
may "disappear" and the new background may be identical to that of
the slides/file into which the material was pasted.
[0007] In another example variation, the user may be given choices
of formatting or of operations to be performed on the copied
material, possibly also involving the target into which that
material is being pasted. In the system Excel, it is possible to
copy a range of cells in a spreadsheet, select a target "insert"
point (or range), and then select from the "Edit" menu the option
"Paste Special." This option lists alternative styles for rendering
the contents of the copied area, for example, pasting/reproducing
the formulas from the copied cells, or the formats, or the values,
or combinations of such features. Furthermore, the user may have
the program perform operations on the target with the copied
material, such as multiplying the content or contents of the source
cells by the content or contents of the target cells, etc. The
result of the operation, then, is neither a copy of the source nor
a preservation of the target, rather, is "new" content derived from
both source and target.
[0008] Both of the preceding examples from the PowerPoint and Excel
systems qualify as advanced operations, but they involve nothing
more than manipulations of the immediately available encoding of
text or cell contents and operations; they do not reflect an
independent semantic basis for the processing that is performed.
Annotations and localized control of reference to elements, the
contents of expressions, and formatting contexts do not provide a
basis for semantic processing. To achieve semantics, we must both
exploit such types of localized control and also go beyond this to
ground and represent the contents of information objects in
independently established and maintained representation
schemata.
[0009] While there has been a great deal of work on semantic
representations, on text mining, on the identification of entities
and fact extraction, on text understanding and generation, there
has been no development of a system that supports general
information object (document) operations that are based on semantic
principles. Thus, the need exists for a system and method for
semantically anchoring surface expressions to an ontological model
of semantic information, and for manipulating text having such
semantically anchored expressions in ways informed by the semantic
and ontological information.
SUMMARY
[0010] According to one embodiment of the present disclosure, a
semantically informed text operation comprises selecting a source
region of a document and identifying a surface region in the source
region that is linked to a remotely stored semantic object. The
remotely stored semantic object is associated with the surface
region which links to the semantic object. An operation is
performed on the text in the source region of the document
consistent with the semantic object and presentation constraints
associated with the source region.
[0011] According to another embodiment of the present disclosure a
semantically informed text operation comprises selecting a source
region of a document and identifying a surface region in the source
region that is linked to a remotely stored semantic object. The
remotely stored semantic object is associated with the surface
region which links to the semantic object. A target region is
selected in the same or another document. An operation is performed
on the text in the target region of the document consistent with
the semantic object and presentation constraints associated with
the target region.
[0012] A semantically informed text operation comprises selecting a
source region of a document and determining if the source region
has a surface region bi-directionally coupled to a semantic object.
The coupled semantic object is identified as are the
presentation(s) associated with the semantic object. A target
region of the same or another document is selected. Any of the
presentations that cannot be expressed in the target region are
eliminated to identify a set of remaining presentations. A set of
semantic choices based on the remaining presentations is
determined. One of the semantic choices is selected and executed in
the target region.
[0013] The disclosed method and apparatus provide specific
techniques for a variety of "copy and paste" or "cut and paste"
operations in which specific types of contents of the source
material (copied), the target context, and the result of the
operation (paste) are based on and determined, in part, by
reference to the semantic models that underlie the specific types
of material and their contexts. The disclosed method and apparatus
also provide the user of "copy and paste" and "cut and paste"
operations with information about the alternative possibilities of
selecting or filtering for specific types of contents and their
combinations or transformations in the target context and to give
the user choices among outcomes. The transformation of source and
target material is based on semantic operations that use
information about the semantic objects in the scope of operation
that may not be stored locally in the information objects of the
source or target. As a result, the expression of the result of the
operation may contain new information that was not present in
either the original source or target.
BRIEF DESCRIPTION OF THE DRAWINGS
[0014] For the method and apparatus of the present disclosure to be
easily practiced and readily understood, the method and apparatus
will now be described, for purposes of illustration and not
limitation, in connection with the following figures wherein:
[0015] FIG. 1 is an example of overlapping surface regions
controlled by semantically anchored expressions;
[0016] FIG. 2 illustrates one example of an architecture of
semantically anchored expressions;
[0017] FIG. 3 illustrates the local and remote components of one
example of semantically anchored expressions;
[0018] FIG. 4 illustrates an ontologically complex event that
contains activities and plain text;
[0019] FIG. 5 is a flowchart illustrating one example of a method
for generating a semantically anchored expression;
[0020] FIG. 6 is a flowchart illustrating one example of a method
for rendering a semantically anchored expression;
[0021] FIG. 7 illustrates a system within which the disclosed
method may be practiced;
[0022] FIG. 8 is a flowchart illustrating one embodiment of a
semantic replace operation according to the teachings of the
present disclosure;
[0023] FIG. 9 is an example that schematically illustrates one
embodiment of a semantic replace operation according to the
teachings of the present disclosure;
[0024] FIG. 10 is another example of semantic replace in which
co-references are changed as needed;
[0025] FIG. 11 is a flowchart illustrating one embodiment of a
semantic update operation according to the teachings of the present
disclosure;
[0026] FIGS. 12A and 12B are an example that schematically
illustrates one embodiment of a semantic update operation according
to the teachings of the present disclosure;
[0027] FIG. 13 is a flowchart illustrating a more general version
of semantic replace that supports the replacement of multiple
source and target semantic objects;
[0028] FIG. 14 is a flowchart illustrating one example of a
semantically informed text operation, specifically the steps of a
copy and paste operation;
[0029] FIGS. 15 and 16 illustrate schematically the process shown
in the flowchart of FIG. 14;
[0030] FIG. 17 is a simplified view of the process shown in the
flowchart of FIG. 14 with examples of what the various options in
the menu might look like;
[0031] FIG. 18 illustrates the effect of semantic copy and paste in
which copied material expresses previously unavailable information
when pasted into a target region;
[0032] FIG. 19 is a flowchart illustrating one example of a
semantic merge according to one embodiment of the present
disclosure;
[0033] FIG. 20 is a flowchart illustrating the steps of one example
of the default merge process shown in FIG. 19;
[0034] FIG. 21 is a flowchart illustrating the steps of one example
of the Merge (SO.sub.j, SRs, TR) process shown in FIG. 19;
[0035] FIG. 22 is a flowchart illustrating another example of a
semantic merge according to another embodiment of the present
disclosure;
[0036] FIG. 23 is a flowchart illustrating the steps of one example
of the source merge process shown in FIG. 22; and
[0037] FIGS. 24, 25A, 25B, and 26 illustrate schematically various
examples of the merge processes.
DESCRIPTION
[0038] The method and apparatus of the present disclosure address
the problems set forth above by anchoring surface regions
encompassing words, phrases, and other surface expressions to
semantic objects based on an ontology. In the example paragraph
above, words like "Bill," "Jane," and "wife" can be semantically
anchored to semantic objects which allow the computer to understand
what those words mean, to the extent meaning can be found in either
the semantic object or the ontology. By "semantically anchored" we
mean that there is a bidirectional coupling of the surface regions
which, from the user's perspective, appear as surface expressions,
to the semantic object and of the semantic object to the surface
region. The expression that appears in the region under
Presentation (defined below) may be derived from the underlying
semantic object, or may be completely arbitrary and user-defined.
The association to the region exists independently of any
particular surface expression that appears there. Before we can
begin to explain the method and apparatus of the present
disclosure, we should introduce a few terms. Note that the
introduction of these terms is intended to provide context for the
disclosed embodiments and to satisfy the best mode requirement.
These terms, when used in the claims, should be broadly interpreted
to the extent allowed by the prior art and not limited to the
following definitions.
[0039] In this disclosure, we will be dealing with Information
Objects (IOs). An IO is simply a source of information. An IO may
encompasses images, graphical objects, audio files, and structured
or semi-structured material, as well as text (with or without
mark-up). An IO might be an entire document. An IO might be an
"invisible" point, area of white space, etc. that is nevertheless
able to be pointed to and described. A "point IO" is an information
source--it has information about a location in a document. If the
document is actually an audio file, an IO might be the part of the
audio file where a particular word is spoken, or the "dead air"
between words. One cannot enumerate or even unambiguously identify
all potential IOs in a document because the IOs can be composed.
For example, "John K Smith" could be an IO, but so could "John" and
"K" and "Smith", and the whitespace between the words, etc. There
is a very large (though finite, due to storage granularity) number
of potential IOs in a document.
[0040] A surface region (or surface form) is the region of a
document under the scope of a Semantically Anchored Expression
(SAEs are defined below). The surface region may comprise a point,
word, phrase, or location within the local document as well as
contiguous and noncontiguous or even overlapping points, words,
phrases, or locations within the local document. See FIG. 1 for an
example of overlapping surface regions controlled by SAEs. The
surface region is defined or selected at the time of creation.
[0041] A surface expression is the appearance of the surface region
associated with an SAE at the time of presentation, i.e., runtime.
The surface expression is the visual or behavioral result of a
Presentation (Ps are defined below) interpreting an SAE. For
example, in the case of Identity and Attribute types of SAEs, the
surface expression at runtime will be congruent with the appearance
of the surface region at the time SAE was authored, i.e., design
time. But the surface expression can be anything because it's the
output of a P process. The surface expression could be different in
different Ps. A company directory might appear as a list of names
and extensions--but if the person viewing the document is a new
hire, or unknown to the system, the P might additionally put
pictures and phone numbers next to the names (and suppress that
information for longer term employees).
[0042] An ontology is a specification of the structure of concepts
in a domain of discourse. It may be convenient to think of an
ontology as a model of some type of domain knowledge.
[0043] A Semantic Object (SO) is a named, typed, and structured
entry in a data repository representing an entity and its
associated set of properties (e.g., relations; attributes and
values; other constraints and conditions). An SO is pointed to by
one or more surface regions. The generation of the links that
couple the surface region to the SOs is discussed in greater detail
below. The properties of the SOs are determined to some extent by
the ontology.
[0044] SOs have unique "types" that are apparent to or discoverable
by the system. Ultimately, the lowest-level constituents are
expressible as "primitive" types that can be processed by the
system in standard ways (e.g., strings, integers, doubles).
[0045] An SO may have certain required attributes and possibly
required values (i.e., specified, non-empty values), which
represent the definitional ("analytic") properties and relations of
the SO, and an arbitrary number of optional or additional
attributes and values (A-Vs), representing its contingent
("synthetic") properties and relations.
[0046] For example, traditionally, the notion "human" includes, by
definition, the property "mortal" and the attribute "age," even if
the value of age is not known in the case of a particular human.
However, the property "married" is not required for the SO to be
"semantically complete." In the case of constructed knowledge
representations, we are free to require arbitrary A-Vs in the SOs.
For example, we may require that any SO in our data repository
representing an employee must contain valid values for the
attributes "Name" and "Social Security Number."
[0047] SOs may have associated version numbers. When an SO is
updated, the system may increment the version number, in accordance
with standard industry practice and algorithms well known to
practitioners of the art, such that previous versions remain
accessible. The information associated with an SO may be stored in
distributed data structures. An SO's version number may be
incremented, for example, whenever any constituent component (or
its relationship to other components) is changed.
[0048] An SO that is self-complete and cannot be decomposed into
other SOs may be referred to as a primitive SO. A Complex Semantic
Object (CSO) is an SO that depends on or consists of other SOs for
its definition. A relatively straightforward example is a
collective entity. For example, "The Gang of Four," might be
represented by an SO that has an attribute "Composed-of" with
"value" given by a set of pointers to the four SOs representing the
individual members of the Gang of Four.
[0049] A more complicated example is an ontologically complex
concept, such as a "Sales Event." This might be represented by an
SO composed of one or more "Sales Activities," which, in turn, are
composed of one or more "Sales Actions." Each of the (nested)
substructures may be represented as distinct SOs in the system;
each of these SOs may be composed of or point to yet other SOs
(such as the SOs representing the location of the event, the
individuals who participated, etc.).
[0050] A property of an SO may be any of the valid structures of an
SO, including a system specific global unique identifier. As a
practical matter, properties of an SO may serve as a cover term for
any of the attributes, values, or relations of the SO, along with
the SO's identity.
[0051] A Presentation (P) is a software module expressing a set of
specifications of the format and other conditions necessary to
render an SO for viewing by a user, including, for each
presentation type, a list of the attributes and values (properties)
that are required and how the information associated with the
entities, attributes, and values is to be structured or combined
with other information for display. A P is capable of rendering
text, images, and other media types as required. Additionally, it
contains a set of modules for expressing the content of SAEs.
Minimally, a P would be capable of automatically generating data
repository queries for Value-type SAEs, using information stored in
the SAE or in data repository meta-data (e.g., the "key" property
of a given SO). A P could also contain a set of modules for
processing SAEs with named functional types.
[0052] With those terms defined, we can now turn to FIG. 2 and a
discussion of the architecture of a semantically anchored
expression (SAE). In FIG. 2, consider the local document 10. A user
has identified an image as Information Object 1 (IO-1) and a point
IO-2 immediately following the image. These information objects are
co-extensive with their surface regions. The surface regions are
coupled via links 12, 14, respectively, to semantic object number 1
(SO1) stored in a remote semantic object data repository 16. SO1
may be of semantic type "facility." See FIG. 3 for an example of
the coding to implement this coupling.
[0053] The user has identified the surface region from just before
the "R" to just after the "N" in "Raymond Mussman" as IO-3 and
coupled it via link 18 to SO4 in the repository 20. SO4 may be of
semantic type "PersonName." The discontinuous surface region
"stell" . . . "an!" is IO-6 that is coupled via link 20 to SO4. The
IO-6 may have a function-type ("Expletive") with supplementary data
"Verb=anstellen." See FIG. 3.
[0054] Finally, the surface regions that encompass "Gordon Yazzie"
IO-4 and "Yazzie" IO-5 are coupled via links 22, 24, respectively,
with SO2 which is an SO of the type PersonName. IO-4 is a Value
type and IO-5 is an Attribute-type with the property
"LastName=LAST." See FIG. 3.
[0055] The links 12, 14, 18, 20, 22, and 24 provide a coupling
between their respective surface regions and the SOs to which they
are coupled. The semantic object repository 16 may contain,
although it may be contained elsewhere, an inverted index 26. The
inverted index 26 is basically a table for providing an association
between each SO and each surface region which points to or is
coupled with that SO. In that manner, a coupling is created from
the SOs back to each surface region. This process of
bi-directionally coupling a surface region and an SO is referred to
as creating a semantically anchored expression.
[0056] Upon encountering a functional type SAE and verifying that
the module required for processing it is present, the P would
invoke the module to render the SAE. The function invocation
requires the region to be rendered, the SO(s) referenced by the SAE
(or retrieved as a result of query processing), and possibly
further refining parameters from the SAE properties or user input.
Because CSOs contain other SOs, this is a recursive process. For
example, FIG. 4 shows an ontologically complex CSO (an "Event")
that contains activities and plain text. The ontology specifies
that Events require date and location fields to be semantically
complete. Also, Activities require time and attendee fields, and
Activities are related to events (in this case, by simple
containment, though the ontology supports arbitrary relation
types). The semantic object repository schematic shows how CSOs are
built up from component SOs. Event type objects contain fields
required by the ontology, plus a set of Activity objects. These, in
turn, contain ontologically necessary fields, plus some optional
fields (at the discretion of the particular implementation).
Activities contain employee CSOs, which contain several fields
including a person SO. The SAEs link the event region, activity
region, and the name region to the Bob Person SO. It would also be
possible to link regions to the Employee CSO, or to the
Activity/Event CSOs. Regions could also be noncontiguous (e.g., the
activity region could "skip" an area of free text).
[0057] From the foregoing, we can conclude the following about
semantically anchored expressions (SAEs) in this embodiment. SAEs
are expressions created by users or automatically and displayed
through Ps. The appearance and behavior of SAEs under editing
reflect the linked SO(s) in a semantically coherent manner,
according to the relation expressed by the link(s), and constrained
by the context provided by the P. (Note: appearance includes the
"null" case where the SAE has no expression, visual or otherwise,
and editing behavior includes the case where user modifications are
prohibited.)
[0058] SAEs are coupled to one or more SOs by a persistent,
explicit link stored locally with the local document. These links
exist whether or not the document 10 in FIG. 2 is "open" or being
viewed. Furthermore, the coupling is bidirectional: links from the
document 10 to the repository 16 in one direction and by means of
the inverted index 26 in the other direction. The inverted index 26
makes it possible to locate all SAEs in all documents that link to
a particular SO, and all SAEs that link to a CSO containing or
otherwise ontologically related to a particular SO. In general, the
anchoring is accomplished through expressing the link in some query
language that can uniquely identify and retrieve an SO from the
data repository. For instance, the data repository may consist of
an RDB with XML integration, accessible through the SPARQL or
XQuery languages.
[0059] SAEs are semantic--the SO types, including their attributes
and relations, are ultimately derived from an ontology. Anchoring
ensures semantic identity, and the system ensures consistency of
reference across the entire document collection. One SAE is
referentially identical to another SAE if both SAEs are linked to
all and only the same SOs. One SAE is referentially similar to
another SAE if both SAEs are linked to at least one identical
SO.
[0060] An SAE has the following (minimal) structure:
[0061] a surface region (word, phrase, point, etc.)--as it appears
in the IO or document.
[0062] a link to one or more SOs (perhaps including the SO's
version).
[0063] a type (e.g., one of the set {Identity, Attribute, Value},
or an arbitrary Functional type whose behavior is determined in
large part by associated Ps). An Identity type SAE refers to
(stands for) the entire SO. Conceptually, it is the SO in the
context of a given document and P. An Attribute type SAE expresses
an indirect relation to some aspect of an SO (e.g., when the
reference is to the attribute itself rather than to its value),
while a Value type SAE directly expresses some part of an SO
(usually through generation of a surface form). A functional type
SAE performs custom processing as defined by a P.
[0064] an association to enable all SAEs that link to an SO to be
identified.
[0065] In general, the surface form, link, and type will be stored
locally with the IO or document. The association and the SOs will
generally be stored remotely.
[0066] When expressions are anchored semantically, changes to one
part of a document (or to the underlying data repository) may
propagate throughout the document collection in ways that are
unexpected and may seem "miraculous".
[0067] Some existing systems accommodate persistent links to
resources (usually within the document, but occasionally from
external sources). Changing the resource updates all the links
(e.g., Word's mail merge, document fields, or OLE objects).
However, bidirectional semantic couplings enable more powerful and
surprising operations. Bi-directional semantic couplings enable a
highly flexible indirection--typed links can refer to component
parts or specific interpretations of underlying data objects,
allowing the system to judge whether and how certain changes need
to be propagated. So a replace or update may propagate to only a
subset of the linked expressions, and may change their visual
representation or behavior in different ways.
[0068] Bi-directional semantic couplings are sensitive to context.
Because the referent has a rich underlying data model (based on an
ontology), a copy and paste operation into a constrained target
context (such as a table) may result in more data appearing in the
target than was present at the source--a surprising result, with
the extra data coming from the data repository.
[0069] Similarly, context sensitivity and indirection allow
information from multiple regions to be merged in a way that is
semantically consistent and coherent (with reference to an
ontology), and such that the structure and organization of the
resulting new content reflects constraints imposed by the
presentation (P).
[0070] Turning now to FIG. 5, the basic process of creating
semantically anchored expressions, i.e., the mechanism through
which the user links a point or region within a document to one or
more SOs in the data repository and further associates each
referenced SO back to the point or region in the document, will now
be explained. The steps of one embodiment are shown in FIG. 5,
although several alternative implementations will also be discussed
in this section.
[0071] The user selects at 110 a region having an IO in the
document corresponding to the surface form of the desired SAE. This
may be accomplished by highlighting or otherwise selecting a
demarcated IO, or by simply placing the cursor or otherwise
pointing to a point or location in the document (in the case of
SAEs with null surface forms), and indicating by some means (e.g.,
context menu selection) the intent to create an SAE.
[0072] Alternatively, the system may nominate regions for SAEs
using grammars, rules, patterns, among others, (generally referred
to as Resources) and indicate the regions to the user through some
form of user feedback (e.g., green squiggly lines under the surface
expression of the IO). The operation would proceed upon indication
or confirmation of intent to create an SAE.
[0073] Having selected a surface region for an SAE within the
document, the user must then select at 120 the type of the semantic
object(s) to which the SAE will point. If the SAE is to point to
more than one SO, the SO types must be identical or at least
compatible. An SO (SO1) is "type-compatible" with another SO (SO2)
if (a) the constraints on the attributes of SO are completely
encompassed by those of SO2, and one of (b) both SOs are of the
same semantic type, or (c) SO1 is of a semantic type that is
subsumed by that of SO2 in an Ontology or vice versa, or (d) b or c
applies to SO1 and a component SO of SO2.
[0074] It is possible for the user to wish the target of the SAE to
be an SO that does not currently exist in the data repository. For
instance, the user may be a salesperson identifying a new customer
unknown to the system. If the user has permission to add SOs to the
data repository, the system prompts at 130 for this. The user
creates the new SO at 140 and, if the user needs to create more
than one new SO, a loop from 140 to 130 is traversed until all new
SOs have been created.
[0075] If the data repository access model does not permit
modifications, steps 130 and 140 may not be present in some
implementations, in which case the system would proceed directly to
150 to query the data repository for existing matching SOs.
[0076] Also and independently, some implementations might select
the SO type(s) after querying the data repository, in which case
step 120 could follow 130 (or 150 if the data repository is
read-only).
[0077] Given an SO type (or set of compatible types), the system
nominates at 150 a candidate set of SOs from the data repository,
and prompts the user to select one or more of the candidate SOs.
This set may contain newly created SOs as well as existing data
repository entries. In some implementations, if the user created
new SOs, the set may be limited to those created in 140.
[0078] In the preferred implementation, the application would query
the data repository once (perhaps upon initial connection) to
determine possible SO types, organize the SOs for ease of display
(e.g., into a hierarchical menu), and use Resources to examine the
surface expression of the IO (if any), using that information to
further narrow down the list of SO types. (For instance, if the
user identified the text string "John Smith" as the surface
expression of the SAE, and the system identified that string as a
person name, the system might suggest the PersonName SO type.)
[0079] It is also possible for an implementation to additionally
query the user for the SAE type and supplemental properties, then
use that information to narrow the field of potential SOs. For
instance, inserting a Value SAE that expresses the middle initial
of a PersonName might cause the system to promote PersonName SOs
with known middle initials over those with unknown middle
initials.
[0080] After one of the foregoing scenarios is carried out, the
system presents the user at 160 with a list of SOs. If the desired
set of SOs is not in the repository, it is not possible to create
the SAE, so the process stops. Otherwise, the user selects at 170
one or more SOs to associate with the SAE.
[0081] At this point, the user has identified the location (and
optional surface expression) of the SAE, and the SOs to associate
with it. Next, the user specifies at 180 the type and properties of
the SAE. The type may be one of the set {Identity, Attribute,
Value} or a user-defined functional type (with an arbitrary name).
Depending on the logical requirements of the SO and SAE types, the
user may also specify properties: a set of arbitrary attributes and
values providing additional information required to express the SAE
according to a P. For instance, an Attribute or Value type SAE
might specify the name of the SO attribute to be expressed.
[0082] At this point, the system has all the information it needs
to create at 190 the local portion of the SAE, a structured text
specification (e.g., in XML) that is stored persistently with the
document. This creates the coupling between the surface region and
the SO in the data repository. However, the system must also
associate at 195 each referenced SO in the data repository back to
the surface region. That may be accomplished through a type of
inverted index, which must minimally associate with each SO a list
of pointers to SAEs within documents (e.g., through XPointer/XPath
for XML documents), and the SAEs' type. This information enables
efficient implementation of semantic operations such as
Replace/Update and Merge.
[0083] Turning now to FIG. 6, FIG. 6 describes the basic process of
displaying or otherwise expressing semantically anchored
expressions according to a P. The steps of a preferred embodiment
are shown in FIG. 6, although alternative implementations will be
apparent to practitioners of the art.
[0084] Through a given P, the system attempts to display or
otherwise express at 210 an IO associated with an SAE. Note that
this expression may be non-visual (i.e., behavioral) in some
applications. For example, the system may respond with a beep or
popup form whenever the user hovers over a region associated with a
person SO.
[0085] The system next determines at 220 the type of the SAE. If
the type is one of {Identity, Attribute}, the surface form of the
SAE is displayed at 230. Otherwise, the SAE is either a Value or
special Functional type, and the system requires information from
the repository to express the SAE. If the SAE is a Functional type
as determined at step 240 (i.e., has a name that does not match one
of the set {Identity, Attribute, Value}), the system checks at 250
to see whether the P recognizes the type. If the function name is
unrecognized by the current P, the system can only make a default
interpretation at 260. This is implementation-dependent; options
include displaying the surface form of the SAE, displaying an error
message or placeholder, ignoring the SAE, etc.
[0086] If the SAE is a Value type or a Functional type, the system
retrieves at 270 the indicated information from the SO(s) in the
data repository. If it is a Value type as determined at 280, the
value of the indicated SO attribute is expressed at 290. An example
would be the LastName field of a PersonName SO. If it is a Function
type, the system invokes the function on the P over the SO(s),
using any supplied properties as arguments at 295.
[0087] Those of ordinary skill in the art will recognize that
numerous alternative embodiments are possible such that there is no
criticality to the ordering of most of the steps in FIGS. 5 and
6.
[0088] A system 40 is illustrated in FIG. 7 within which the
methods disclosed herein may be practiced. The system 40 consists
of local workstations 50 through n. The workstations 50, n
communicate with a document repository 60 and a SO data repository
70 through a communication network 80. The document repository 60
may be any type of storage device for storing documents 10 of the
type illustrated in FIG. 2. The data repository 70 is a mechanism
that records and maintains information (whether structured or
unstructured) related to or derived from the IOs that are processed
by the system. The data repository 70 may be realized operationally
as a number of different data-storage devices or structures. For
example, the data repository 70 may encompass a discrete SO
repository along with an inverted index that maintains such
information as where references (associations) to specific SOs are
located in various IOs in Documents. The SO repository may combine
structured text storage (e.g., XML) with traditional relational
tables. The network 80 may be any type of LAN, WAN, the Internet,
etc. as circumstances dictate.
[0089] The terms "local" and "remote" may be defined with reference
to FIG. 7. For example, workstations 50, n might be located in
adjacent offices, and the document repository 60 and data
repository 70 might be on the same floor. Alternatively,
workstation 50 might be located in Pittsburgh, workstation n might
be located in Philadelphia, with local document repositories
resident on each of the workstations and a data repository 70
located in Toronto. Similarly, the software for creating the SAEs
and displaying SAEs may be distributed within the system 40. Those
of ordinary skill in the art will recognize that the configuration
of any particular system 40 will depend in large measure on the
current resources and assets of the particular enterprise in
question.
[0090] Semantic Replace and Semantic Update
[0091] There are a variety of ways to change information in an
existing semantic document, but they can be reduced to semantic
replace and semantic update. The semantic replace operation
consists of switching the link between an SAE and an SO to a
different SO. The semantic update operation consists of changing
the value of a particular attribute of an SO. Note that if the
value is itself an SO, this may effectively result in a semantic
replace operation. Let us consider the ways in which these two
operations can be invoked.
[0092] Semantic replace may be invoked when the user selects an SAE
and chooses to replace one of its SOs with another. This may or may
not involve a change to the surface form of the SAE. For example,
the "John Smith" mentioned in document A may not be the same "John
Smith" mentioned in document B. In this case, the user may wish to
replace SO["John Smith", 01] with SO["John Smith", 02] in document
B. That change would not result in any changes to the surface
expression of the SAE. Additionally, this change may need to be
made just once, everywhere in document B, or in a variety of places
throughout the document collection. Alternatively, the user could
directly query to the SO repository and indicate the desire to
replace one SO with another. In either case, the change may need to
be propagated, based on user preferences or system defaults, to
other SAEs linked to the same source SO in the replace
operation.
[0093] Semantic update may be invoked when the user selects an SO
and changes the value of an attribute. If the attribute has a
simple type, it is only necessary to verify that all the SAEs with
a value relation to that SO are consistent with the new value. If
the attribute is itself a semantic object, the semantic replace
operation is invoked on all SAEs with the appropriate attribute or
value relation to that SO. A semantic update may also be invoked
when the user changes the surface form of an SAE that has a value
relation to an SO.
[0094] If an SAE has more than one SO and/or the replace operation
is targeted for more than one SO, then a more complex semantic set
replace operation (discussed below) is required.
[0095] For any change to a semantic document repository, the scope
of the operation should be defined. A typical scope might be a
single SAE only, all referentially identical SAEs in one document,
or all referentially identical SAEs in the document repository. The
scope may be determined manually, automatically, or in a
semi-automatic manner. The scope might also be based on the type of
the SAE. For example, if the user is updating the value of a
particular attribute in an SO, one might typically expect that all
SAEs of matching attribute or value type will automatically be
within the scope. The user might also wish to look at SAEs with an
identity type to make sure that the surrounding text is consistent
with the updated SO. For example, consider a document containing a
list of salespeople, including the SAE:[John Smith], linked with an
identity relation to the SO:[John Smith]. If John Smith is
promoted, and his job title attribute is changed to Sales Manager,
then the user may wish to remove him from this list. Of course, a
better way to solve this problem would be to build the list using a
P that filters for people with the salesperson job title. In that
case, the change would happen automatically at runtime. However, we
cannot guarantee that all information objects in semantic documents
are constructed in the best possible manner.
[0096] In a traditional text replace operation, scope is limited to
a single document. Even in this case, the user may be required to
examine and approve hundreds of individual changes. This problem is
magnified in a semantic document repository, as a single semantic
object may be linked to hundreds or thousands of SAEs across many
documents. Therefore, it is likely that a fully functional semantic
document processing system will have a sophisticated interactive
scope selection environment that will enable the user to make
high-level decisions about where to apply a change without having
to view each SAE individually. This environment might summarize the
linked SAEs for a given SO according to a variety of parameters,
including: surface form, document type, document age, document
directory, SAE type, or any number of customized parameters.
[0097] Furthermore, a fully functional semantic document processing
system will have some mechanism for permissions and document access
control. Most users will not have permission to modify every
document in the repository. Therefore, it is important to introduce
several additional concepts: semantic object versioning and delayed
replacement.
[0098] Let us assume that the user wishes to replace person A with
person B everywhere in the document repository because person A has
left the company, but the user does not have permission to modify
all the documents. In this case, instant replacement is limited to
a subset of SAEs and the remaining SAEs are marked for delayed
replacement. Delayed replacement means that the linking change is
delayed until the next time a user with permission to change the
document actually opens the document. The pending replace operation
is cached somewhere in the system. Until the replace is completed
(or rejected), other users may be given a cue that there is a
pending replace for that particular information object. Delayed
replacement could be applied to any document, not just to those
with a read-only status for a given user.
[0099] In a semantic update operation, a similar problem with
access control arises. This can be handled by semantic object
versioning. Any change to a semantic object may (e.g., depending on
user settings) result in a new version being created. Typically, an
SAE will point to the latest version of a semantic object, but it
might temporarily point to an older version until an authorized
user approves the update operation. In some cases (e.g., historical
published documents) the SAE may permanently point to an older
version of the semantic object. Users can create a published
version of any individual document at any time that simply freezes
the version numbers of all links to semantic objects in the
document.
[0100] FIG. 8 is a flowchart illustrating one embodiment of a
semantic replace operation according to the teachings of the
present disclosure, although several alternative implementations
will also be discussed in connection with FIG. 8. FIG. 9 is an
example that schematically illustrates the embodiment of semantic
replace according to the flowchart of FIG. 8.
[0101] The user selects at 310 an SAE in the document, with the SAE
being linked to a first or source SO. The user selects at 320 a
second or target SO from the data repository with the goal of
replacing the source SO with the target SO according to some scope,
which the user may be prompted at 330 to select. The scope is
either selected by the user at 335 or determined automatically by
the system at 340. Alternatively, the system starts with a default
scope that is further refined by the user. Some common scopes
include: this SAE only, all SAEs in this document, or all SAEs
linked to this SO in the document repository. The scope defines a
set R of SAEs eligible for replacement and is further filtered at
350 to include only those SAEs referentially identical to the
selected SAE. SAE selection 310, target SO selection 320, and
(optional) manual scope selection 335 may be performed in any
order. If manual scope selection precedes SAE selection, then SAE
selection may not be necessary. SAE selection 310 may be
accomplished by choosing the SAE directly or by selecting a region
of the document, seeing the SAEs overlapping with that region, and
then picking one of those SAEs. The target SO 320 may come from the
data repository or it may be created on-the-fly by the user.
[0102] At this point, we have a source SO, a target SO, and a set R
of SAEs pointing to the source SO. We now iterate at 360, 370
through the set R of SAEs and execute a replace operation 390 which
replaces target SO for the source SO for the SAE in issue. The user
may be asked to accept or reject each replacement at 380 or this
decision may be made automatically by the system. Furthermore, the
decision may be manual for some SAEs and automatic for other SAEs
based on some features of each individual SAE.
[0103] While steps 360, 370 demonstrate an iteration mechanism that
removes elements from the set, any iteration mechanism that returns
each element of the set exactly once can be used in this phase. The
actual replace operation may be executed immediately as it is
approved or the intention to replace may be cached and the actual
execution may occur in one or more batches either during or after
iteration is completed. The replace operation on the initial SAE
may be executed immediately (e.g., any time after 310 and 320 but
before 360) or the selected SAE may be included in the set R and
replaced during the normal iteration sequence. Steps 360, 370
demonstrate an iteration mechanism over individual SAEs. Iteration
may also be implemented over one or more groups of SAEs. In this
case, the decision to replace 380 may be made either manually or
automatically for the group as a whole. For example, the SAEs may
be grouped by surface form. The replace function 390 has three
arguments: an SAE, a source SO, and a target SO. The replace
function changes the SAE link from the source SO to the target SO
and updates the SO/SAE association table.
[0104] FIG. 10 is another example of semantic replace. In this
example, "Mark Chen" has been replaced with "Jennifer Chu." Because
the user replaced a semantic object and the gender is different,
all SAE's linked to that semantic object that are no longer
consistent will change accordingly. The change is made based on the
gender attribute of the entity in the semantic object
repository.
[0105] FIG. 11 is a flowchart illustrating one embodiment of a
semantic update operation according to the teachings of the present
disclosure. FIGS. 12A and 12B are an example that schematically
illustrates the embodiment of semantic update according to the
flowchart of FIG. 11.
[0106] The user begins by selecting at 410 an SO. At 420 the user
changes the SO, typically by changing the values of one or more the
attributes of that SO or by adding new attributes. The user is
prompted at 430 to select a scope. At 435 the user may manually
select a scope or the scope may be automatically selected by the
system at 440. Alternatively, the system starts with a default
scope that is further refined by the user. The scope defines a set
R of SAEs linked to the SO that are eligible for update and will
typically include those SAEs that have an attribute or value
relation with at least one of the changed attributes in the SO.
Steps 410, 420, 430, 435, 440 can be completed in many different
orders. For example, attribute selection 420 may follow scope
selection 430, 435, 440. Scope selection may include SO selection,
in which case 410 is no longer required.
[0107] At this point, we have an updated SO and a set R of SAE's
linked to that SO. We now iterate 460, 470 through the set R of
SAEs. At 480 a determination is made whether the SAE is consistent
with the updated SO, and, if the SAE is no longer consistent, an
update function 490 is executed to update the SAE and make it
consistent with the changed SO. An SAE in a value relation with the
SO is not consistent if the surface form does not satisfy the
constraints of the attribute. The constraints may take a number of
forms, such as, but not limited to: exact match to a string value,
membership in a set, or numeric value in a certain range.
Consistency testing and updating may be automatic in some cases and
manual in other cases, depending on the nature of the attributes
and its constraints, or user preference. The update function 490
changes the surface form of the SAE in such a way that it satisfies
the attribute constraints of the linked SO.
[0108] Replace/update operations are potentially recursive, and the
surface form reconciliation process must take into account a wide
range of possible data types within complex SO structures, as well
as display and behavioral constraints specific to presentations. It
is therefore conceivable that an SO might be updated in such a way
as to preclude consistency with one or more SAEs. In this special
case, the system might disable the link with notification, perhaps
prompting the user to delete it. One possible implementation would
define a common function type for "invalid" or "expired" SAEs, and
change the SAE type to this value. Presentations could then
interpret these SAEs in specific ways; e.g., ignore them, highlight
them, etc. Changing the SAE type locally in the document also
implies an update of the data repository (which stores the SAE
types in its inverted index). This in turn has implications for
various semantic operations (e.g., for propagation of
replace/update operations; the system would likely not follow
"expired" links).
[0109] While steps 460, 470 demonstrate an iteration mechanism that
removes elements from the set, any iteration mechanism that returns
each element of the set exactly once can be used in this phase. The
actual update operation may be executed immediately or the
intention to update may be cached and the actual execution may
occur in one or more batches either during or after iteration is
completed. Steps 460, 470 demonstrate an iteration mechanism over
individual SAEs. Iteration may also be implemented over one or more
groups of SAEs.
[0110] FIG. 13 is a flowchart which illustrates a more general
version of semantic replace that supports the replacement of
multiple source and target semantic objects. In the discussions of
semantic replace so far, it has been assumed that the SAE was
linked to exactly one source SO that was being replaced by exactly
one target SO. In semantic set replace, the SAE may be linked to
more than one SO, or the replacement target may be more than one
SO, or both conditions may hold. The user selects at 510 an SAE in
the document, and then chooses at 520 a non-empty subset S of
source SOs linked to the SAE and a non-empty subset T of target
SOs. In this operation, it is assumed that the cardinality of at
least one of these sets (if not both) is greater than one to
differentiate from the basic semantic replace operation.
[0111] In response to a prompt at 530 to select a scope, the scope
of the operation is either selected at 535 by the user or
determined automatically at 540 by the system. Alternatively, the
system starts with a default scope that is further refined by the
user. The scope defines a set R of SAEs eligible for replacement
and is further filtered at 550 to include only those SAEs
referentially similar to the initial SAE. Source SAE selection 510,
target SO selection (second part of 520), and (optional) manual
scope selection 535 may be performed in any order. If manual scope
selection precedes SAE selection, then SAE selection may not be
necessary. SAE selection 510 may be accomplished by choosing the
SAE directly or by selecting a region of the document, seeing the
SAEs overlapping with that region, and then picking one of those
SAEs. The target set T of SOs 520 may come entirely from the data
repository or one or more may be created on-the-fly by the
user.
[0112] At this point, we have a set S of source SOs, a set T of
target SOs, and a set R of SAEs pointing to at least one of the
source SOs. We now iterate 560, 570 through the set R of SAEs and
execute a set replace operation 590 which takes the SAE, the set T,
and elements of set S linked to the SAE. The user may be asked at
580 to accept or reject each replacement or this decision may be
made automatically by the system. Furthermore, the decision may be
manual for some SAEs and automatic for other SAEs based on some
features of each individual SAE.
[0113] While steps 560, 570 demonstrate an iteration mechanism that
removes elements from the set, any iteration mechanism that returns
each element of the set exactly once can be used in this phase. The
actual set replace operation may be executed immediately as it is
approved or the intention to replace may be cached and the actual
execution may occur in one or more batches either during or after
iteration is completed. The set replace operation on the initial
SAE may be executed immediately (e.g., any time after 510 and 520
but before 560) or the initial SAE may be included in the set R and
replaced during the normal iteration sequence. Steps 560, 570
demonstrate an iteration mechanism over individual SAEs. Iteration
may also be implemented over one or more groups of SAEs. In this
case, the decision 580 to replace may be made either manually or
automatically for the group as a whole. For example, the SAEs may
be grouped by surface form. The set replace function 590 has three
arguments: an SAE, a set of source SOs, and a set of target SOs.
The set replace function removes links to the source SOs and adds
links to the target SOs.
[0114] Semantic Copy and Paste and Semantic Cut and Paste
[0115] Turning now to FIG. 14, the steps of a preferred
implementation of a semantic copy and paste operation are shown.
The user selects at 605 a Source Region (SR) in the IO, for
example, a document. Selection may be either manual or via some
automated process. The system determines at decision step 610
whether SAEs are present in the SR and, if so, identifies at 615 a
unique set, U, of SOs that the SAEs are linked to. This identifying
may be performed based on existing mark-up or, alternatively, a
process may be run using Resources. As a practical matter, this may
involve the look-up of SAEs in a table (index) or the sorting of
the link references on the SAEs in the SR.
[0116] The system then identifies at 620 a set, S, of Ps in a Menu
Library ML that are referentially compatible with the SOs in the
set U. This involves identifying the unique set of types of
properties of the SOs in the set U and comparing these with the
required and optional types for each of the Ps in the ML.
[0117] A Menu (M) is a display that lists the actions that may be
performed by the system given (a) the contents of a buffer
(possibly null), (b) a location in the local document(e.g., point
in a document or data structure), and (c) a set of operations the
system can perform. Menus may be designed to be "fixed" in their
location (e.g., as in an item in a menu bar) or dynamic (as in a
"pop-up" presentation). A menu may include auditory presentations.
The Menu can be invoked in a variety of ways (well represented in
contemporary systems). As a practical matter, the Menu will list
the types of "paste" actions that a user can request the system to
perform.
[0118] The Menu Library (ML) is a set of Ps reflecting the
structure and display characteristics of the data on which Menu
actions can be performed. An example of a P in the ML might be the
specifications for the presentation of a list of specific types of
items; or a list that displays a specific set of Properties of SOs;
or a table with rows and columns filled in with particular types of
information; or a graph of a particular type (e.g., pie chart)
where the input values derive from a function on Properties of SOs
of particular types; etc.
[0119] A P is "referentially-compatible" to one or more SOs if and
only if (a) all the required, (b) any optional, and (c) none of the
prohibited attribute/value/property types of the P are present in
at least one of the SOs
[0120] Returning to FIG. 14, if there are no SAEs in the SR as
determined in step 610, then the system does not attempt to select
Ps from ML. The user then selects at 625 a Target Region (TR) into
which the copied material will be pasted. Again, the selection can
be either manual or through some automated process. Note that the
TR may be a point in an IO or may be a span of an IO or may be an
existing structured object. The system determines at 630 whether
the TR is a structured object and, if so, removes at 635 from the
set S any P that is not expressible in the structured object. If
the TR is not structured, there is no need to modify the set S.
[0121] At this point, the system is prepared to enable the choices
in the menu along with the associated required actions for the Ps
in the set S as shown at 640. In the menu, the choices
corresponding to the Ps may be organized, e.g., hierarchically in
cascading sub-menus, for more efficient display. Note that the set
S may be empty in step 640 as a result of incompatibility of the Ps
in the set S with the TR and the filtering of the set S in step
635. The set S may also be empty because there were no SAEs in the
SR as determined in step 610. In the event that the set S is empty,
the system indicates that no semantic copy operation is possible.
However, the system may be configured to perform one or more
default non-semantic copy operations, provided they are compatible
with the TR. Based on the available operations, including defaults,
choices are displayed at 645 in the menu.
[0122] There are a variety of techniques in common practice for
making the menu available to the user, including having the user
navigate to a fixed location in a menu tab or having the user
invoke a pop-up display of the menu through an action such as
depressing a mouse button. The disclosed method does not depend on
any particular method. When the user indicates at 655 which
operation to perform, the system executes the operation in the TR
at step 155. Execution involves a process in which the required
attributes/values/properties for display (insertion) are retrieved
from the SOs and presented in the format specified by the P
(possibly determined, in part, by the P or Ps or other
features/constraints in the context that scope over the selected
TR).
[0123] Note that the steps 610, 615, and 620 (designated "A" in
FIG. 14) could be performed after step 625 (designated "B" in FIG.
14) without loss of functionality. The selection of an SR provides
the system with information about the contents that will be subject
to a paste operation and the selection of a TR provides the system
with information about the location where the paste operation is to
be performed and the constraints, if any, on the operation. The SOs
in the SR can be discovered and the characteristics of the TR can
be determined after both regions have been selected.
[0124] The steps for semantic copy and paste as described above can
also provide the functionality required for semantic cut and paste.
The difference is that, upon execution of the paste operation, the
system deletes from the IO the contents of the SR.
[0125] The process illustrated in the flowchart of FIG. 14 is
illustrated schematically in FIGS. 15 and 16. The process shown in
FIGS. 15 and 16 is the embodiment where B in FIG. 14 is performed
before A. FIGS. 15 and 16 illustrate examples of what the source
region, target region, semantic paste menu and completed document
after the semantic copy and paste operation are completed might
look like.
[0126] FIG. 17 is a simplified view of the process shown in the
flowchart of FIG. 14 with examples of what the various options in
the menu might look like.
[0127] FIG. 18 illustrates the effect of a semantic copy and paste
operation in which copied material expresses previously unavailable
information when pasted into a target region. The "before"
representation represents the user's selection of the SR in the IO
(step 605 in FIG. 14). The "after" representation represents what
the TR looks like after the semantic copy and past operation is
completed. Note the disambiguation, uniquely identified persons,
and "discovery" of new information. Even with a detailed
understanding of the semantic underpinnings, the "after"
presentation is clearly a surprising result.
[0128] Semantically informed text operations require maintenance of
the links between surface regions and semantic objects, both in the
local documents where the surface regions appear and in the remote
semantic object repository. Paste operations generally require the
creation of new semantically anchored expressions in the target
region; the system would copy the type, properties, and link(s) to
form the new SAEs, while altering their surface region
specifications to match the target location. At runtime, the system
would interpret the SAEs in the new location such that their
surface expressions would usually match that of the source, though
in general the surface expression of copied SAEs might be different
due to local presentation constraints (e.g., copying data from a
free text region into a structured table). Cut operations generally
require the deletion of content from the source region, including
SAEs with surface regions that fall within its boundaries. (SAEs
that are discontinuous or otherwise have only partial extension
within the source region are special cases that must be handled
separately, perhaps by truncating their associated surface
regions.) Thus, those of ordinary skill in the art will recognize
such "housekeeping" matters are necessary for the system to keep
track of the location and changes in location of the surface form
of the SAE. Such matters are well within the skill of those of
ordinary skill in the art and therefore need not be further
discussed.
[0129] Semantic Merge
[0130] An advantage of a merge operation informed with semantic
information is that, when the user chooses multiple sources to
merge, the document management system will try to identify the
semantic relevance of the sources and merge together those parts of
the sources that are semantically the most relevant. The merge
result will also be formatted with respect to the constraints of
the target region. Such an operation is more refined and results in
merged content that is semantically more coherent than that derived
from simple appending, and avoids manual adjustment by the
user.
[0131] Semantic merge is invoked when the user selects a number of
source regions (which can be whole documents) and a target region.
The system will first identify SOs in the target region that other
SOs can be merged into, and then iterate through each such SO to
retrieve type-compatible SOs in the source regions and position
them at the right locations in the target SO. Finally the target
region SOs will be formatted and displayed under the constraints of
the target region. The source regions can be of three types as
listed in the table below, or any combinations of them. The target
region can also be any of the three types or combinations of
them.
TABLE-US-00001 TABLE 1 Merge different types of document regions
Free Text Semi-Structured Document encompassing Complex Semantic
Complex Semantic Regions Primitive SOs Object (CSO) Object (SCSO)
Free Text CSO x SCSO x
Table 1: Merge different types of document regions
[0132] In general, a complex SO cannot be merged into a primitive
one, and merging different types of regions is subject to the
constraints in the target region.
[0133] FIGS. 19 and 22 are flowcharts describing two basic semantic
merge operations according to the present disclosure. The
embodiment shown in FIG. 19 is more restrictive and does not give
the user as many choices as the embodiment in FIG. 22. In FIG. 19,
the user selects at 710 a number of Source Regions (SRs) and a
Target Region (TR) in an information object or document with the
goal of merging the content of the SRs and putting the results in
the TR. The target region can either be one of the source regions,
or be a separate region from the selected source regions. The
system will automatically identify at 720 all the semantic objects
(SOs) encompassed by the target region. If there is no SO in the
target region, the region will be of minimum structure or
unstructured as determined at 730. In this case, a default merge
process 735 will be executed, as described below in conjunction
with FIG. 20.
[0134] If on the other hand the target region contains at least one
SO, the system will then check at 740 to determine if the target
region contains a Complex SO (CSO). If not, that means that only
primitive SOs are present and primitive SOs do not allow other SOs
to be merged into them. In that case, the semantic merge process
will end and a default process may be applied, such as a simple
append of the SR to the TR such as is discussed below. When there
is at least one Complex SOs in the target region, the system will
determine at 750 which SOs to merge. The user may select at 755 a
list of SOs, or a default strategy will create at 760 a list of all
SOs in the target region. The system then iterates through the list
of SOs, retrieving at 775 the next SO from the list and performing
a merge operation at 780 on the retrieved SO as described below in
conjunction with FIG. 21, until the list is empty as determined at
decision step 770. Finally, the system formats the merged SOs with
respect to the presentation specifications of the target region,
and presents the merged document to the user at 790.
[0135] Turning to FIG. 20, the steps of the default merge process
735 are shown. This process creates at 810 a list of all SOs in the
source regions. It then iterates through the list, retrieving at
840 the next SO from the list, and appending the retrieved SO at
the end of the previous SO at 850, until the list is empty as
determined at 830. Finally it returns at 860 the SO that contains
all appended SOs in the source regions.
[0136] Turning to FIG. 21, the steps of the merge process 780 of
FIG. 19 are shown. This is a general sub-process that is called
upon by other processes to perform the actual merge. This process
takes a target SO, a number of source regions, and a target region
as parameters, and tries to merge compatible SOs in the source
regions into the target SO. The system first checks at 910 if the
target SO is a Complex SO. If not, it will return at 990 the target
SO without merging anything into it. If yes, the system finds at
920 a list of all identical or type-compatible SOs from the source
regions. If the list is empty as determined at decision step 930,
the system returns at 990 the target SO, again without merging.
Otherwise, it iterates through the list, retrieves at 940 the next
SO from the list, and checks at 950 if this is a Complex SO.
Depending on the type of the SO, the system will either merge at
955 the sub-components of the two Complex SOs, or append at 960
this SO into the sub-components of the target SO. An optional step,
step 970, determines if the TR is null. The above steps iterate
until the list is empty.
[0137] The steps of a second embodiment are shown in FIG. 22.
Similar to the first embodiment, the user selects at 1210 a number
of source regions and a target region, and the system will identify
at 1220 all SOs encompassed by the target region. If the system at
1225 finds no SO in the region or finds at 1230 no Complex SO in
the region, the system will execute a source merge process 1240
(described below in conjunction with FIG. 23), rather than the
default merge process 735 as in the first embodiment.
[0138] If there is at least one Complex SO in the target region,
the system will order the list of all SOs in order of their
occurrence at 1235. When there is only one SO in the list, the
system retrieves at 1265 the first SO and executes the merge
operation 780 (see FIG. 21) on it. When there is more than one SO
in the list, the system will query the user at 1255 for the type of
merge to perform. The user can choose among Merge First, which is
the same as the previously described merge, Merge Select, which is
the same as that described in the first implementation, and Merge
All. For this third choice, the system will iterate beginning at
1275 through the list of SOs, retrieve at 1280 the next SO from the
list and perform the merge operation 780, until the list is empty
as determined at 1275. Finally, the system formats the merged SOs
with respect to the presentation specifications of the target
region, and presents at 1290 the merged document to the user.
[0139] Turning to FIG. 23, the steps of source merge process 1240
are shown. This process queries the user at 1310 for the type of
semantic merge to perform. The user can choose between the default
merge process 735 (See FIG. 20), or an ordered merge at step 1320.
For this second choice, the system will find at 1330 the list of
all SOs in the source regions, and order at 1340 the list of SOs by
their complexity. That is, a Complex SO will be ranked higher than
a Semi-structured Complex SO, which in turn is ranked higher than a
primitive SO. When two SOs are of the same complexity, ties may be
broken by any of a number of methods, such as by the size of the
surface regions that are associated with the SOs. The system then
treats the highest ranked SO as the target SO at step 1350, removes
at 1360 this SO and its associated source region from the lists,
and executes at 1370 the merge operation on this SO. Finally the
process will return the merged SO at step 1390.
[0140] FIGS. 24, 25A, 25B, and 26 illustrate schematically various
examples of the merge processes. FIG. 24 illustrates a merge
between two semi-structured complex semantic objects. FIGS. 25A and
25B illustrates a merge of a textual document into a
Semi-Structured CSO. FIG. 26 illustrates a merge of two CSOs.
[0141] The reader will recognize that the flowcharts presented
herein do not reflect all possible conditions and circumstances
that may arise when performing the various methods. Those of
ordinary skill in the art will recognize that additional steps,
procedures, etc., may be required to enable the methods to be
practiced in a manner capable of dealing with atypical
situations.
[0142] While the present invention has been described in
conjunction with preferred embodiments thereof, those of ordinary
skill in the art will recognize that many modifications and
variations are possible. For example, the present invention may be
implemented in connection with a variety of different hardware
configurations. Additionally, actions such as "select",
"determine", "define", "retrieve", "remove", etc., should be
understood broadly, and be understood as being capable of being
performed manually by a user, in an automated manner by the system,
or some combination of both. Also, the reader should understand the
results of "selecting", "determining", "defining", "retrieving",
"removing", etc., may result in a zero or null result. Such
meanings, modifications and variations fall within the scope of the
present invention which is limited only by the following
claims.
* * * * *