U.S. patent application number 11/638253 was filed with the patent office on 2007-04-26 for method for managing real-time data history of a file system.
This patent application is currently assigned to Asempra Technologies, Inc.. Invention is credited to Siew Yong Sim-Tang.
Application Number | 20070094312 11/638253 |
Document ID | / |
Family ID | 35376451 |
Filed Date | 2007-04-26 |
United States Patent
Application |
20070094312 |
Kind Code |
A1 |
Sim-Tang; Siew Yong |
April 26, 2007 |
Method for managing real-time data history of a file system
Abstract
A data management method is provided for storing a real-time
history of a file system, or a component thereof, such as a
directory or a file. The real-time history is stored as an
object-oriented logical representation comprising at least a set of
version metadata objects, and a set of one or more links that
associate given objects of the set of version metadata objects. As
one or more events occur in the real-time history, the logical
representation is restructured dynamically. The logical
representation is useful to provide any point-in-time
reconstruction of the file system component on an as-needed
basis.
Inventors: |
Sim-Tang; Siew Yong;
(Saratoga, CA) |
Correspondence
Address: |
LAW OFFICE OF DAVID H. JUDSON
15950 DALLAS PARKWAY
SUITE 225
DALLAS
TX
75248
US
|
Assignee: |
Asempra Technologies, Inc.
|
Family ID: |
35376451 |
Appl. No.: |
11/638253 |
Filed: |
December 13, 2006 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
11123994 |
May 6, 2005 |
|
|
|
11638253 |
Dec 13, 2006 |
|
|
|
60569164 |
May 7, 2004 |
|
|
|
Current U.S.
Class: |
1/1 ;
707/999.204; 707/E17.01 |
Current CPC
Class: |
G06F 11/3495 20130101;
G06F 2201/86 20130101; G06F 16/125 20190101 |
Class at
Publication: |
707/204 |
International
Class: |
G06F 17/30 20060101
G06F017/30 |
Claims
1. A data management method, comprising: storing a real-time
history of a data set, or a component thereof, as a logical
representation, the data set having a hierarchical data structure,
wherein the logical representation comprises at least a set of
version metadata objects, and a set of one or more links that
associate given objects of the set of version metadata objects; and
reconstituting the logical representation in response to a new
event in the real-time history.
2. The data management method as described in claim 1 wherein the
data set having a hierarchical data structure is a file system.
3. The data management method as described in claim 2 wherein the
component is a component of the file system and is one of: a
directory, and a file.
4. The data management method as described in claim 1 wherein the
set of one or more links comprise at least one of: a version link,
a parent-child link, and a data link.
5. The data management method as described in claim 2 wherein the
new event is one of: a file create, a file modify, a file delete, a
file rename, and a file relocate.
6. The data management method as described in claim 1 wherein the
new event is one of: a directory create, a directory modify, a
directory delete, a directory rename, and a directory relocate.
7. The data management method as described in claim 1 wherein the
logical representation is n-dimensional.
8. The data management method as described in claim 1 wherein the
set of version metadata objects conform to an object hierarchy.
9. The data management method as described in claim 8 wherein the
object hierarchy comprises a set of data objects, wherein each data
object in the set of data objects conforms to a given schema.
10. The data management method as described in claim 9 wherein the
given schema includes at least one of: a directory class schema,
and a file class schema.
11. The data management method as described in claim 1 further
including: generating the real-time history; and transferring the
real-time history from a first location to a second location remote
from the first location.
12. The data management method as described in claim 11 wherein the
first location is a data source having an associated host driver
and the second location is a data store.
13. The data management method as described in claim 12 wherein the
real-time history provides a data protection service.
14. The data management method as described in claim 11 wherein the
first location is a first data store and the second location is a
second data store.
15. The data management method as described in claim 14 wherein the
real-time history provides a data distribution service.
16. A data management method, comprising: generating a real-time
event data stream by capturing data changes associated with one or
more events associated with a file system data source; receiving
the real-time event data stream at a given location remote from the
file system data source; and storing the real-time event data
stream at the given location as a first n-dimensional logical
representation, wherein the n-dimensional logical representation
comprises a set of version objects, and a set of links associating
at least first and second of the version objects.
17. The method as described in claim 16 further including: deriving
a second n-dimensional logical representation from the first
n-dimensional logical representation in response to a given event
in the real-time event data stream.
18. The data management method as described in claim 16 wherein the
set of one or more links comprise at least one of: a version link,
a parent-child link, and a data link.
19. The data management method as described in claim 16 wherein the
file system data source is one of: a directory, and a file.
20. The data management method as described in claim 16 wherein the
given event is one of: a file create, a file modify, a file delete,
a file rename, and a file relocate.
21. The data management method as described in claim 16 wherein the
given event is one of: a directory create, a directory modify, a
directory delete, a directory rename, and a directory relocate.
22. A data structure supported in a data store, comprising: a set
of data objects each of which conform to an object hierarchy, at
least first and second of the data objects being associated with
one another and representing a first version of a file system
component and a subsequent version of the file system component,
the file system component being one of: a directory, and a file;
and a set of links associating the set of data objects, wherein the
set of links comprise at least one version link associating the
first version of the file system component and the subsequent
version of the file system component; wherein the set of data
objects and the set of links provide an n-dimensional logical
representation of the file system component at a given
point-in-time.
23. The data management method as described in claim 1 wherein the
new event is one of: a transaction, a checkpoint, a software
upgrade, a virus detection, a user or business tag, and a network
event.
Description
[0001] This application is a continuation-in-part of U.S. Ser. No.
11/123,994, filed May 6, 2005, which application was based on and
claimed priority to U.S. Ser. No. 60/569,164, filed May 7,
2004.
RELATED APPLICATIONS
[0002] This application is related to the following commonly-owned
applications:
[0003] Ser. No. 10/841,398, filed May 7, 2004, titled "Method and
system for automated, no downtime, real-time, continuous data
protection,"
[0004] Ser. No. 10/842,286, filed May 10, 2004, titled "Method and
system for real-time event journaling to provide enterprise data
services,"
[0005] Ser. No. 10/863,117, filed Jun. 8, 2004, titled "Method and
system for no downtime, real-time, continuous data protection,"
[0006] Ser. No. 10/862,971, filed Jun. 8, 2004, titled "Method and
system for no downtime, resynchronization for real-time, continuous
data protection,"
[0007] Ser. No. 11/185,313, filed Jul. 20, 2005, titled "Method and
system for virtual on-demand recovery for real-time, continuous
data protection," and
[0008] Ser. No. 10/943,541, filed Sep. 17, 2004, titled "Method and
system for data protection."
BACKGROUND OF THE INVENTION
[0009] 1. Technical Field
[0010] The present invention relates generally to enterprise data
management.
[0011] 2. Background of the Related Art
[0012] A critical information technology (IT) problem is how to
cost-effectively deliver network wide data protection and rapid
data recovery. In 2002, for example, companies spent an estimated
$50B worldwide managing data backup/restore and an estimated $30 B
in system downtime costs. The "code red" virus alone cost an
estimated $2.8 B in downtime, data loss, and recovery. The reason
for these staggering costs is simple--traditional schedule based
tape and in-storage data protection and recovery approaches can no
longer keep pace with rapid data growth, geographically distributed
operations, and the real time requirements of 24.times.7.times.265
enterprise data centers.
[0013] Although many enterprises have embarked on availability and
recovery improvement programs, many of these programs have been
focused on the redundancy of the infrastructure, not on the data
itself. Yet, without data availability, applications cannot be
available.
[0014] Today's legacy data protection and recovery solutions are
highly fragmented across a wide variety of applications, systems,
and storage models. The overhead and data management maze that
existing approaches bring to the network, storage, tape, and
application infrastructure has caused increasing expenditures with
little tangible returns for the enterprise. Worse, manual recovery
techniques compound the problem with the same issues that cause
downtime in the first place--human errors and process issues
constitute 80% of unplanned downtime.
[0015] As a result, businesses are enduring high costs, high risk,
and a constant drag on productivity. A recent survey by Aberdeen
highlights IT managers' top data storage problems: managing backup
and restore (78%), deploying disaster recovery (80%), and
delivering required service levels (60%).
[0016] One recently-introduced technique for addressing the complex
problem of providing heterogeneous, enterprise-wide data management
is illustrated in FIG. 1. FIG. 1 illustrates a representative
enterprise 100 in which a data management system (DMS) is
implemented to provide enterprise data protection. A commercial
version of this architecture is available from Asempra
Technologies, Inc., of Sunnyvale, Calif. In this illustrative
example, an enterprise 100 comprises a primary data tier 102 and a
secondary data tier 104 distributed over IP-based wide area
networks 106 and 108. Wide area network 106 interconnects two
primary data centers 110 and 112, and wide area network 108
interconnects a regional or satellite office 114 to the rest of the
enterprise. The primary data tier 102 comprises application servers
116 running various applications such as databases, email servers,
file servers, and the like, together with associated primary
storage 118 (e.g., direct attached storage (DAS), network attached
storage (NAS), storage area network (SAN)). The secondary data tier
104 typically comprises one or more data management server nodes,
and secondary storage 120, which may be DAS, NAS, and SAN. The
secondary storage may be serial ATA interconnection through SCSI,
Fibre Channel (FC or the like), or iSCSI. The data management
server nodes create a logical layer that offers object
virtualization and protected data storage. The secondary data tier
is interconnected to the primary data tier, preferably through one
or more host drivers to provide real-time data services. Data
management policies 126 are implemented across the secondary
storage in a well-known manner. A similar architecture is provided
in data center 112. In this example, the regional office 114 does
not have its own secondary storage, but relies instead on the
facilities in the primary data centers.
[0017] As described in co-pending application Ser. No. 10/841,398,
the DMS system associates a "host driver" 128 with one or more of
the application(s) running in the application servers 116 to
transparently and efficiently capture the real-time, continuous
history of all (or substantially all) transactions and changes to
data associated with such application(s) across the enterprise
network. This facilitates real-time, so-called "application aware"
protection, with substantially no data loss, to provide continuous
data protection and other data services including, without
limitation, data distribution, data replication, data copy, data
access, and the like. In operation, a given host driver 128
intercepts data events between an application and its primary data
storage, and it may also receive data and application events
directly from the application and database. The host driver 128 may
be embedded in the host application server 116 where the
application resides; alternatively, the host driver is embedded in
the network on the application data path. By intercepting data
through the application, fine grain (but opaque) data is captured
to facilitate the data service(s). To this end, and as also
illustrated in FIG. 1, each of the primary data centers includes a
set of one or more data management servers 130a-n that cooperate
with the host drivers 128 to facilitate the data services. The DMS
servers provide a distributed object storage that can be built
above raw storage devices, a traditional file system, a special
purpose file system, a clustered file system, a database, or the
like. In this illustrative example, the data center 110 supports a
first core region 130, and the data center 112 supports a second
core region 132.
[0018] As described in co-pending application Ser. No. 11/123,994,
each DMS node executes an object runtime environment. This object
runtime environment includes an object manager that manages the
lifecycle of all the DMS objects during runtime. The object manager
creates DMS objects, and the object manager saves them in the
shared storage. The objects continually undergo modification as the
system protects data in the enterprise's primary storage. In an
illustrative embodiment, the system automatically creates a trail
of objects called versions; typically, the versions do not actually
exist on primary storage, outside of the data management system.
The DMS manages the creation, storage, display, recovery to primary
storage, deletion (automatic via policy, or manual) and the like,
of these versions. The host drivers protect data into the
continuous object data store. Using this architecture, data in
primary storage can be recovered to any point-in-time.
BRIEF SUMMARY OF THE INVENTION
[0019] A data management method is provided for storing a real-time
history of a file system, or a component thereof, such as a
directory or a file. The real-time history is stored as an
object-oriented logical representation comprising at least a set of
version metadata objects, and a set of one or more links that
associate given objects of the set of version metadata objects. As
one or more events occur in the real-time history, the logical
representation is restructured dynamically. The logical
representation is useful to provide any point-in-time
reconstruction of the file system component on an as-needed
basis.
[0020] The object-oriented logical representation is advantageous
as it provides an efficient way to index a real-time history of
changing data in a file system. This representation preferably is
constructed over a database, such as a relational database, a raw
file system, or any other set of one or more storage devices.
[0021] The foregoing has outlined some of the more pertinent
features of the invention. These features should be construed to be
merely illustrative. Many other beneficial results can be attained
by applying the disclosed invention in a different manner or by
modifying the invention as will be described.
BRIEF DESCRIPTION OF THE DRAWINGS
[0022] For a more complete understanding of the present invention
and the advantages thereof, reference is now made to the following
descriptions taken in conjunction with the accompanying drawings,
in which:
[0023] FIG. 1 is an illustrative enterprise network in which the
present invention may be deployed;
[0024] FIG. 2 is an illustration of a general data management
system (DMS) of the present invention;
[0025] FIG. 3 is an illustration of a representative DMS network
according to one embodiment of the present invention;
[0026] FIG. 4 illustrates an object data structure for tracking
data history according to a preferred embodiment;
[0027] FIG. 5 illustrates a representative DMS object instance
hierarchy according to a preferred embodiment;
[0028] FIG. 6 illustrates a sample instance of a directory
object;
[0029] FIG. 7 illustrates a sample instance of a file object;
[0030] FIG. 8 illustrates a sample file system history according to
the present invention;
[0031] FIG. 9 illustrates another sample file system history;
[0032] FIG. 10 is a flow diagram illustrating a process for
creating a file or a directory in the DMS file system history
structure;
[0033] FIG. 11 illustrates the sample file history of FIG. 9 after
a new file object has been created in the DMS;
[0034] FIG. 12 is a flow diagram illustrating a process for
modifying a file or a directory in the DMS file system history
structure;
[0035] FIG. 13 illustrates how modifying a directory object alters
the DMS object store of FIG. 11;
[0036] FIG. 14 illustrates how modifying a file object alters the
DMS object store of FIG. 11;
[0037] FIG. 15 is a flow diagram illustrating a process for
deleting a file or a directory in the DMS file system history
structure;
[0038] FIG. 16 illustrates how deleting a file object alters the
DMS object store of FIG. 14;
[0039] FIG. 17 is a flow diagram illustrating a process for
renaming or relocating a file or a directory in the DMS file system
history structure;
[0040] FIG. 18 illustrates representative DMS data structure
changes (with respect to FIG. 16) that occur when a file is renamed
and the parent does not track a child name;
[0041] FIG. 19 illustrates representative DMS data structure
changes (with respect to FIG. 16) when the file is renamed and the
parent does track child name;
[0042] FIG. 20 illustrates representative DMS data structure
changes (with respect to FIG. 19) when a file is relocated and
versioned by object instance;
[0043] FIG. 21 illustrates representative DMS data structure
changes (once again with respect to FIG. 19) when the file is
relocated and versioned by object path;
[0044] FIG. 22 illustrates several variants of the relocation
technique of FIG. 21;
[0045] FIG. 23 illustrates a sample baseline file system object
store prior to a directory relocation;
[0046] FIG. 24 illustrates a directory relocation based on a
versioned by object instance model using FIG. 23 as the initial
baseline;
[0047] FIG. 25; illustrates a directory relocation based on
versioned by object path model using FIG. 23 as the initial
baseline; and
[0048] FIG. 26 illustrates a directory relocation based on a
combination of versioned by object path and versioned by object
instance, once again using FIG. 23 as the initial baseline.
DETAILED DESCRIPTION OF AN EMBODIMENT
[0049] FIG. 2 illustrates a preferred hierarchical structure of a
data management system 200 in which the present invention may be
implemented. As illustrated, the data management system 200
comprises one or more regions 202a-n, with each region 202
comprising one or more clusters 204a-n. A given cluster 204
includes one or more nodes 206a-n and a shared storage 208 shared
by the nodes 206 within the cluster 204. A given node 206 is a data
management server as described above with respect to FIG. 1. Within
a DMS cluster 204, preferably all the nodes 206 perform parallel
access to the data in the shared storage 208. Preferably, the nodes
206 are hot swappable to enable new nodes to be added and existing
nodes to be removed without causing cluster downtime. Preferably, a
cluster is a tightly-coupled, share everything grouping of nodes.
At a higher level, the DMS is a loosely-coupled share nothing
grouping of DMS clusters. Preferably, all DMS clusters have shared
knowledge of the entire network, and all clusters preferably share
partial or summary information about the data that they possess.
Network connections (e.g., sessions) to one DMS node in a DMS
cluster may be re-directed to another DMS node in another cluster
when data is not present in the first DMS cluster but may be
present in the second DMS cluster. Also, new DMS clusters may be
added to the DMS cloud without interfering with the operation of
the existing DMS clusters. When a DMS cluster fails, its data may
be accessed in another cluster transparently, and its data service
responsibility may be passed on to another DMS cluster.
[0050] FIG. 3 illustrates the data management system (DMS) as a
network (in effect, a wide area network "cloud") of peer-to-peer
DMS service nodes. As discussed above with respect to FIG. 2, the
DMS cloud 300 typically comprises one or more DMS regions, with
each region comprising one or more DMS "clusters." In the
illustrative embodiment of FIG. 3, typically there are two
different types of DMS regions, in this example an "edge" region
306 and a "core" region 308. This nomenclature is not to be taken
to limit the invention, of course. An edge region 306 typically is
a smaller office or data center where the amount of data hosted is
limited and/or where a single node DMS cluster is sufficient to
provide necessary data services. Typically, core regions 308 are
medium or large size data centers where one or more multi-node
clusters are required or desired to provide the necessary data
services. The DMS preferably also includes a management gateway 310
for controlling the system. As seen in FIG. 3, conceptually the DMS
can be visualized as a set of data sources 312. A data source is a
representation of a related group of fine grain data. For example,
a data source may be a directory of files and subdirectory, or it
may be a database, or a combination of both. A data source 312
inside a DMS cluster captures a range of history and continuous
changes of, for example, an external data source in a host server.
A data source may reside in one cluster, and it may replicate to
other clusters or regions based on subscription rules. If a data
source exists in the storage of a DMS cluster, preferably it can be
accessed through any one of the DMS nodes in that cluster. If a
data source does not exist in a DMS cluster, then the requesting
session may be redirected to another DMS cluster that has the data;
alternatively, the current DMS cluster may perform an on-demand
caching to bring in the data.
[0051] As described in U.S. Ser. Nos. 10/841,398 and 11/123,994,
the DMS provides real time data services, such as continuous data
protection, data replication, data distribution, any-point-in-time
recovery, and any-point-in-time snapshot. To support these
services, preferably the DMS host driver resides in an application
host or the network, monitoring and capturing application events
and data changes in real time, and then processing and forwarding
actual data changes, events, and metadata to a DMS node. The host
driver preferably performs delta reduction (e.g., to extract byte
level changes), identifies metadata changes such as access control,
detects application checkpoint events, and then forwards this
information as a stream to a DMS node in a DMS cluster. A DMS
cluster is a group of DMS nodes that share a storage module. These
nodes work as a cooperative unit. Preferably, they obey a set of
access rules such as acquiring lock of different classes, and they
honor the access locks of the others so as to perform parallel
access to the storage module. These nodes also watch for the health
of one another and when one node fails, the other nodes preferably
repair any partially modified or corrupted data that may be caused
by the failure, and take over the tasks of the failed node.
[0052] The DMS nodes are the entities that provides real-time data
services. When providing continuous data protection and data
distribution as subscriber, the nodes take incoming data streams,
preferably translate the streams into an object-oriented data
structure (or another structure that provides similar data
associations and indices), and save the data in a storage module
that is referred to herein as an object store. The object store is
designed with the purpose of managing real-time continuous history.
When providing data replication, data recovery, and generating a
snapshot, the DMS node navigates its object store, reconstructs a
desired point-in-time data object, and forms outbound data streams
that are then delivered to target nodes or host machines. To
provide continuous replication, once replicating a point-in-time
data object, the DMS node also forwards, to a remote DMS or a
remote host server, a continuous redo log of the objects (in the
form of a real-time event journal). A goal of the DMS is to store
fine grain and real-time data history. Thus, the DMS object store
is designed to track fine grain data changes without using
excessive storage. The DMS preferably also indexes by time and
events all fine grain objects, application checkpoints, and
metadata globally across DMS clusters. The events may include any
object events such as email arrival, transaction activity, a file
open, a file modification, a file close, a directory change, an
application checkpoint, a system upgrade, a virus detection (such
as from an external network service), a business event tag, or the
like.
[0053] The DMS nodes create distributed object storage to provide
the necessary real-time data management services. The objects
created by the DMS nodes are sometimes referred to herein as active
objects. The active objects at any moment in time may be dormant in
the storage or instantiated by the DMS nodes to handle requests and
to perform activities. The details of active objects are discussed
in the following sections.
[0054] The distributed object store can be built above raw storage
devices, a traditional file system, a special purpose file system,
a clustered file system, a database, and so on. Preferably, DMS
chooses to build the distributed object store over a special
purpose file system for storage and access efficiency. The files in
the special purpose file system and the active objects in the DMS
preferably are all addressed by a (e.g., 128 bit) global unique
identifier (GUID). During runtime, a GUID can be de-referenced to a
physical address in a storage device. By doing so, this allows the
object store to scale beyond a single storage device, such that an
object (1) in a device (A) can refer to another object (2) in
device (B), e.g., by referring to the GUID of object (2).
[0055] Preferably, each DMS node executes an object runtime
environment. This object runtime environment includes an object
manager that manages the lifecycle of all the DMS objects during
runtime. The object manager creates DMS objects, namely the active
objects, and the object manager saves them in the shared storage.
When requested, the object manager loads an existing active object
from the storage, and then routes object requests directly to the
instantiated active object. Once an active object is created or
loaded (instantiated) into the memory, it is responsible for
executing requests routed from the object manager. The object
manager may perform authentication and/or authorization before
allowing any access to an active object. An active object, upon
request, may update its internal information, execute an object
specific program, and terminate itself from the runtime
environment. Both the object manager and the active objects are
responsible for acquiring shared lock as necessary so that all the
nodes can have parallel access to the same objects. The object
manager is also responsible for permanently removing active objects
from the shared storage when requested. In a non object-oriented
embodiment, an object manager may not be required. Also, while the
use of an object manner is preferred, it is also possible to
implement the present invention by storing the object information
in a different manner while still achieving similar results, and
the processes and object functions that change the information may
be modified accordingly.
[0056] Preferably, an instance of an active object has a set of
properties, with each property having a label and value pair. For
example, an active object may have one property labeled as "name"
with an associated value being "The design of a PC," and another
property labeled "content" which associated value is a binary blob.
A property has a value type definition, for example, the value of
the "name" property is a string, and the value of the "content"
property is an opaque binary chunk of data.
[0057] For example, when DMS protects a file from a server, the DMS
active object for the file may have a list of representative
properties such as shown below:
[0058] ObjectClass: <a 128 bit file object class
identifier>
[0059] ObjGUID: <a 128 bit unique identifier for this
object>
[0060] Creator: <a user identifier>
[0061] ExtemalCreationDateTime: <a time stamp>
[0062] DMSCreationDateTime: <a time stamp>
[0063] Name: <a string>
[0064] ParentObject: <a GUID of a directory object>
[0065] ACL: <an object GUID>
[0066] Version: <integer or time stamp>
[0067] Size: <integer>
[0068]
[0069] ExternalModifiedDateTime:<a time stamp>
[0070] DMSModifiedDateTime:<a time stamp>
[0071] DMSTerminationDateTime: <a time stamp>
[0072] ModifiedBy: <a user identifier>
[0073] Company: <a string>
[0074] Department: <a string>
[0075] Title: <a string>
[0076] Subject: <a string>
[0077] Keywords: <a string>
[0078] Comments: <a string>
[0079] Content: <a random binary blob>
[0080] In the context of a traditional file system, preferably all
properties beside the "content" property are classified as metadata
whereas, in the DMS, preferably all properties including the
"content" itself are managed as metadata. Preferably, the DMS
active objects store metadata from the protected server as well as
metadata generated by the DMS itself. In DMS active object point of
view, all the properties are metadata, including the binary content
from the external world, while binary content is just a specific
property type (random access binary blob type).
[0081] A property on an active object preferably also has specific
attributes such as -modifiable, modifiable-internal, read-able,
versionable, single-value vs multi-value, inheritable, index,
mandatory, replicate-able, and the like. Some object properties,
such as ObjectClass, ObjGUID, Creator, ExternalCreationDateTime,
and DMSCreationDateTime do not change once the object is created,
while the other properties can be modified. There are also
properties, such as Version, DMSModifiedDateTime, and
DMSTerminationDateTime, that are not modifiable by any external
entity besides the Object Manager and the object itself.
[0082] The following is a table of possible property types:
TABLE-US-00001 Property Type Description Integer a number String a
Text string GUID Preferably a 128 bit global unique ID across all
DMS nodes. This property store the GUID of another object that
allows objects to be linked. Constant a set of related unique
numbers that represent some specific information Random access
binary blob Binary data for random access Sequential access binary
blob Sequential records Boolean True/false ComplexType Combination
of one or more of the above
[0083] The following is a table of possible attributes for each
property: TABLE-US-00002 Property attributes Default Description
Modifiable True Property once created, can be modified by an
internal or external request Modifiable- True Property once
created, can be only be internal modified by the DMS internally
Read-able True Property can be accessed by external request
Version-able True Property can be versioned. For example,
modification-date-time is a version-able property. Multi-value
False If Multi-value is True, the property has many values. For
example, the children property of a directory object is a
multi-value property. If Multi-value is False, the property is
single value property. Single value is the default of all property.
Inheritable False If True, when object receives a request for the
property, and if the property is not set, the object can request
for the value from its parent in the object hierarchy. For example,
a file object may request for a value from its directory object. By
default, all properties are not inheritable. Index False If True,
the DMS automatically generates a specific index for the property
to accelerate search. Indeed, the entire object structure may be
considered an indexing structure; however, by having a specific
property index, the system allows for a focused and efficient
search on that property. Once indexed, the object can be searched
using the index of the property. The indexed properties are name,
fingerprint, and the like. By default, a property is specifically
indexed. If a property is not indexed, preferably it is still
searchable by an algorithm iterating through all the objects.
Replicate-able True If Replicate-able is True, then when the
associated property is replicated when the object is
replicated.
[0084] To track real-time changes, some object properties are
defined as version-able. In the DMS, an object data structure for
tracking data history is as shown in FIG. 4. In FIG. 4, pages are
simply logical and variable size chunk of data entities. Each page
is labeled with a GUID. An anchor page 401 contains the
<property, value> of those metadata that are not version-able
and do not change over time, while the metadata page (402, 403 and
404) of each version contains only the versioned properties. The
pages refer to one another by GUID (which is an address or a link
to another group of information). In addition, an object may have
an access control list (ACL) that specifies who has what level of
access right to the data. In the case of DMS, the ACL is stored in
a separate page 405 or 406, such that multiple objects that have
the same ACL can refer to the same ACL page. Multiple ACL pages can
also be grouped into a sorted structure and stored in a physical
storage unit. Access control lists can also be stored within the
version metadata pages or as separate active objects. The object
anchor and version pages can also be grouped and re-organized for
storage efficiency to fit in one or multiple physical storage
units. As noted above, in an alternative embodiment, these
different pages can be structured in a non object-oriented way,
such as by using relational database tables. As long as the
information can be linked and indexed, the present invention can be
realized by such different implementations.
[0085] In DMS, preferably all the anchor and version metadata pages
are combined together into a variable sized file. If desired, each
one of the pages can be stored in a separate file, or in raw
storage blocks. When stored in files, each file is also named by
GUID. There are page GUID to file GUID mappings, and file GUID to
physical address mappings so that the physical data of an object
can be retrieved. An object can be reference by the GUID of its
anchor page, or the GUID of its version metadata page. When
referred by the GUID of its version metadata page, a point-in-time
object is presented.
[0086] According to another aspect of the inventive DMS, an active
object has a basic set of behaviors and some specific set of
behaviors that are schema dependent and may be created specifically
for the class definition. The following are examples of interfaces
that initiate specific object activities. In particular, a basic
set of behaviors that may be initiated by the interface for life
cycle management, getting and setting attributes: [0087]
CreateObject (of a specific class) [0088] DestroyObject (an object
GUID) [0089] ObjectOpen (an object GUID, a point-in-time, and mode)
[0090] ObjectClose (an opened object handle) [0091] ObjectTerminate
(an opened object handle) [0092] ObjectLock (an opened object
handle, and mode) [0093] ObjectGet (an opened object handle, a list
of properties) [0094] ObjectSet (an opened object handle, a
property, a value) [0095] ObjectMVGetFirst (an opened object
handle, a property) [0096] ObjectMVGetNext (an opened object
handle, a property) [0097] ObjectMVGet (an opened object handle, a
property, key) [0098] ObjectMVAdd (an opened object handle, a
property, value) [0099] ObjectMVDelete (an opened object handle, a
property, key) [0100] ObjectRead (an opened object handle, a
property, an offset, a length) [0101] ObjectWrite (an opened object
handle, a property, an offset, a length, data) [0102] ObjectApply
(an opened object handle, a property, a delta string) [0103]
ObjectRecordAppend (an opened object handle, a property, record,
length) [0104] ObjectRecordGetFirst (an opened object handle, a
property) [0105] ObjectRecordGetNext (an opened object handle, a
property) [0106] ObjectRecordGetAt (an opened object handle, a
property, a position) [0107] ObjectExecute (an open object handle,
a function, parameters) These functions may be implemented readily
in software code, i.e., as a set of program instructions executable
in a processor. CreateObject( ) creates a physical active object in
the DMS object store, while DestroyObject( ) removes the physical
object completely. Once created, an active object can be
instantiated by ObjectOpen( ) and it can be manipulated.
ObjectClose( ) ends the execution cycle of an object.
ObjectTerminate( ) terminates an object version and prevents a new
version from ever be created. ObjectGet( ) and ObjectSet( ) are for
accessing a single value property; the generic behavior for setting
a property is to first validate the property type before allowing
the update to occur. ObjectMVGetFirst( ), ObjectMVGetNext( ),
ObjectMVGet( ), ObjectMVAdd( ), and ObjectMVDelete( ) are for
accessing a multi-value property. A multi-value property has unique
key, for example, CHILDREN may be a multi-value property, and its
unique key may be the name or the GUID of the child. ObjectRead( ),
ObjectWrite( ), and ObjectApply( ) are for accessing metadata of a
random access binary blob type. ObjectRecordAppend( ),
ObjectRecordGetFirst( ), ObjectRecordGetNext( ), and
ObjectRecordGetAt( ) are for accessing metadata of sequential
access binary blob type.
[0108] The above object interfaces are a representative subset of
the actual basic object behaviors of the DMS. There are merely
illustrative of the functional behavior of the active objects. If
desired, an object class may define its own set of specific
behaviors.
[0109] In DMS, and as will be described in more detail below,
preferably there are many data source active object classes, for
example, a directory object, a file object, database object, and
the like. FIGS. 6-7 illustrate sample instances of a respective
directory object and a file object. In particular, in FIG. 6 the
directory object (FOO) has three versions. In the first version,
the directory object only has the version 1 of a subdirectory
object--DOO. The subdirectory object changed its name to WOO, thus
a version 2 of the subdirectory object is created; as a result, a
version 2 of FOO is created to link to the version 2 of the
subdirectory. On version 3 of FOO, a new file under the directory
FOO is created. The links, shown as dotted arrows, on the directory
object FOO are stored as a "CHILDREN" property, and this property
is of multi-value GUID type. These links allow the active object to
build up object relationships or an object hierarchy; in this case,
which is merely representative, it is parent-child relationship.
This is a logical view of the directory data structure. For
conservation of storage usage, the directory version pages may be
combined into a table or a journal, and the table or journal may be
stored in a special purpose file or a raw device block. For
simplicity, the above diagram intentionally does not show the
entire subdirectory and file objects (for example, the anchor pages
are not shown).
[0110] FIG. 7 is an example of a DMS file object, which is an
active object that tracks history, as opposed to a file in a
traditional file system. The DMS uses this object structure for
storing the history of a file from an external host server. As
previously mentioned, in one embodiment of the invention, the DMS
overlays the object structure of its object store over a special
purpose file system for storage usage efficiency. Thus, the object
store is a logical structure and the file system is the physical
structure. In the DMS file object, preferably there is a property
called "CONTENT," and this property is of the type random access
binary blob. The binary value of this property type may be stored
inside or outside of the metadata page. In this case, the binary
data of version 1 is in a binary page 716 that has its own GUID.
The changes (deltas) that are made to the file for version 2 may be
stored as a sequence of forward deltas in a delta page 717. The
changes (deltas) of version 3 may also be appended to the same
delta page or another new delta page. Both the binary and delta
pages may be stored in one special purpose file, be broken up and
stored in multiple special purpose files, be stored in a database,
or be stored in raw storage devices. The purpose of storing a
baseline and a set of deltas is for storage efficiency, and this
binary data format can be used with other embodiments (such as
where the logical representation is not in object-oriented form).
Also, for access efficiency, there may be a new baseline after some
number of deltas. This means that, for objects with a long history,
there may be multiple baselines and a set of deltas for each
baseline.
[0111] Active object binary data management is designed for
managing history of random access binary blob property type. As
shown in FIG. 8, the property type of random access binary blob may
be stored inside a metadata page, or it may be stored outside a
metadata page in both binary and delta pages. Regardless of how the
random access binary data are stored, the DMS manages this data the
same way, preferably through a sparse index. As mentioned earlier,
for binary data management, an initial full binary content is first
captured into a binary page, and then the random changes to the
binary contents are stored as a sequence of forward deltas (or
delta strings) in delta pages. Delta strings preferably are of
specific syntax. A delta string can represent an insertion, a
deletion, or a replacement to an existing binary blob. To avoid
having to apply deltas in real-time when a file version is
accessed, preferably a byte level index is maintained as part of
the random access binary blob property. The sparse index for
version 1 of a file may specify that the entire binary content of
the file is in a specific binary page. The sparse index for version
2 of the same file may specify that certain byte ranges of the
binary content of version 2 are in some specific locations of the
binary page, while other byte ranges are in some specific locations
of the delta pages.
[0112] For the active objects to manage history of sequential
access binary blob such as database journal activities, a binary
page of sequentially appended records structure can be used in the
DMS. Records management is designed for managing property type of
sequential access binary blob. There are three different types of
record management namely--formatted records, unformatted records,
and file object associated records. Formatted records are a
sequence of well defined records, each record is of specific
structure of fields, and each field has well defined data type and
length. A record schema (record structure definition) is defined
for formatted record property type. This type of record can be used
to track SQL requests of a database history, or email activities of
an email server history. A formatted record can also be used to
track real-time events associated with any data object. Unformatted
records are a sequence of binary record chunks, in this case, the
record chunks may be appended continuously to a binary data page
with or without a header that specifies the length of the record.
Alternatively, records can be appended to a binary data page
without a header, in which case, the boundary of each record chunk
is tracked separately. The boundary of unformatted records can be
tracked using formatted record management structure. This type of
record can be used to track sequential binary journal of a database
or sequential journal file of any application. The characteristic
of this type of binary journal is that records are always appended
to the end and never override previous records. File object
associated records are sequences of meta-information containing
binary data updates to an associated file object. A file object
associated record is used to track the location and length of each
modification to a file object. Besides tracking the file
modifications, a file object associated record can also be used to
track checkpoints that occur with respect to the file.
[0113] Once an object schema is created, an active object instance
can be created from the schema. The active object instance has the
defined metadata and behavior. As in any object-oriented system, an
object schema may be defined based on another object schema so that
metadata and behaviors can be inherited, or so that coded functions
can be reused. In DMS, a generic object is clsObject, which defines
basic metadata such as name, creation date and time, creator,
modification data and time, and so on. It also defines the basic
object behavior. Preferably, other object schemas are defined based
on clsObject (i.e., they inherit from clsObject). The object
inheritance feature is an advantage of the object-oriented
embodiment, however, it is not a limitation of the invention.
[0114] Thus, to provide real-time data management services, DMS
preferably defines a set of data management specific object schemas
as shown in FIG. 5. These object schemas are used to create the
"active" objects that have specific metadata and behaviors as
defined in the schema. The DMS object definition set forth below is
a preferred (but non-limiting) way of organizing the control, data,
and functional structure for the DMS to provide real-time data
management services.
[0115] The schema clsDMSSystem is a class for creating a DMS cloud
active object 520 that represents the logical network of the entire
DMS system (with multiple DMS clusters over multiple regions).
Preferably, there is only one instance of clsDMSSystem in a DMS
network, as it is the root object instance of the entire DMS
network. Preferably, this object is used for tracking DMS regions
512 (each as an instance of a clsRegion schema as described below)
and DMS functional groups that own data across regions 516 (each as
an instance of a clsGroup schema as described below). The instance
typically has a randomly assigned unique identifier. The instance
preferably is created automatically by the DMS network when a first
cluster is configured, i.e. it is created by a first node. This
object instance is populated to all the storage clusters in the
entire DMS network. Preferably, there is only one master copy of
this object, which is the original copy, the one that was created
first. When the properties of the instance change, the properties
are populated to all replicas.
[0116] The schema clsRegion is a class for creating DMS region
active objects 512 that represents and tracks a DMS cluster
network, data network, and server network. Preferably, there is one
instance of clsRegion in each physical location. An active object
instance of clsRegion is used for tracking all the DMS clusters 530
(each as an instance of a clsCluster schema as described below),
repositories 514 (each as an instance of a clsRepository schema as
described below), and host servers 528 (each as an instance of a
clsHost schema as described below) in the region. Because each
region may have multiple storage clusters, the local instance of
the clsRegion object is replicated to all the local storage
clusters. The GUID of each instance of clsRegion are randomly
assigned when created. Preferably, policies are encoded as
properties in clsDMSSystem, clsRegion, clsRepository, clsGroup, and
clsXXDataSource.
[0117] The schema clsRepository is a class for creating a DMS data
container 514 for storing protected data sources. A repository
instance may have sub-repository instances 514 and/or protected
data sources 518. A root repository object that is directly under a
region represents a segment of a data network. A repository may be
a child of a region or a child of another repository. The child of
a region is the root of a DMS data object hierarchy. The repository
object provides regional data grouping and policy enforcement. The
policies in a repository are executed against all the data sources
within the scope of the repository. Alternatively, a separate
policy object may be defined and used for storing policies
explicitly in the data hierarchy. If policy object instances are
used, they can be attached to any one of the data container object
instances.
[0118] The schema clsXXDataSource is a class for creating data
source instances 518. Preferably there are three basic data source
schemas, clsFSDataSource, clsDatabaseDataSource,
clsCompoundDataSource. If desired, there may be more schemas for
application data other than the file system and one or more
databases. An active object instance of a clsXXDataSource is a root
container for a protected data source where a data source from a
host is streamed. An instance of clsFSDataSource contains a file, a
directory, or a volume of a file system and its history, while an
instance of a clsDatabaseDataSource contains one or more databases
and their history from a database server. An instance of a
clsCompoundDataSource is a container for multiple data source
instances. Unlike a repository that only provides logical
containership, a compound data source instance preferably provides
activity sequencing and indexing, as well as consistency marking to
the real-time activities of its related group of data sources so
that group consistency can be maintained.
[0119] The class clsFile is a schema for creating object instances
for the DMS to store the information of a file 520 from a host
server and also to track the history of that file in the host. An
instance of a clsFile is similar to a file in a file system, except
that an instance captures, indexes and manages file history. In
DMS, this object is used for data protection, with each instance of
clsFile used to represent an external file in an external host.
[0120] The class clsDirectory is a schema for creating object
instances for the DMS to store the information of a directory 522
from a host server and also to track the history of that directory
in the host. An instance of a directory simply represents a
container of files and other sub-directories.
[0121] The class clsDatabase is a schema for creating object
instances for the DMS to store the information of a database 524
within a database server, and also for tracking and indexing the
history and checkpoints of that database in the server. This object
is used to provide database protection services. An instance of a
clsDatabase represents a continuous consistent image of a database,
across a range of time, from an external host server.
[0122] The class clsJournalGroup is a schema for creating object
instances 526 for the DMS to journal the redo and undo log journal)
activities of a database. The database journal activities may be a
sequence of updates to a group of related journal log files, or
application level transaction records.
[0123] The class clsRecordFile is a schema for creating object
instances 527 for the DMS to track sequential journal entries
within a journal group.
[0124] An active object instance of the clsHost is created whenever
a host driver from a new host server first connects to the DMS
network. This object allows the DMS to track the data services
provided to the information on the host 528. This object also
associates the protected data sources in the DMS to the data source
on its host server. An instance of clsHost preferably contains
information such as the platform of the host, the operating system,
the host configuration, data sources that are protected from the
host, DMS data sources that are replicated to the host, and the
like. The protected or replicated data source properties preferably
include the host path, the size of the sources in the host, the
activities and statistical information about those data sources,
and the GUID of the clsXXDataSource instance.
[0125] An active object instance of the clsDMSCluster schema
represents a DMS cluster 530 with one or more DMS nodes and the DMS
storage. This instance provides statistics and status information
of its specific cluster. Typically, there is only instance per
storage cluster, thus the processes (e.g., the object runtime
environment) of all the nodes use this instance as shared memory to
keep information such as node availability, master election, and
the like. Information about a DMS cluster (as instances of a
clsDMSCluster), a DMS node (as instances of clsDMSNode), and DMS
storage (as instances of clsDMSStorage) may be stored together with
the other active objects or may be in a specific volume used
exclusively by the cluster manager.
[0126] An active object instance of the clsDMSNode schema
represents a DMS node 532 within a DMS cluster. This instance
provides statistics and status information about the DMS node it
represents. Preferably, the object runtime environment of a node is
responsible for locating a cluster and joining that cluster. Once
joined in a cluster, the runtime environment creates the clsDMSNode
instance.
[0127] An active object instance of the clsDMSStorage schema
represents the storage volumes 534 of a DMS cluster. This instance
allows the DMS storage to be configured, and it also provides
statistics and status information of the storage volumes.
[0128] An active object instance of the clsGroup schema is a data
container that also represents a logical group 516 in an
organization. This allows user to map data sources from one or
multiple repositories in one or more regions to a functional group
of an organization. Its purpose is to enable an administrator or
other permitted entity to assign data management policy across
multiple regions.
[0129] As described above, in the FIG. 5 hierarchy preferably
policies are stored as properties in a given data container
although, in an alternate embodiment, a separate policy object
(e.g., clsPolicyProfile) may be used. An active instance of the
clsPolicyProfile schema may contain a set of data management
policies. There may be one or many policy profiles in the DMS
network. A policy profile object can be assigned (as a default data
management policy) to any data container, such as the universe (an
instance of clsDMSSystem), regions, repositories, groups, or
protected data sources, or to any data object, such as files,
directories, and databases. When assigned to a container, all
sub-containers or the data objects within that root container are
governed by the set of policy rules. As noted above, a region (or a
repository) object allow an administrator to set policies for data
in the same region, while a functional group object allows an
administrator to set policies for data in multiple regions.
Typically, a policy is a combination of a set of properties, e.g.,
a rule, an override rule, one or more qualifying events, one or
more qualifying property values, and/or a schedule. A rule may be a
Boolean expression with an action, and a rule may be nested.
[0130] Similar to an instance of a clsPolicyProfile object, an
active object instance of a clsPolicyOverride can be introduced
that may also contain a subset of data management policies. When
assigned to a data container or data object, the policies in the
override object takes precedent over the default policy on an
assigned policy profile objects.
[0131] The DMS object definition discussed above is merely one way
of organizing the control, data and functional structure for the
DMS to provide real-time data management services. One could easily
reorganize the structure to achieve that goal, and the present
invention is not limited to any specific organization. Thus, for
example, clsRegion may be broken down in to multiple hierarchies to
represent local lines of business and departments, clsDMSCluster
may include and nodes and storage so as to eliminate clsDMSNode and
clsDMSStorage definitions, clsJournalGroup may be part of
clsDatabase definition, and so on. Also, as described above, the
effect of this object-oriented hierarchy can be realized in
different non object-oriented structures, such as a relational
database. All of these variants are within the scope of the present
invention.
[0132] FIG. 5 illustrates a relationship among DMS active objects.
This diagram does not show any object history (object versions).
Policy profile and policy override objects are also not shown in
this figure to avoid complexity.
[0133] In FIG. 5, an active object instance is represented by
I<object schema> (note that a schema is the same as a class;
it is the definition of an object). The "I" stands for instance,
and object schema is the definition of that object class. As
illustrated, there is only one instance of the DMS system object
510 (i.e. one DMS network). As illustrated, one or more regions
512, and zero or more functional groups 516 can be created under
the DMS network. As noted above, the region and group active
objects are used for storing configuration information about the
region and the functional group. Functional groups may have
sub-groups (i.e. group within group). Data repositories 514 can be
created under a region 512. Much like groups, repository may have
sub-repositories 514, as has been described. Protected data sources
518 reside within a repository 514. Data may be streamed into a
data source from a protected host server, or streamed into a data
source from another DMS data source through remote distribution
service provided by the DMS. A data source may be configured to
replicate to a remote repository. Within a data source 518, the
real-time history of data files 520, directories 522, databases
524, database journals 526, email databases, email messages, and
the like, are captured and indexed. The present invention focuses
on file system data history, as will be seen. The groups 516,
repositories 514, protected data sources 518, and the data objects
within the data sources are known as the data network 502. Although
not shown in this diagram, policy can be assigned to all the
objects in the data network and all the objects above the hierarchy
of the data network. Preferably, policies are enforced in
hierarchical order and with specific override rules.
[0134] Whenever a DMS host driver is installed into a host server,
the host driver reports to the DMS, and this operation results in
an instance of host object 528 being created in the DMS. As noted
above, preferably a host object 528 contains information such as
the host OS and platform about the host server. Once a host object
is created, IT administrators can identify host data to be
protected, and then configure for the host data to be protected. An
IT administrator can also configure for DMS protected data to be
replicated to a host server. As noted above, the host active object
refers to the data source(s) that are protected from its host
server or data sources that are replicating to its host (as
illustrated by the link between 518 and 528). The host objects in
the DMS form an external host server network 506.
[0135] A region may have one or more DMS clusters, with all DMS
clusters preferably tracked by the DMS via DMS cluster active
objects 530. Each cluster has a representation active object that
refers to the node active objects 532 and storage active objects
534. The cluster object also contains cluster statistic and status
information. A node object 532 contains configuration information,
statistics and status about the node. The storage object contains
storage volume information, and storage group information. Volume
information includes all the raw storage volumes that are
provisioned to the DMS. It also includes the DMS partitioning of
the raw storage volumes, and the assignment of the partitions to
storage groups. In the DMS, a protected data source has its own
storage space that is called a storage group. A storage group is an
aggregated DMS storage partitions carved out from the volume
groups. The cluster, storage, and node objects form a DMS server
network 504.
File System Data History
[0136] The following section describes representative object
schemas defined for protecting a file system according to the
present invention. Preferably, and with reference to FIG. 5, there
are three object classes (schemas) for file system protection,
namely File System Data Source class (clsFSDataSource), Directory
class (clsDirectory) and File class (clsFile).
[0137] In the context of a file system at a host server, typically
all properties except the content are usually known as metadata. As
described above, the DMS active objects store metadata from the
protected host server as well as metadata generated by the DMS
itself. In the DMS, all the properties are metadata including the
binary content from the external world. For a clsFile object,
binary content is a "content" property with random access binary
blob type.
[0138] ClsFSDataSource
[0139] This is a schema of a file system data source; as noted
above, this object preferably serves as a container for the history
of a protected file system or folder in a host. It is also a data
service entity for managing inbound and outbound traffic for the
file system, the policy management entity for its data, and a
security guard for any access to the protected data history.
[0140] The properties of this object class typically include the
configuration of the protected data source. The following table
illustrates representative property examples of this object class:
TABLE-US-00003 Properties of clsFSDataSource Descriptions ID GUID
Name Name of the data source Parent GUID of its parent container (a
repository object) DateTimeCreated Timestamp when this data source
container is created Owner The user ID of the creator ACL Key or
GUID to the access control list of this object DataSourceType File
system RuntimeStates Protecting, replicating, disconnected from
host, and the like Status Active, archived Master GUID of the
original protected data source (if this is a replica) Replicas GUID
of the replicas that need input from this object Host GUID of the
associated host object where the data source resides HostPath The
path name in the host that is protected by this object
ProtectedDateTime Timestamp when protection begun ArchivedDateTime
Timestamp when this data source became idle Children The root
directories or files in this container EventTags List of entries
with event and timestamp. The events are in data source level, may
be set by users.
[0141] As can be seen, the above table is a subset of properties
used in the DMS; one may add more or remove some of the above
properties. For example, policy may be added, and one may not need
RuntimeStates. In one embodiment, the properties of this object are
not versioned so that the history of the object is not tracked.
Alternatively, one can version some of the properties, such as
HostPath, ProtectedDateTime, ArchivedDateTime, and Children, such
that these configuration changes are recorded in time. When
properties are versioned, it is desirable to track the version
begin and end timestamp.
[0142] ClsDirectory
[0143] The ClsDirectory schema is defined for tracking the history
of a directory (folder) in a file system. This schema is used for
protecting a directory, and it is capable of recovering a directory
to any point-in-time in the past.
[0144] The properties of this object class preferably include the
following: TABLE-US-00004 Properties of clsDirectory Descriptions
Non-versioned Properties: ID GUID of this object instance, also
GUID of this anchor page DataSourceParent GUID of its protection
data source container DateTimeCreated Timestamp when this object is
created (from the protected host) Creator The user ID of the
creator DateTimeTerminated Timestamp when this object is deleted
(from the protected host) EventTags List of entries with event and
timestamp. The event tags may be set by users for tracking
purposes. FirstVersionID GUID of the first version LatestVersionID
GUID of the latest version VersionCount Total number of versions
Versioned Properties: ID The version GUID of this object, also GUID
of this page AnchorID The GUID of the anchor page (i.e., the GUID
of this object) PreviousVersionID The version GUID of the previous
version NextVersionID The version GUID of the next version Parent
GUID of parent object which can either be the data source container
or another directory Name Name of the directory DateTimeModified
Timestamp when the version is created (or when the modification
occurs) ModifiedBy ID of a user who modified the directory
DateTimeEnded Timestamp when this version is ended (i.e., a new
modification results in another version being created and old
version ended) ACL Key or GUID to the access control list of this
object ChildrenCount Number of children Children A list of its
children which can be sub-directories or files. Each entry is a
reference to the version ID of a child.
[0145] These properties are merely illustrative of what a directory
active object contains. One may include more properties, such as a
full path name, children name, policies, metadata from the host
server, and more.
[0146] As can be seen, the above table shows the logical
information of a directory object. Whenever a new child is added,
an existing child is deleted, the directory is moved, the directory
is renamed, or the name of its child has changed (if child name is
also captured in this object), a new logical version of the
directory is created. Whenever a new version is created, preferably
a previous version is terminated. DateTimeModified of a new version
must be after or same as the DateTimeEnd of the previous version.
ModifiedBy indicates the user who made the modification to create
the new version. If desired, an Event tag may be added; preferably,
it is a sequence list of entries each having a timestamp.
[0147] This logical layout may map directly into a physical store
(e.g., overlaying a file system or a database), but this is not
required. With respect to persistent storage, preferably DMS stores
multiple logical versions of a directory in one physical store
unit. Or, DMS may also combine multiple versions into one directory
journal. For example, one physical storage unit of a directory
object may contain the initial directory baseline information and
all the changes to a directory within a period of time. A directory
journal entry may be "version ID, DateTimeModified, add new child,
child GUID, . . . " and so on. Once that is done, a logical version
of a directory can be constructed on demand (i.e., upon request) by
applying the necessary activities to a baseline directory image.
The present invention is not limited to any physical layout of the
object in the physical store.
[0148] FIG. 6 shows a sample instance of a directory object. As can
be seen, a representative directory object (FOO) has three
versions. The anchor 610 represents the entire directory object,
and the anchor has a reference (FirstVersionID) to a first version
page 612, and a reference (LastVersionID) to a last version page
614. All three versions (612, 613 and 614) have reverse reference
(Parent) to their object anchor. All three versions are linked
together with their PreviousVersionID and NextVersionID. With these
references, one can traverse a directory object and locate the
directory information at any point-in-time in the past. In this
example, all versions also reference to an ACL page (615 and 616)
so that the access control at a point-in-time can also be located.
In each of the version pages (612, 613, and 614) there are
DateTimeModified and DateTimeEnded properties. As noted above, the
DateTimeModified of version 2 should be before or equal to the
DateTimeEnded of version 1; this property allows one to find out
what version existed at what time frame. The version pages also
preferably contain the ModifiedBy property, which allows one to
find out who did what to the directory object when. User events and
any system events that are meaningful to the directory can be
tagged at the anchor page. When the directory is deleted, the
DateTimeTerminated property in the anchor page is set to the
deletion time. As can be seen, the this data structure and the
associated metadata allows one to track a directory history from
what information has changed at what time, by whom, through what
event, as well as what meaningful events to this object occurred
during its lifecycle.
[0149] In this example, in the first version 612, the directory
object only has the version 1 of a subdirectory object, namely, DOO
617. At some point in time, the subdirectory object changed its
name to WOO; thus, a version 2 of the subdirectory object was
created and is represented by reference number 618. In this
example, the directory active object tracks its children's name;
thus, version 2 of FOO 613 was created to link to the version 2 of
the subdirectory WOO 618. On version 3 of FOO 614, a new file of
the name GOO under the directory FOO was created. This is
illustrated by reference numeral 619. The links, shown as dotted
arrows, are references of the CHILDREN property of the directory
object. As can be seen, these links allow active objects to build
up object relationships or object hierarchy (e.g., a parent-child
relationship). The result is a logical view of the directory data
structure. To conserve data storage usage, the directory version
pages may be combined into a table or a journal, and the table or
journal may be stored in a physical storage unit. For simplicity,
it should be appreciated that the above diagram does not show the
complete subdirectory and file objects (for example, the anchor
pages of the file and subdirectory objects are not shown).
[0150] ClsFile
[0151] The ClsFile schema is defined for tracking the history of a
file from a file system. This schema is used for protecting a file,
and it allows for the recovery of the protected file to any
point-in-time in the past.
[0152] The properties of this object class preferably include the
following: TABLE-US-00005 Properties of clsFile Descriptions
Non-versioned Properties: ID GUID of this object instance, also the
GUID of this anchor page DataSourceParent GUID of its protection
data source container DateTimeCreated Timestamp when this object is
created (from the protected host) Creator The user ID of the
creator DateTimeTerminated Timestamp when this object is deleted
(from the protected host) AccessLog List of entries with timestamp,
user id, and access mode. EventTags List of entries with event and
timestamp. The event tags may be set by users for tracking
purposes. FirstVersionID GUID of the first version LatestVersionID
GUID of the latest version VersionCount Total number of versions
Versioned Properties: ID The version GUID of this object also GUID
of this page AnchorID The GUID of the anchor page (the ID of this
object) PreviousVersionID The version GUID of the previous version
NextVersionID The version GUID of the next version Parent GUID of
parent object which can either be the data source container or a
directory Name Name of this file at this version DateTimeModified
Timestamp when the version is created (or when the modification
occurs) ModifiedBy ID of a user who modified the file DateTimeEnded
Timestamp when this version is ended (i.e., a new modification
results in another version being created and an old version ended)
Status Consistency, DMS checkpoint, suspect ACL Key or GUID to the
access control list of this object Fingerprint A hash key of the
entire content (e.g. MD5, SHA-1, CRC, or the like) Signatures A
sequence of hash keys each generated from a contiguous chunk of the
content Content The sparse index of this version. Sparse index is
byte level reference to the binary content. Binary contents are in
baseline binary pages and delta pages. Additional metadata and
Information from the original document attributes
[0153] The above table is merely illustrative; other properties,
such as a full path name, policies, and the like, may be included.
As can be seen, the non-versioned properties preferably include a
timestamp when the object is created, creator information,
identification of access journal for forensic purposes, and event
tags for tracking user events across time line. The versioned
properties preferably include name, modification information,
status, ACL, and content. Preferably, a new logical version is
created when the name of the document changes, the content of the
document changes, the ACL changes, document metadata or attributes
change, or when the document is moved. When the document is deleted
from the protected data source, the file object at the DMS is
terminated with (DateTimeTerminated timestamped), and the last
version ended.
[0154] FIG. 7 is an example of a DMS file object that stores data
history of a file from an external host server according to the
present invention. As described above, preferably the file object
has an anchor page 710 for its non-versioned metadata. This example
shows a file object with three versions (711, 712, and 713). The
FirstVersionID and LatestVersionID properties from the anchor refer
to the first and third version of the file. The AnchorID property
on each version contains the GUID of its anchor page. The versions
are connected into a double link list with the PreviousVersionID
and NextVersionID properties on each version property set. In this
example, each version has an ACL property that refers to the access
control list. Each version page also has a pair of timestamps,
DateTimeModified and DateTimeEnded, to indicate when the version
becomes existent and when the version is ended and a new version
born. There is also a ModifiedBy property to capture the user who
modifies the file. When a file is deleted, the DateTimeEnded
property on the last version page and the DateTimeTerminate
property in the anchor page are set to the deletion date. The File
version is created when a file is modified and ended when the file
is closed.
[0155] In the DMS file object, preferably each version has an
associated property called "CONTENT," and this property is of the
type random access binary blob. The binary value of this property
type may be stored inside or outside of the metadata page. In this
example, the binary data of version 1 is in the binary page 716,
which has its own GUID. The changes (deltas) that are made to the
file for version 2 are stored as a sequence of forward deltas in
the delta page 717. The changes (deltas) of version 3 are appended
to the same delta page 717 or another new delta page. A file object
may have one or multiple binary pages. The binary pages contain the
baseline data. A file object also may have one or multiple delta
pages for all its changes. The sparse index refers to the data in
both the baseline and the deltas to make up the content for the
version. The binary and delta pages may be stored in one physical
storage unit; alternatively, the pages may be broken up and stored
in multiple physical storage units. As one of ordinary skill will
appreciate, the above-described example is simply one way in which
DMS structures the binary data. Alternatively, each version may
have its own binary pages so that no delta has to be kept. Yet
another alternative is to store reverse deltas or multiple
baselines at different versions with a combination of reverse and
forward deltas.
[0156] This file object structure and the associated metadata
allows DMS to track a file history, e.g., what information has
changed at what time, by whom, through what event, and what
meaningful events to this object occurred during the object's
lifecycle. To conserve data storage usage, preferably the metadata
of the version pages may be combined into a table or a journal, and
preferably the table or journal may be stored in a physical storage
unit. Also, this structure can be stored over a raw storage device,
or by overlaying a file system or a database.
[0157] Although not a limitation of the invention, the file object
may also optimize storage usage through a sparse index, as more
particularly described in Ser. No. 10/943,541, filed Sep. 17, 2004,
which application is incorporated herein by reference.
History of Protected File System
[0158] FIG. 8 shows a sample file system history captured by the
DMS according to the present invention. For simplicity, only the
version metadata pages are shown; the anchor pages and the ACL
pages are omitted. As mentioned above, when a newer version of a
file or directory is created, preferably the older version is
terminated. Therefore, as indicated. each version preferably has a
DateTimeModified and DateTimeEnded. The DateTimeEnded of the latest
version is NULL (i.e., not ended yet).
[0159] In this example, it is assumed that the entire file system
is uploaded to the DMS for protection from a host server at T1.
This is represented by FSDS 801 (DataSource 1). At T1, there is a
version 1 of the directories 811 and 821, and of the files 841 and
851. While this representation refers to a time T1, this time may
be a time range or period, as upload of the entire file system
typically takes a given amount of time.
[0160] At T2, the content of the file by the name "fo1" changed,
which is represented by the new object 842. Because this is a
content change, the parent "do1" v1 (reference numeral 811) is
unaffected.
[0161] At T3, the directory by the name "do2" changed its name to
"dox" (as now indicated by object 822). Because in this example a
directory also tracks the name of its children, this change causes
parent "do1" (which has reference to this object) to also generate
a new version 812.
[0162] At T4, a new file by the name "fo3" is added to "do1" (as
now indicated by object 831); thus, a third version of "do1" is
generated as indicated by reference 813.
[0163] As can be seen, within this DMS file system history,
preferably there are version links, parent-child links (i.e.,
object relationships), as well as the data links to binary and
delta pages (852, 843, 844). Therefore, this structure forms a
three dimensional (3-D) object store. Navigating this object store
enables DMS to identify the history and state of the DataSource 1
at any point-in-time, as will be described in more detail
below.
[0164] As mentioned in the above sections, physical structure of
the object versions can be organized in any fashion to optimize for
the storage usage at the DMS.
[0165] As also mentioned, an alternate non object-oriented
embodiment may be implemented with techniques that link and index
the object versions, object relationships and data changes by
applying a schema over a relational database, together with a set
of processes (described below) that update the database according
to the invention.
File System Events and DMS Operations
[0166] The above sections discuss the DMS logical object data
structure for file system protection. The following sections
discuss the functional behavior of file system protection in more
detail.
[0167] The present invention is directed to the end-to-end process
(e.g., from host driver to active objects in one cluster, or from
active objects in one cluster to active objects in another cluster)
that changes the DMS logical object store structure when a file
system history is captured. In particular, the present invention
describes the processes involved in the storing of real-time
history of a file system, and the use of the real-time history for
any point-in-time recovery for guaranteed consistency. The DMS file
system history structure changes are triggered by one or more
events from file system, users, or applications.
[0168] Thus, as illustrated in FIG. 8, DMS file system data source
object (801) in a DMS cluster receives file system activities and
other events, typically from a host driver that resides in a server
or from another file system data source object that resides in
another DMS cluster. If a data source receives a data stream
directly from a host driver, it is providing a data protection
service, and the data source/storage group in the DMS storage is a
master data source. If a data source object receives a data stream
from another data source object, the service is called data
distribution, and the data source/storage group in the DMS storage
is a replica. As described in Ser. No. 10/842,286, filed May 10,
2004, a file system data protection or data distribution stream
contains real-time event journal of a file system. The file system
event journal is a stream of file system events, metadata, and
associated data. Metadata may include, for example, information
such as who did what when. Typically, the events include OPEN,
CLOSE, CREATE, DELETE, MOVE, RENAME, MODIFY (data, attributes or
metadata such as ACL), DELTA APPLY, FLUSH, and CHECKPOINT. CREATE
event results in new file system active object (version 1) being
created. Host drivers (for data protection) or active objects in a
source cluster (for data distribution) may accumulate changed data
depending on the data type; host drivers send (either
instantaneously or periodically) MODIFY and DELTA APPLY along with
the data and metadata to the target DMS active objects. DELTA APPLY
is used to send delta changes after delta reduction. For some data
types, such as database log activities, the host driver does not
perform delta reduction and does not accumulate changes; rather,
the host driver instead forwards changes with MODIFY events as soon
as the events are received. FLUSH, CLOSE, and CHECKPOINT each
trigger all accumulated changes to be sent to the target DMS active
objects through MODIFY and DELTA APPLY events. CLOSE and CHECKPOINT
events result in an old version being terminated, and a new object
version created. MOVE and RENAME re-locate or modify the name
property of an active object version, and DELETE ends the life of
an active object by setting a timestamp on DateTimeTerminated and
ends the last version.
DMS Processes for File System
[0169] The following sections describe the DMS (host driver and
active object) processes and the associated data history structure
changes in response to specific file system events. Illustrations
of sample DMS file system history are shown for each case. For
illustrative purposes, FIG. 9 is the historical starting point of
the sample file system. As can be seen, the history comprises first
and second versions 911 and 912 of the directory Dir-1, first and
second versions 921 and 922 of the directory Dir-2, first and
second versions 941 and 942 of File-1, and a first version 951 of
File-2. Binary and delta pages (952, 943 and 944) are also
referenced.
[0170] Creation of a File or a Directory
[0171] The creation of a file or a directory in the DMS file system
history structure is initiated by a CREATE event captured by a DMS
host driver operating to provide data protection, or by a similar
event distributed from one data source to another data source. This
function is illustrated in the process flow of FIG. 10. This method
is implemented in software (a set of one or more program
instructions) executable in one or more processors. In a data
distribution scenario, the same event would be forwarded from the
master DMS data source to another replicated DMS data source, which
would cause an equivalent data structure change in the replica. For
simplicity, the following paragraph describes the data protection
scenario.
[0172] This process begins with the DMS host driver capturing a
file or directory CREATE event at step 1071. The host driver then
issues an Object Create command to the associated DMS file system
data source to create an active object with the following
parameters--object class, parent path, object name, creation
timestamp, owner, ACL, and other attributes. This is step 1073. An
object manager at the DMS handles the request for the DMS file
system data source object by opening the parent active object and
creating a new version thereof. This is step 1075, and this
operation has been described and illustrated above. The
DateTimeEnded of the parent's previous version is set to T, and the
DateTimeModified of its new version is set to T as well. The object
manager also creates a new file or directory object with a first
version and sets all the properties as necessary. This is step
1077. The DateTimeCreated property in the anchor of the new object
is set to T, as is the DateTimeModified property in the first
version. Preferably, the state of the first version of this new
object is temporarily set to SUSPECT, as it has not closed yet.
Once the object is created, it is linked to its parent, preferably
by adding its GUID to the CHILDREN property of its parent's latest
version. This also occurs in step 1077. Thereafter, the parent
object is closed and the new child handle remain open; this handle
is passed over to the host driver, which can issue more updates to
the new child. This is step 1079. If any failures occur during the
above-described steps of the process, the transaction is rolled
back at step 1081 and the host driver is notified; the host driver
then performs a resynchronization on that object at step 1084. The
host driver keeps the child handle in an open handle list at step
1083. If the new object is a file object, the host driver forwards
data (via a MODIFY event) to the DMS file object handle and then
closes the file handle to generate a consistent (State=CONSISTENT)
version. At this point, the new object state is SUSPECT not
CONSISTENT.
[0173] In the data distribution scenario, a master DMS data source
object is forwarding events and a data stream to a target DMS
replicated data source object. The process described in the
flowchart of FIG. 10 applies in this data distribution scenario as
well; in that scenario, the host driver is replaced by a master DMS
data source object in a first cluster, and the file system data
source (FSDS) object is replaced by a target DMS replicated data
source object in a second cluster. Otherwise, the operations are
similar.
[0174] FIG. 11 illustrates how a new file by the name "fo3" is
created and added to the DMS file system data source shown in FIG.
9. For simplicity again, the anchor page of the objects are not
shown. An anchor page stores the permanent properties of an object,
which properties do not changed over time. The creation of "fo3"
results in new version of "do1" 1150 being created. The
DateTimeEnded is set to T, and the DateTimeModified of object 1150
is also set to T. The anchor page (not shown in the diagram),
version page 1151 (version 1), as well as binary page 1152 for the
new file object, are also created. The DateTimeCreated property in
the anchor page (not shown) is set to T, as is the DateTimeModified
property for the version page 1151. The State property of page 1151
is set to SUSPECT temporarily (see the discussion below concerning
closing a file for additional details). The GUID of page 1151 is
added to the Children property in object 1150, which links the
parent-child relationship. The sparse index of the Content property
refers to the content in binary page 1152. If the new object added
is a directory object, there is no binary page.
[0175] The following is a representative DMS file system object
(file and directory) creation algorithm. This algorithm is
associated with steps 1075 and 1077 in the process flow diagram
described above. Execution of this algorithm produces FIG. 11 from
the starting point in FIG. 9. TABLE-US-00006 Event = CREATE Create
(object A, in directory B, with metadata X, at time T) Where object
A may be a file or a directory. Metadata includes creator, ACL, and
so on. Open parent directory object B, which has n version. Create
new version (n + 1) for object B, copy all the properties from
version n. Set DateTimeModified (object B, version n + 1) = T,
DateTimeEnded (object B, version n + 1) = NULL Set DateTimeEnded
(object B, version n) = T. Create new object A, create 1 version.
Set properties of A (anchor page and version 1 page) = metadata X.
Set DateTimeCreated (object A, anchor) = T, DateTimeTerminated
(object A, anchor) = NULL Set DateTimeModified (object A, version
1) = T, DateTimeEnded (object A, version 1) = NULL If object A is
file, set State (object A, version 1) = Suspect Add object (A,
version 1) to the property Children (object B, version n + 1) Set
PARENT property of (A, version 1) to refer to (B, version n + 1)
Open parent directory of object B. If B is the root, then the
parent is the file system data source object (FSDS). Call this
object C. Change the Children property of C at the latest version
to refer to the new version of B*. Close object B and C (if C there
is a C) Object A handle is left open for further updates.
[0176] This step is done to ensure that the live version
(non-deleted and most current version) of a parent always refers to
the most current version of the child. By doing so, the time
stamped on DateTimeEnded of a directory version is always equal to
or prior to the time stamped on DateTimeEnded of its children's
version pages. This optimization simplifies navigating in both
child-parent and time dimensions.
[0177] The above algorithm is merely representative of the steps
that may be used to transform the logical representation of FIG. 9
to that shown in FIG. 11 in response to CREATE events. Because the
logical representation can be structured in many different ways,
however, the algorithm for maintaining such different structures
may vary accordingly. For example, as noted above the logical
representation can overlay a relational database, or the logical
representation may overlay a file system or even raw storage blocks
(in effect such that there are only virtual version pages). In
embodiments where the version pages are, in effect, virtual, the
version pages may be constructed as needed, and information
necessary for constructing the version pages may be stored as a
journal or in any other form, such as in a file or raw storage
blocks.
[0178] Modification of a File or a Directory
[0179] A file or a directory modification in the DMS object store
preferably is represented by the creation of a new object version.
The modification of a file system object preferably is broken into
three phases, and each phase involves a set of events. The three
phases of active object modification cycle are (1) instantiating an
active object, (2) modifying the object properties, and (3)
freezing the version. FIG. 12 illustrates a process flow that is
implemented (e.g., in software) to modify a file or a directory in
a master data source operating in data protection scenario. Similar
object store changes occur when the same events are forwarded from
a master data source to a replicated data source in the data
distribution scenario.
[0180] As noted above, phase 1 is instantiation of the target
active object, which can either be a CREATE (as in FIG. 11) or an
OPEN process, as represented by block 1201. Thus, instantiating an
active object can be initiated by a CREATE event or an OPEN (write
mode) event. When a CREATE event is received, a new object with the
first version is created in the object store. The object handle
remains open for further update until the target DMS active object
is explicitly closed, e.g., in response to events to freeze the
object version. FIG. 11 described the process for creating a file
or directory. When a DMS host driver captures a file system OPEN
event to update a file or a directory at step 1202, it issues an
Object Open request to the associated file system data source
active object. This is step 1203. When the DMS active object is
opened, its handle is returned to the host driver at step 1205. The
flow sub-diagram (in block 1201) describes the object open process
in response to OPEN event. The host driver sends to DMS the
timestamp when a file system object is created or opened; the
timestamp is used for stamping on the DateTimeEnded property of the
previous version of the target DMS object, and on the
DateTimeModified property of the new version created.
[0181] Phase 2 is the actual modification of the target object,
which occurs either in block 1225 or in block 1209. The events that
modify object properties preferably include MODIFY (metadata) and
WRITE (binary data). There may be many more modification events
depending on the application and system to which the DMS is
applied. The test at step 1208 determines which event has occurred.
A MODIFY event changes file system object metadata (such as access
control list (ACL), object title, company name, document statistics
and other user defined properties) that is associated with a file
system object. The flow sub-diagram at block 1225 describes the
process to modify object metadata. In particular, in response to
capture of the modification event (steps 1207 and 1208), at step
1226 the host driver issues an object MODIFY event to the target
active object (via the opened object handle) at the DMS, which
target object then changes the object property in the latest object
version at step 1227.
[0182] The WRITE event is for file object only; it represents
binary content updates. In this path, when host driver captures the
WRITE event (steps 1207 1208), it enters the processing block 1209
in the file modification process. Host driver first decides if it
should buffer the changes. This is step 1210. For a database log
file, for example, the host driver would not buffer the log
entries; however, when dealing with a regular document changes are
buffered. If changes are buffered, then the host driver decides if
it is time to forward the changes to the DMS target active file
object. This is step 1211. If no data should be forwarded at the
meantime, the host driver continues to watch for events at step
1207. Otherwise, the host driver decides if delta reduction should
be applied on the changes to extract the actual changed byte range.
This is step 1212. Again, for database log entries, delta reduction
is not necessary because log entries are appended onto the file. If
no delta reduction is necessary, the changed data is packaged with
a WRITE event and forwarded to the DMS target active object. This
is step 1213. Otherwise, the host driver retrieves the last binary
content signatures and, for one or more byte ranges that actually
changed (deltas), the host driver calculates and forwards those
deltas to the DMS target object. This is done using a host
driver-generated (namely, APPLYDELTA) event and occurs at step
1214. The target DMS file object receives both WRITE and APPLYDELTA
events, applies second stage delta reduction as necessary to
extract the exact bytes changed, saves the deltas or binary data in
the binary or delta pages, and then creates or updates the sparse
index in the working (last) object version. This is step 1215. If
the file has not been closed, the host driver continues to capture
more modification events for the file at step 1207; otherwise, the
closing process continues, which is step 1222. If any failure
occurs, the host driver abandons all the events associated with the
target file and performs resynchronization of the entire object at
step 1230. In the preferred embodiment, the host driver uses a
FLUSH event as one of the events to decide if buffered data should
be forwarded to the DMS without causing the DMS target object to
generate new version.
[0183] Phase 3 freezes the changes into an object version. This is
block 1221. The possible events that cause a DMS target file or
directory active object to freeze the latest object version (and
thus to prevent any more changes into that version) are CLOSE,
CHECKPOINT, and FORCE-CLOSE. The flow diagram in block 1221
illustrates the processing of handling of these events to freeze a
DMS active file or directory object version. A CLOSE is a file
system event. When this event is detected by the host driver, the
associated file is consistent. A CHECKPOINT may be generated by an
application with or without user initiative, or it may be generated
by the host driver. A CHECKPOINT event is generated when the
associated application or the host driver have taken some
appropriate action to ensure that the version to be frozen is
consistent in its application point of view. For example, a
database manager periodically flushes its memory and generates an
internal CHECKPOINT, during which a set of database files are
consistent. The CHECKPOINT event from a database manager can be
detected by the host driver, which indicates that the group of
files belonging to the particular database is consistent and a DMS
version of each of the files should be frozen. In yet another
example, an application or the host driver may generate a
CHECKPOINT to all the system state files (or even the entire file
system) to create a snapshot of all the files all at once (an exact
time). This CHECKPOINT forces all the related files to freeze their
latest version all at once. If a CLOSE or CHECKPOINT event is
detected by the host driver at step 1208, the host driver checks if
there is any buffered data for that object. This is step 1220. If
so, the host driver first forces the data to be forwarded to the
target DMS object. After that, the host driver forwards the event
to the DMS target object. This is step 1222. Both CLOSE and
CHECKPOINT events generate a consistent DMS object version with the
State property of the frozen version set to "Consistent." This is
step 1223.
[0184] As noted above, another event that freezes a file or a
directory object is FORCE-CLOSE. This event may be generated by an
application or the host driver itself (step 1207), or it can be
generated by the DMS (step 1225) when an error is encountered. The
error could be that the host driver connection to the DMS has
failed. The FORCE-CLOSE event generated by an application or host
driver goes through the same process as the CLOSE and CHECKPOINT
events (steps 1220, 1209, 1222, and 1223). FORCE-CLOSE freeze the
latest version, as it is in Suspect state (i.e., the consistency is
unknown). This is step 1223. Any DMS object that ends with Suspect
state is eventually resynchronized with its corresponding file
system object in the host, which is step 1230.
[0185] In one embodiment, the DMS data source object automatically
creates a new object version when it handles a CREATE or OPEN
event. In this case, step 1223 involves freeing an unmodified
object version. An alternate embodiment forces the DMS data source
object to create a new object version when it receives the first
modification event.
[0186] The above described process with respect to the given events
is merely illustrative. Other events and/or other handling
techniques may be implemented. Thus, e.g., an alternative
embodiment may choose to freeze file system objects more frequently
(by using events such as FLUSH or possibly the WRITE events with
TIMEOUT in step 1221). Another embodiment includes other
application and host driver generated events, such as AUTO-SAVE.
Also, with alternate embodiments, the process may be somewhat
different than as illustrated in FIG. 12. For example, if a
relational database is used to store the logical representation,
the object-oriented behavior (with object methods/functions) would
not be required.
[0187] In the data distribution scenario as noted above, a master
DMS data source object is forwarding events and a data stream to a
target DMS replicated data source object. The process described in
FIG. 12 applies in this data distribution scenario as well. In
particular, the host driver is replaced by a master DMS data source
object in a first cluster, and the DMS file system data source
object (FSDS) and DMS objects are replaced by a target DMS
replicated objects in a second cluster.
[0188] FIG. 13 illustrates how modifying a directory object alters
the DMS object store of FIG. 11. FIG. 14 illustrates how modifying
a file object alters the DMS object store of FIG. 13.
[0189] In particular, FIG. 13 illustrates the modification of an
ACL on version 3 of the Dir-1 directory object, which was object
1150 in FIG. 11. Once again, for simplicity the anchor page of all
objects is not shown. In this case, a new version (object 1350) of
"do1" is created with its ACL property changed; other properties,
such as the children links, remain unchanged. In other words, both
version 4 (object 1350) and version 3 (object 1150) refer to the
same version of their children. Suppose the ACL of "do1" is
modified at T2, in such case, T2 would be stamped on the
DateTimeEnded property of version 3 (object 1150), and on the
DateTimeModified property of version 4 (object 1350) of the
directory "do1." After the new version 1350 is created, the parent
of "do1, in this case object 901, is modified to refer to the new
version.
[0190] FIG. 14 illustrates the modification of a file object
(labeled 1151 in FIG. 11), on FIG. 13 whose content is modified.
Note that FIG. 13 is derived from FIG. 11 as a result of a change
to "do1." This file object modification generates a new version of
File-3, which is represented as object 1460. In this case, the
deltas of the modification are saved in a delta page (object 1461)
or in the binary page (object 1152) associated with the prior
version. Preferably, the sparse index in version 2 (object 1460)
refers to the byte ranges in both the binary and delta pages for
the version 2 content. Suppose the file "fo3" is modified on T3, in
such case T3 is stamped on the DateTimeEnded property of version 1
(object 1151) and on the DateTimeModified property of version 2
(object 1460). Because the content change to "fo3" does not affect
directory "do1," no new version is created. Version 4 of "do1"
continues after T3, however, thus note that the child link to "fo3"
(object 1150) is changed to refer to version 2 (object 1460) of
"fo3" instead of version 1 (object 1151). This reflects the fact
that, at the current point-in-time, the information of "do1" is in
its V4 and that from there, the current point-in-time "fo3" can be
located.
[0191] The following is a representative DMS file system object
(file and directory) modification algorithm for implementing the
object store changes described and illustrated above.
TABLE-US-00007 Phase 1: Events = CREATE, OPEN. Create (object A, in
directory B, with metadata X, at time T) Open (object A, write
mode, at time T) (step 1205) 1. Open object A, which has m version.
2. Create new version (m + 1) for object A, copy all the properties
from version m. 3. Set DateTimeModified (object A, version m + 1) =
T, DateTimeEnded (object A, version m + 1) = NULL 4. Set
DateTimeEnded (object A, version m) = T. 5. If object A is file,
set State (object A, version 1) = Suspect 6. Open parent of object
A, and change the child link to this latest version (m + 1) of A.
Phase 2: Events = WRITE, MODIFY, APPLYDELTA (may repeat multiple
times) If FLUSH is used in phase 2, it simply triggers execution of
phase 2 as if there is a zero length WRITE, which forces the
accumulated changes to be recorded on DMS. Modify (object A,
property = value) (step 1227) 1. Set property (object A, version m
+ 1) = value Write (object A, content = (offset, length, data))
(step 1215) ApplyDelta (object A, content = delta string) 1. If
first version or if there is no need for delta reduction, write to
binary page 2. otherwise, 2.sup.nd stage delta reduction, write to
delta page 3. Update sparse index of object A, version m + 1 Phase
3: Events = CLOSE, CHECKPOINT, and FORCECLOSE. FLUSH may be
applied. (Step 1223) Close (object A) Checkpoint (object A) 1. if
file, set state (object A, version m + 1) = Consistent 2. close the
object A and its parent object ForceClose (object A) 1. Close
object A and its parent object (in this case state remains
Suspect).
[0192] Deletion of a File or a Directory
[0193] FIG. 15 illustrates a process for deleting a file or a
directory in the DMS file system history structure. This process is
initiated by a DELETE event captured by a DMS host driver operating
in the data protection scenario. In the data distribution scenario,
the same event is forwarded from the master DMS data source to
another replicated DMS data source, which causes equivalent data
structure changes in the replica. The following section describes
the data protection scenario.
[0194] FIG. 15 begins with the DMS host driver capturing a DELETE
event for a file or a directory. This is step 1501. The event is
forwarded to the file system data source active object, together
with the target object name and timestamp T. This is step 1503. The
file system data source object then begins a transaction for the
modification of properties in both the target object and its
parent. In particular, on behalf of the file system data source
object, the object manager opens the parent of the target DMS
object and creates a new object version for the parent. The
DateTimeEnded property of the previous version of the parent object
is set to T, while the DateTimeModified property of the new version
is also set to T. This is step 1505. Thereafter, the target object
is instantiated, and both the DateTimeTerminated anchor property
and DateTimeEnded property at the latest version page are set to T.
This is step 1507. The next step is to remove from the latest
parent version page the child reference to the target object. This
also occurs in step 1507. Once the process is completed, the
transaction is committed at step 1509. Any failure results in
rolling back the transaction (step 1511) and complete
resynchronization of the target object between the host and the DMS
(step 1515). Step 1513 illustrates that the process has
succeeded.
[0195] In a data distribution scenario, as described above a master
DMS data source object forwards events and the data stream to a
target DMS replicated data source object. The process described
FIG. 15 applies in this scenario as well. In particular, the host
driver is replaced by a master DMS data source object in a first
cluster, and the file system data source (FSDS) object is replaced
by a target DMS replicated data source object in a second
cluster.
[0196] FIG. 16 illustrates the deletion of a file object "fo1" 942
from the file system data source in FIG. 14. Once again, the anchor
pages of the objects are not shown. At version 4, "do1" (object
1350) has three children, namely "dox" v2, "fo1" v2, and "fo3" v2.
When the DELETE event (to remove "fo1") is detected at time T4, a
new version of do1's parent is created, and this new version is
object 1650. This new version has its reference to "fo1" removed so
that it is no longer linked to the deleted child. The DateTimeEnded
property of object 1350 is set to time T4 and the DateTimeModified
property of object 1650 is also set to T4.
[0197] In the "fo1" file object, the DateTimeTerminated property in
the anchor page (not shown) is set to T4 to indicate that the
object does not exist beyond T4. The latest version of "fo1," which
is version 2 (object 942) is ended by its DateTimeEnded property
set to T4. As can be seen, preferably the object "fo1" does not get
removed physically from the DMS object store when it is deleted.
Instead, its history terminates at T4. Preferably, the DMS object
store keeps growing its history until a user explicitly prunes the
history, either manually or via retention policy, during which
older versions of the active objects get dropped off at the tail
end.
[0198] Although FIG. 16 shows the deletion of a file, the DMS file
system history changes are similar when a directory object is
deleted. In particular, a version is added to the parent directory
of the target directory with the link between the parent and the
target removed, and the target directory is terminated (in the same
manner as if a target file object is terminated as illustrated). In
deleting a directory, all the directory children should be
terminated at once. In particular, the process traverses down the
children of the directory object, sets the DateTimeTerminated
property (at each anchor page of each child) to the time when the
target directory is deleted, and sets the DateTimeEnded property at
the latest version page (of all the children) to the same
termination timestamp.
[0199] The following is a representative DMS file system object
(file and directory) deletion algorithm for implementing the object
store changes described and illustrated above. TABLE-US-00008 Event
= DELETE Delete (object A, at time T) (steps 1505, 1507) Where
object A may be a file or a directory. Open parent directory object
B, which has n version. Create new version (n + 1) for object B,
copy all the properties from version n. Set DateTimeModified
(object B, version n + 1) = T, DateTimeEnded (object B, version n +
1) = NULL Set DateTimeEnded (object B, version n) = T. Open parent
of object B, change the child reference of the latest version to
refer to version n + 1 of B. Open object A, which has m version Set
DateTimeTerminated of A (anchor page) = T. Set DateTimeEnded
(object A, m) = T If object A is directory, traverse all its
children. For all its children, open them, set DateTimeTerminated
(all its children's anchor page) = T, and set Date TimeEnded (all
its children's latest version page) = T. Remove object A from
property Children (object B, version n + 1) Close all the
objects
[0200] Relocating or renaming a File or a Directory
[0201] In this section, renaming a file system object refers to
changing the name of the object without moving it away from its
directory. While relocating a file system object refers to moving
an object from one directory to another different directory, the
name of the object may also change during relocation. Relocation is
a superset of renaming; one can relocate from within the same
directory for changing an object's name path. In either case, the
destination name either of the rename or relocate operation may
already exist and be used by another object, in which case the
original destination object is deleted and the target object to be
renamed or relocated takes over the destination name.
[0202] In some file systems, a MOVE event is used for both renaming
a file system object (file or directory) as well as relocating
(moving) a file system object from one directory to another. In
other file systems, a RENAME event may be used for both renaming
and relocating of a file system object. There are also file systems
where a MOVE event is used for relocating file system objects while
a RENAME event is used for re-labeling (renaming) file system
objects. In one embodiment of this invention, both MOVE and RENAME
events are treated the same, in other words, both events initiate
renaming and relocating of file system objects.
[0203] FIG. 17 is a flow diagram illustrating a process of moving a
file or a directory according to the invention. Typically, the
process involves a host driver capturing a MOVE or a RENAME event
(at step 1701), and passing the event to the DMS file system data
source object (at step 1703). The DMS file system data source
object then opens the target object (and its parent object(s)) to
handle the event. This is step 1705. DMS handling of a MOVE or
RENAME event occurs at steps 1705 and 1707; the particular
functionality for object relocation varies depending on which
object history model is adopted, and the details are discussed in
following sections. If step 1707 does not fail, the transaction is
committed at step 1709. Step 1713 indicates a success. If, however,
a failure occurs during the renaming or relocation process, object
resynchronization is performed at step 1711.
[0204] As described previously, in the data distribution scenario
the host driver in the process flow is replaced by a master DMS
file system data source, and the DMS file system data source is a
replication data source. Otherwise, the process flow in FIG. 17 is
identical.
[0205] FIG. 18 shows DMS data structure changes when a file is
renamed and a parent object does not track a child object name;
FIG. 19 shows DMS data structure changes when the file is renamed
and parent object does track child object name. In both cases the
resulting data structures illustrated are built from the data
structure in FIG. 16. As before, no anchor pages are shown in the
diagrams to simplify the representation. In a rename operation, the
target file system object does not move from one parent to another;
thus, the process (corresponding to steps 1705 and 1707 in FIG. 17)
is relatively straightforward. In such case, the target object is
opened, a new version is created with the given timestamp T (i.e.,
the old version DateTimeEnded property as well as the new version
DateTimeModified property are set to T), and the NAME property in
the new version is changed to the new name. The new version is
object 1850, which is seen in FIG. 18. If object name is not
tracked (in the CHILDREN property of the parent directory), the
CHILDREN property link (of the last version of the new object's
parent) is changed to refer to the new object version (see, objects
1650 and 1850). If, however, object name is tracked in the parent's
CHILDREN property, the parent object is opened, and once again a
new version of the parent is created. In this case, however, the
new version is seen as object 1951 in FIG. 19. Here, the new object
name is changed on the CHILDREN property of the new parent version.
This process is the same regardless of the whether the target
object is a file or a directory. In the two examples (FIGS. 18 and
19), the destination name "foz" does not exist before the rename.
In the event "foz" already exists and is owned by another object, a
relocate process is used instead of the rename operation.
[0206] The following is a representative rename algorithm for use
in the present invention. As noted above, the file system may use a
MOVE or RENAME event to rename a file or a directory object.
TABLE-US-00009 Event = MOVE or RENAME Rename (object A, old name,
new name, at time T) (Step 1705, 1707) Where object A may be a file
or a directory. Open object A which has m version. Create a new
version (m + 1). Copy all properties of m to m + 1. Set
DateTimeEnded (object A, version m) = T. Set DateTimeModified
(object A, version m + 1) = T, DateTimeEnded (object A, version m +
1) = NULL Open parent directory (object B which has n versions) of
object A. If Object B does not track child name, simply change its
child reference to (m + 1). (FIG. 18) If Object B does track child
name, create new version (n + 1) for object B, copy all the
properties from version n. (FIG. 19) Set DateTimeModified (object
B, version n + 1) = T, DateTimeEnded (object B, version n + 1) =
NULL Set DateTimeEnded (object B, version n) = T. Change child
reference of version (n + 1) to object A (m + 1). Open parent of
object B, change the child reference of the latest version to refer
to version n + 1 of B. Close all the objects
[0207] FIGS. 20 and 21 show DMS object structure changes associated
with relocating a file object. For convenience, both FIGS. 20 and
21 are built upon the data structure that remains in FIG. 19. This
is not a limitation of the invention. No anchor pages are shown in
these diagrams to simplify the representation. According to this
feature, there are two (2) different models for tracking history of
file system object relocation, one (FIG. 20) is versioned by object
instant while the other (FIG. 21) is versioned by path.
[0208] In FIG. 20 (the versioned by object instant model), object
versions are connected across the lifetime of the object regardless
of where the object is relocated. Thus, with reference to FIG. 20,
for example, when "fo2" relocated from "dox" to "do1," new versions
of the affected objects are created. This results in the following
new versions: (2055 "fo2" v2), (2054 "dox" v3) and (2053 "do1" v7).
The DateTimeModified property of these new versions is set to the
time when "fo2" is relocated, and the DateTimeEnded property of the
new versions is set to NULL. The DateTimeEnded property of the
previous versions is set to the relocation time of "fo2." The most
current version of the parent-child link (CHILDREN property) for
both "dox" and "do1" are fixed (with FSDS referring to object 2053,
and object 2053 referring to object 2054). The parent-child link
(object 2054 to object 2055) from "dox" to "fo2" is removed, and
parent-child link (object 2053 to object 2055) from "do1" to "fo2"
is connected via CHILDREN property in object 2053. In this model,
although "fo2" is relocated, its history continues to connect (via
VERSION property) back to when it was a child under "dox" (object
2055 and object 951 are connected, and object 2055 is v2 and object
951 is v1). In this case, where an object already exists that owns
the destination pathname, the object is terminated (deleted).
[0209] In FIG. 21 (the versioned by object path model), object
versions are only connected when the object is in the same
directory container (i.e., the same parent pathname). When the same
object is moved out of the container, a new one is born on the new
container, and the old one from the old container is terminated. In
FIG. 21, for example, when "fo2" is relocated from "dox" to "do1,"
a new version of "dox" (object 2157) and a new version of "do1"
(object 2156) are created. The parent of "do1," which FSDS 901 is
now referring to "do1" v7 (object 2156), and the parent of"dox,"
which is "do1" v7 (2156), is now referring to "dox" v3 (object
2157). In this model, because object versions are connected only if
their path does not change, the path of "fo2" is changed from
"\do1\dox\fo2" to "\do1\fo2"; thus, a new object (not version) with
first version is created. This is illustrated as object 2158, "fo2"
v1. Because this is a new object and has nothing to do with the old
one, there is no connection between object 2158 and object 951 (as
was the case in the FIG. 20 embodiment). The old "fo2" object
(object 951) is "deleted" or "terminated" by having its
DateTimeTerminated anchor property set to the object relocation
time, and its DateTimeEnded property in the latest version also set
to object relocation time. Because objects 951 and 2158 have the
same content, they both share the same binary page 952.
[0210] In the versioned by object path model of FIG. 21, there are
two variants of history management structure. These two variants of
structure depend on if there exists a file object of the same name
in the destination in the timeline. In either case, if an object of
the same name already exists in the destination and is still
active, the destination object is terminated (deleted). In one
variant, indicated by reference numeral 980 in FIG. 22, all objects
stand alone and are not connected to the past "deceased" objects.
In this example, "f1" is moved into "dir1" at T4. As seen in the
drawing, in the past a completely different object of the name "f1"
existed; the former object was created at T1, modified at T2, and
deleted/terminated at T3. In variant 980, the newly arriving has a
new anchor page and a first version page 996 that has nothing to do
with object 995. In this structure, if one were to take a snapshot
at T4, "f1" would be seen to have only one version. In a second
variant, indicated by reference numeral 981 in FIG. 22, objects of
the same name in the same path are always connected in versions and
share the same anchor page as if they are the same object. For
example, in this variant 981, "f1" (object 999) is moved into
"dir1" at T4; back before T3, an object of the same name existed
but was terminated. In this case, the old object is re-born; its
DateTimeTerminated at object 992 is reset from T3 to NULL. No new
object is created; instead, a new version page (object 999) is
created for the old object to fit in the new file. The
DateTimeEnded of object 998 is set to T3 and the DateTimeModified
of object 999 is set to T4, so there is a version time gap during
which the file never existed. At T4, in this case, there are three
versions of "f1" in the entire history.
[0211] A representative file object relocation algorithm is set
forth below: TABLE-US-00010 Event = MOVE or RENAME RelocateFile
(File A, old path, new path, at time T) (Step 1705, 1707) Open
current parent of A, P1, using the old path. P1 has k versions.
Create a new version (k + 1) and copy all the properties of k to (k
+ 1). Set DateTimeEnded (P1, version k) = T. Set DateTimeModified
(P1, version k + 1) = T, DateTimeEnded (P1, version k + 1) = NULL
Remove A from CHILDREN property (P1, version k + 1) Change CHILDREN
property in latest version of parent of P1 to refer to (k + 1). If
new parent of A is not the same as its current parent, do the
following: otherwise, P2 is the same as P1: Open new parent of A,
P2, using the new path. P2 has i versions. Create a new version (i
+ 1) and copy all the properties of i to (i + 1). Set DateTimeEnded
(P2, version i) = T. Set DateTimeModified (P2, version i + 1) = T,
DateTimeEnded (P2, version i + 1) = NULL Change CHILDREN property
in latest version of parent of P2 to refer to (i + 1). If an object
existed in the new path (call it Z with r number of version),
terminate it by setting DateTimeEnded (Z, version r) = T, and
DateTimeTerminated (Z, anchor) = T. Remove version r of Z from the
parent (P1) CHILDREN list. If versioned by object instant (FIG. 21)
Open file object A which has m version. Create a new version (m +
1). Copy all properties of m to m + 1. Set DateTimeEnded (A,
version m) = T Set DateTimeModified (A, version m + 1) = T,
DateTimeEnded (A, version m + 1) = NULL Set Name (A, version m + 1)
if name is to be changed as well Add Object (A, version m + 1) to
CHILDREN property of (P2, version i + 1). Set PARENT property of
(A, version m + 1) to (P2, version i + 1) If versioned by object
path without connecting to dead object (FIG. 20, and variant 980 of
FIG. 22) Create new object C with version 1. Open file object A
which has m version. Copy properties (object A, version m) to
properties (object C, version 1) --this copies over the sparse
index so that the binary and delta pages can be shared. Set
DateTimeCreated (object C, anchor) = T, DateTimeTerminated (object
C, anchor) = NULL, DateTimeModified (Object C, version 1) = T,
DateTimeEnded (Object C, version 1) = NULL Set Name (C, version 1)
if name is to be changed as well Set DateTimeTerminated (Object A,
anchor) = T, DateTimeEnded (Object A, version m) = T. Add Object
(C, version 1) to CHILDREN property of (P2, version i + 1). Set
PARENT property (C, version 1) to (P2, version i + 1) If versioned
by object path and connecting to dead object (FIG. 21, and variant
981 of FIG. 22) Open object C which is a dead object in the new
path with the same final name as object A. Object C has n version.
Create n + 1 version. Open file object A which has m version. Copy
properties (object A, version m) to properties (object C, version n
+ 1) --this copies over the sparse index so that the binary and
delta pages can be shared. Set DateTimeTerminated (object C,
anchor) = NULL, DateTimeModified (Object C, version n + 1) = T,
DateTimeEnded (Object C, version n + 1) = NULL Set Name (C, version
n + 1) if name is to be changed as well Set DateTimeTerminated
(Object A, anchor) = T, DateTimeEnded (Object A, version m) = T.
Add Object (C, version n + 1) to CHILDREN property of (P2, version
i + 1). Set PARENT property of (C, version n + 1) to (P2, version i
+ 1). Close all the objects.
[0212] The algorithm set forth above shows relocation of an object
within a protected file system or folder (i.e., moving from a
sub-folder to another sub-folder within a protected file system
tree). When an object is relocated from a protected area to an
unprotected are (e.g., outside of the protected file system), the
handling process is similar to DELETE. This is because the object
disappears from the protected area after the event. Conversely,
when an object is moved from an unprotected area to a protected
area, it is treated as a creation of a new object.
[0213] The DMS may also include a temporary file handling process
to minimize storage and bandwidth usage, e.g., by not transferring
and storing potentially useless temporary files in the file system
history. For example, Microsoft Word creates a temporary file when
a Word document is modified. The updates are entered into a
temporary file; upon a save event, the temporary file is renamed to
the original file name. In this case, preferably creation and/or
modification of the temporary file may be avoided; thus,
preferably, only when the RENAME occurs is the above relocation
process applied and object history linked together by pathname.
[0214] The above algorithm implements relocation of a file object;
relocation of a directory object is more complex, as a directory
may have children. Similar to relocating a file, the history
management of relocation of a directory can also be based on model
1 (versioned by object instance) or model 2 (versioned by path).
Also, one may connect the relocated object with a dead historical
object while versioned by path or without connecting the relocated
object with the historical object.
[0215] FIGS. 23-26 illustrate the DMS data structure changes during
relocation of a directory object. As in the previous examples,
these figures do not show the object anchor pages. FIG. 23
illustrates a sample baseline structure. FIG. 24 illustrates a
directory relocation based on versioned by object instance model
using FIG. 23 as the initial baseline. FIG. 25 illustrates a
directory relocation based on versioned by object path model using
FIG. 23 as the initial baseline. FIG. 26 illustrates a directory
relocation based on a combination of versioned by object path and
versioned by object instance, again using FIG. 23 as the initial
baseline.
[0216] In FIG. 24, "dir4" is relocated from "dir2" to "dir3" using
versioned by object instance model and figure FIG. 23 as the
baseline. In this case, a new version of "dir2" is created (object
1031) and its CHILDREN property is modified such that it no-longer
links to "dir4." The DateTimeEnded of object 1005 and the
DateTimeModified property of object 1031 are set to the time when
"dir4" is relocated. The CHILDREN property of parent of "dir2,"
which is version 2 of "dir1" (object 1002), is adjusted to refer to
object 1031 instead of object 1005. Note that because there is no
addition or deletion to "dir1," there is no need to create a new
version. A new version of "dir3" 1030 is created, its CHILDREN
reference is added "dir4" version 3 (object 1032), and its
DateTimeModified property is set to the time when "dir4" is
relocated. The parent reference of "dir3" (which is version 2 of
"dir1" 1002) is also adjusted to refer to object 1030. New version
of "dir4" 1032 is created with the same CHILDREN reference as its
previous version 1009; the NAME property and PARENT property of
object 1032 is changed accordingly. Versioned by object instance
for directory relocation is straightforward as the version of the
children does not have to be adjusted (because the instance has not
changed).
[0217] FIG. 25 shows relocating of "dir4" from "dir2" to "dir3"
using the versioned by object path model. As in FIG. 24, new
versions of "dir2" and "dir3" are created with their CHILDREN link,
DateTimeModified property, and parent property adjusted
accordingly. In this model, when the object "dir4" 1009 is moved,
the last version (object) 1009 ended, and the object is terminated;
a new object for "di43" is created that has its own anchor page and
a first version page 1042. All the descendents of "dir4", namely
objects 1010, 1013, and 1014, are also terminated and new
corresponding objects (objects 1043, 1044, and 1045) created. The
file object "f2" (version 1, 1043) and "f3" (version 1, 1045) refer
to the same binary and delta pages for shared content storage usage
reduction.
[0218] As can be seen, the versioned by object path model has
relatively high processing and storage cost when relocating a
directory, although it is much simpler to traverse. The alternative
is to have a versioned by object path and versioned by object
instance hybrid as shown in FIG. 26. In FIG. 26, "dir4" version 2
(object 1009) is terminated when it is relocated from "dir2" to
dir3," a new object "dir4" 1052 with a new anchor page and first
version page is created. A new version is created for both "dir2"
1051 and "dir3" 1050, and their parent link is fixed. The new
version of "dir3" (1050) refers to version 1 of the new object
1052, while version 3 of "dir2" (1051) no longer refers to "dir4."
New NAME property may be entered into object 1052 as relocation may
also change the name of the directory. The descendents of "dir4" do
not change in this case.
[0219] The following is a directory object relocating algorithm.
TABLE-US-00011 Event = MOVE or RENAME RelocateDirectory (directory
A, old path, new path, at time T) (Steps 1705, 1707) Open current
parent of A, P1, using the old path. P1 has k versions. Create a
new version (k + 1) and copy all the properties of k to (k + 1).
Set DateTimeEnded (P1, version k) = T. Set DateTimeModified (P1,
version k + 1) = T, DateTimeEnded (P1, version k + 1) = NULL Remove
A from CHILDREN property (P1, version k + 1) Change CHILDREN
property in latest version of parent of P1 to refer to (k + 1). If
new parent of A is the same as the old one, then P2 = P1,
otherwise, do the following: Open new parent of A, P2, using the
new path. P2 has i versions. Create a new version (i + 1) and copy
all the properties of i to (i + 1). Set DateTimeEnded (P2, version
i) = T. Set DateTimeModified (P2, version i + 1) = T, DateTimeEnded
(P2, version i + 1) = NULL Change CHILDREN property in latest
version of parent of P2 to refer to (i + 1). If an object already
owns the new path name, open that object to terminate it. Remove
the object version page from P2. If versioned by object instant
(FIG. 24) Open directory object A which has m version. Create a new
version (m + 1). Copy all properties of m to m + 1. Set
DateTimeEnded (A, version m) = T Set DateTimeModified (A, version m
+ 1) = T, DateTimeEnded (A, version m + 1) = NULL Set Name (A,
version m + 1) if name is to be changed as well Add Object (A,
version m + 1) to CHILDREN property of (P2, version i + 1). Set
PARENT property of A to refer to (P2, version i + 1). If versioned
by combination of object path and object instance and do not
connect to dead object (FIG. 26) Create new object C with version
1. Open directory object A which has m version. Copy properties
(object A, version m) to properties (object C, version 1) --this
copies over all the children links. Set DateTimeCreated (object C,
anchor) = T, DateTimeTerminated (object C, anchor) = NULL,
DateTimeModified (Object C, version 1) = T, DateTimeEnded (Object
C, version 1) = NULL Set Name (C, version 1) if name is to be
changed as well Set DateTimeTerminated (Object A, anchor) = T,
DateTimeEnded (Object A, version m) = T. Add Object (C, version 1)
to CHILDREN property of (P2, version i + 1). Set PARENT property of
C to refer to (P2, version i + 1). If versioned by object path
without connecting to dead object (FIG. 25) Perform object close
for directory A. Traverse to all the descendents of directory A,
terminate all the descendent using relocate with versioned by
object path method. For each of the descendent, terminate the old
one, create a new object with version 1, reset the parent-child
link to connect the new child (version 1 of child) to the new
parent (version 1 of parent). Close all the objects.
[0220] The above algorithm only illustrates the process to relocate
a directory object from a sub-folder to another sub-folder, all
within a protected file system. In the event a directory is
relocated outside of the protected file system, the process is the
same as that for deleting the direct object (i.e., the same process
for a DELETE event). In the event a directory is relocated from an
unprotected file system to a protected file system, the process is
similar triggered by a CREATE event as has been described
above.
[0221] The present invention has been described by a set of example
data structures, but this is not a limitation of the present
invention. Thus, while is has been convenient to explain the
various file and directory object algorithms (create, modify,
delete, rename and relocate) by reference to a continuous sequence
of sample data structures, these algorithms may be used with any
object store as the starting or baseline data structure.
[0222] As one of ordinary skill in the art will appreciate, the
present invention addresses enterprise data protection and data
management problems by continuously protecting all data changes and
transactions in real time across local and wide area networks.
Preferably, and as illustrated in FIG. 1, the method and system of
the invention take advantage of inexpensive, commodity processors
to efficiently parallel process and route application-aware data
changes between applications and low cost near storage.
[0223] While the present invention has been described in the
context of a method or process, the present invention also relates
to apparatus for performing the operations herein. As described
above, this apparatus may be specially constructed for the required
purposes, or it may comprise a general purpose computer selectively
activated or reconfigured by a computer program stored in the
computer. Such a computer program may be stored in a computer
readable storage medium, such as, but is not limited to, any type
of disk including optical disks, CD-ROMs, and magnetic-optical
disks, read-only memories (ROMs), random access memories (RAMs),
magnetic or optical cards, or any type of media suitable for
storing electronic instructions, and each coupled to a computer
system bus.
[0224] While the above written description also describes a
particular order of operations performed by certain embodiments of
the invention, it should be understood that such order is
exemplary, as alternative embodiments may perform the operations in
a different order, combine certain operations, overlap certain
operations, or the like. References in the specification to a given
embodiment indicate that the embodiment described may include a
particular feature, structure, or characteristic, but every
embodiment may not necessarily include the particular feature,
structure, or characteristic.
[0225] Although different data services have been described, it
should be appreciated that one advantage of the present invention
is that a given DMS appliance can provide one or more such
services. It is a particular advantage that a given appliance can
provide a consolidated set of data services on behalf of a
particular data source. As has been described, this operation is
facilitated through the provision of the host driver, which
captures, as a data history, an application-aware data stream. The
application-aware data stream typically comprises data change(s),
events associated with the data change(s), and metadata associated
with the data change(s). As has been described, this information is
streamed continuously to the appliance (or to a set of cooperating
appliances) to facilitate provision by the appliance of the desired
service(s).
[0226] As noted above, it is not required that the present
invention be implemented in an "object-oriented" manner, even
though it is one preferred implementation to do so. As described
above, the invention may be implemented in a non object-oriented
manner, e.g., by overlaying the logical representation over other
data storage such as a relational database. Thus, as used herein,
an "object" is not limited to a construct that is used solely in an
"object-oriented" implementation but should be construed broadly.
Thus, in the context of a relational database implementation, an
"object" may be represented by a row of data from one or more
relational database tables, and an associated "link" may simply be
a reference from one row to another row (e.g., a data row address
or the like).
* * * * *