U.S. patent application number 12/712245 was filed with the patent office on 2010-06-17 for systems and methods for resynchronizing information.
Invention is credited to Vijay Agrawal, David Ngo.
Application Number | 20100153338 12/712245 |
Document ID | / |
Family ID | 42543192 |
Filed Date | 2010-06-17 |
United States Patent
Application |
20100153338 |
Kind Code |
A1 |
Ngo; David ; et al. |
June 17, 2010 |
Systems and Methods for Resynchronizing Information
Abstract
Methods and systems for synchronizing data files in a storage
network between a first and a second storage device is provided.
The method includes storing first data files associated with the
first storage device to a storage medium, whereby the first data
files include first data records. The storage medium may then be
transferred to the second storage device. The first data files from
the storage medium may be loaded onto the second storage device.
The second data records from the first storage device may be
received, and the first and second data records are compared. The
first data files at the second storage device may be updated based
on the comparison of the first and second data records.
Inventors: |
Ngo; David; (Shrewsbury,
NJ) ; Agrawal; Vijay; (Ocean, NJ) |
Correspondence
Address: |
MCDERMOTT WILL & EMERY LLP
600 13TH STREET, N.W.
WASHINGTON
DC
20005-3096
US
|
Family ID: |
42543192 |
Appl. No.: |
12/712245 |
Filed: |
February 25, 2010 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
11640024 |
Dec 15, 2006 |
|
|
|
12712245 |
|
|
|
|
60752201 |
Dec 19, 2005 |
|
|
|
60752198 |
Dec 19, 2005 |
|
|
|
60752196 |
Dec 19, 2005 |
|
|
|
60752202 |
Dec 19, 2005 |
|
|
|
60752197 |
Dec 19, 2005 |
|
|
|
Current U.S.
Class: |
707/610 ;
707/E17.005 |
Current CPC
Class: |
G06F 3/0653 20130101;
G06F 11/1458 20130101; G06F 3/067 20130101; G06F 11/1469 20130101;
G06F 3/061 20130101; H04L 67/1097 20130101; G06F 3/0631 20130101;
G06F 3/0647 20130101; G06F 9/50 20130101; G06F 16/27 20190101; G06F
16/217 20190101; G06F 16/1787 20190101 |
Class at
Publication: |
707/610 ;
707/E17.005 |
International
Class: |
G06F 17/30 20060101
G06F017/30 |
Claims
1. A method of replicating data on an electronic storage system
network comprising: storing a set of data on a first storage
device, the set of data including a record identifier; copying the
set of data to an intermediary storage device; transferring the set
of data from the intermediary storage device to a second storage
device; comparing the record identifiers of the set of data on the
third storage device to the record identifier of the set of data on
the first storage device; and updating the set of data on the third
storage device upon detection of non-identical record identifiers,
the updated data transmitted across the network.
2. The method of claim 1 wherein the record identifiers include an
update sequence number.
3. The method of claim 1 wherein the updated data includes a
highest update sequence number.
4. The method of claim 1 wherein the record identifiers include a
file reference number.
Description
RELATED APPLICATIONS
[0001] This application is a continuation of U.S. patent
application Ser. No. 11/640,024, filed on Dec. 15, 2006, which
claims the benefit under 35 U.S.C. .sctn.120 from Provisional
Application No. 60/752,201, filed December, 19, 2005 and is
incorporated herein by reference.
[0002] This application is related to the following patents and
pending applications, each of which is hereby incorporated herein
by reference in its entirety: [0003] Application titled "Systems
and Methods for Classifying and Transferring Information in a
Storage Network" filed Dec. 19, 2005, attorney docket number
4982/75; [0004] Application Ser. No. 60/752,198 titled "Systems and
Methods for Granular Resource Management in a Storage Network"
filed Dec. 19, 2005, attorney docket number 4982/84; [0005]
Application Serial No. not known, titled "Systems and Methods for
Performing Multi-Path Storage Operations" filed Dec. 19, 2005,
attorney docket number 4982/88; [0006] Application Ser. No.
60/752,196 titled "System and Method for Migrating Components in a
Hierarchical Storage Network" filed Dec. 19, 2005, attorney docket
number 4982/95. [0007] Application Ser. No. 60/752,202 titled
"Systems and Methods for Unified Reconstruction of Data in a
Storage Network" filed Dec. 19, 2005, attorney docket number
4982/97; [0008] Application Ser. No. 60/752,197 titled "Systems and
Methods for Hierarchical Client Group Management" filed Dec. 19,
2005, attorney docket number 4982/102
BACKGROUND OF THE INVENTION
[0009] The invention disclosed herein relates generally to
performing data transfer operations in a data storage system. More
particularly, the present invention relates to facilitating data
synchronization between a source and destination device in a
storage operation system.
[0010] Performing data synchronization is an important task in any
system that processes and manages data. Synchronization is
particularly important when a data volume residing in one location
in a system is to be replicated and maintained on another part of
the system. Replicated data volumes may be used, for example, for
backup repositories, data stores, or in synchronous networks which
may utilize multiple workstations requiring identical data
storage.
[0011] File replication may include continually capturing write
activity on a source computer and transmitting this write activity
from the source computer to a destination or target computer in
real-time or near real-time. A first step in existing file
replication systems, as illustrated in FIG. 1A, is a
synchronization process to ensure that the source data 22 at a
source storage device and the destination data 24 at a destination
storage device are the same. That is, before a destination computer
28 may begin storing write activity associated with the source data
22 at a source computer 26, the system 20 needs to first ensure
that the previously written source data 22 is stored at the
destination computer 28.
[0012] Problems in existing synchronization processes may occur as
a result of low or insufficient bandwidth in a network connection
30 over which the source and destination computers 26, 28
communicate. Insufficient bandwidth over the connection 30
ultimately causes bottlenecks and network congestion. For example,
if the rate of change of data at the source computer 26 is greater
than the bandwidth available on the network connection 30, data
replication may not occur since data at the source computer 26 will
continue to change at a faster rate than it can be updated at the
destination computer 28. Therefore, the attempts to synchronize the
source and destination computers 26, 28 may continue indefinitely
without success and one set of data will always lag behind the
other.
[0013] Additional synchronization problems may arise due to
hardware failure. If either the source computer 26 or the
destination computer 28 were to fail, become unavailable, or have a
failure of one of its storage components, application data may
still be generated without system 20 being able to replicate the
data to the other storage device. Neither computers 26 or 28
possess means of tracking data changes during such a failure. Other
possible sources of disruption of replication operations in
existing systems may include disrupted storage paths, broken
communication links or exceeding the storage capacity of a storage
device.
[0014] Additionally, some existing synchronization systems maintain
continuity across multiple storage volumes using a wholesale copy
routine. Such a routine entails periodically copying the most or
all contents of a storage volume across the network to replace all
the previous replication data. A storage policy or network
administrator may control the operations and determine the
frequency of the storage operation. Copying the entire contents of
a storage volume across a network to a replication storage volume
may be inefficient and can overload the network between the source
computer 26 and the destination computer 28. Copying the entire
volume across the network connection 30 between the two computers
causes the connection 30 to become congested and unavailable for
other operations or to other resources, which may lead to hardware
or software operation failure, over-utilization of storage and
network resources and lost information. A replication operation as
described above may also lack the capability to encrypt or secure
data transmitted across the network connection 30. A replication
operation that takes place over a public network, such as the
Internet, or publicly accessible wide area network ("WAN"), can
subject the data to corruption or theft.
SUMMARY OF THE INVENTION
[0015] In accordance with some aspects of the present invention, a
method of synchronizing data files with a storage operation between
a first and a second storage device is provided. The method may
include storing first data files associated with the first storage
device to a storage medium, whereby the first data files include
first data records. The storage medium may then be transferred to
the second storage device. The first data files from the storage
medium may be stored on the second storage device. The second data
records from the first storage device may be received, and the
first and second data records may be compared. The first data files
at the second storage device may be updated based on the comparison
of the first and second data records.
[0016] In accordance with other embodiments of the present
invention, a method of synchronizing data after an interruption of
data transfer between a first and a second storage device is
provided. The method may include detecting an interruption in the
data transfer between the first and the second storage device, and
comparing first logged data records in a first data log associated
with the first storage device with second logged records in a
second data log associated with the second storage device. Updated
data files from the first storage device may then be sent to the
second storage device based on comparison the first and the second
logged records.
[0017] One embodiment of the present invention includes a method of
synchronizing data between a first and second storage device. The
method may include identifying a first set of data on a first
storage device for replication and capture the set of data in a
first log entry. Changes to the first set of data may be determined
and recorded as a second set data in a suitable log or data
structure for recording such data. Next, the first and second set
of data may be transmitted to the second storage device and any
changes replicated in the second storage device.
[0018] Another embodiment of the present invention includes a
method of synchronizing data after an interruption of data transfer
between a first and a second storage device. When an interruption
in the data transfer between the first and the second storage
device is detected, the first logged data records in a first data
log associated with the first storage device are compared with
second logged records in a second data log associated with the
second storage device. Updated data files from the first storage
device are then sent to the second storage device based on
comparing the first and the second logged records.
[0019] In yet another embodiment, a method of replicating data on
an electronic storage system network is presented. A set of data,
including a record identifier, is stored on a first storage device
and copied to an intermediary storage device. The set of data from
the intermediary storage device may then be transferred to a third
storage device. The record identifier of the set of data on the
third storage device may then be compared to the record identifier
of the set of data on the first storage device. The set of data on
the third storage device is updated upon detection of non-identical
record identifiers, wherein the updated data files are transmitted
across the storage network.
[0020] In another embodiment, a system for replicating data on an
electronic storage network is presented. The system includes a
first and second storage device, a first log, for tracking changes
to data stored on the first storage device, and a replication
manager module. The replication manager module transmits updated
data from the first log to the second storage device.
[0021] In another embodiment, a computer-readable medium having
stored thereon a plurality of sequences of instructions is
presented. When executed by one or more processors the sequences
cause an electronic device to store changes to data on a first
storage device in a first log including record identifiers. Updated
data is transmitted from the first log to a second log on a second
storage device where the record identifier of the data from the
first log is compared to the record identifier of the data from the
second log. The second storage device is updated with the updated
data upon detecting a difference in the record identifiers.
[0022] In another embodiment, a computer-readable medium having
stored thereon a plurality of sequences of instructions is
presented. When executed by one or more processors the sequences
cause an electronic device to detect a failure event in a data
replication operation between first and second storage devices.
Updates of a first set of data are stored in the first storage
device. A second set of data detailing the updates to the first set
of data is logged. The second set of data also includes a record
identifier which is compared to a record identifier of the second
storage device. The updates to the first set of data, identified by
the second set of data, are replicated on the second storage
device.
BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The invention is illustrated in the figures of the
accompanying drawings which are meant to be exemplary and not
limiting, in which like references are intended to refer to like or
corresponding parts, and in which:
[0024] FIG. 1 is a block diagram of a prior art system;
[0025] FIG. 2 is a block diagram of a system for performing storage
operations on electronic data in a computer network according to an
embodiment of the invention;
[0026] FIG. 3A is a block diagram of storage operation system
components utilized during synchronization operations according to
an embodiment of the invention;
[0027] FIG. 3B is an exemplary data format associated with logged
data entries according to an embodiment of the invention;
[0028] FIG. 4A is a block diagram of storage operation system
components utilized during synchronization operations in accordance
with another embodiment of the invention.
[0029] FIG. 4B is an exemplary data format associated with logged
data record entries according to an embodiment of the
invention;
[0030] FIG. 5 is a flowchart illustrating some of the steps
involved in replication according to an embodiment of the
invention;
[0031] FIG. 6 is a flowchart illustrating some of the steps
involved in replication according to an embodiment of the
invention; and
[0032] FIG. 7 is a flowchart illustrating some of the steps
involved in replication according to another embodiment of the
invention.
DETAILED DESCRIPTION OF THE INVENTION
[0033] Detailed embodiments of the present invention are disclosed
herein, however, it is to be understood that the disclosed
embodiments are merely exemplary of the invention, which may be
embodied in various forms. Therefore, specific functional details
disclosed herein are not to be interpreted as limiting, as a
representative basis for teaching one skilled in the art to
variously employ the present invention in any appropriately
detailed embodiment.
[0034] With reference to FIGS. 2-7, exemplary aspects of
embodiments and features of the present invention are presented.
Turning now to FIG. 2, a block diagram of a storage operation cell
50 that may perform storage operations on electronic data in a
computer network in accordance with an embodiment of the present
invention is illustrated. As shown, storage operation cell 50 may
generally include a storage manager 100, a data agent 95, a media
agent 105, a storage device 115, and, may include certain other
components such as a client computer 85, a data or information
store 90, databases 110,111, a jobs agent 120, an interface module
125, a management agent 130, and a resynchronization agent 133.
Such system and elements thereof are exemplary of a modular storage
management system such as the CommVault QiNetix.TM. system, and
also the CommVault GALAXY.TM. backup system, available from
CommVault Systems, Inc. of Oceanport, N.J. , and further described
in U.S. Pat. No. 7,035,880, which is incorporated herein by
reference in its entirety.
[0035] A storage operation cell, such as cell 50, may generally
include combinations of hardware and software components associated
with performing storage operations on electronic data. Exemplary
storage operation cells according to embodiments of the invention
may include, as further described herein, CommCells as embodied in
the QNet storage management system and the QiNetix storage
management system by CommVault Systems of Oceanport, New Jersey.
According to some embodiments of the invention, storage operations
cell 50 may be related to backup cells and provide some or all of
the functionality of backup cells as described in application Ser.
No. 10/877,831 which is hereby incorporated by reference in its
entirety.
[0036] Storage operations performed by storage operation cell 50
may include creating, storing, retrieving, and migrating primary
data copies and secondary data copies (which may include, for
example, snapshot copies, backup copies, HSM (Hierarchical Storage
Management) copies, archive copies, and other types of copies of
electronic data). Storage operation cell 50 may also provide one or
more integrated management consoles for users or system processes
to interface with in order to perform certain storage operations on
electronic data as further described herein. Such integrated
management consoles may be displayed at a central control facility
or several similar consoles distributed throughout multiple network
locations to provide global or geographically specific network data
storage information. The use of integrated management consoles may
provide a unified view of the data operations across the
network.
[0037] A unified view of the data operations collected across the
entire storage network may provide an advantageous benefit in the
management of the network. The unified view may present the system,
or system administrator with a broad view of the utilized resources
of the network. Presenting such data to one centralized management
console may allow for a more complete and efficient administration
of the available resources of the network. The storage manager 100,
either via a preconfigured policy or via a manual operation from a
system administrator, can reallocate resources to more efficiently
run the network. Data paths from storage operation cells may be
re-routed to avoid areas of the network which are congested by
taking advantage of underutilized data paths or operation cells.
Additionally, should a storage operation cell arrive at or exceed a
database size maximum, storage device capacity maximum or fail
outright, several routes of redundancy may be triggered to ensure
the data arrives at the location for which it was intended. A
unified view may provide the manager with a collective status of
the entire network allowing the system to adapt and reallocate the
many resources of the network for faster and more efficient
utilization of those resources.
[0038] In some embodiments, storage operations may be performed
according to a storage policy. A storage policy generally may be a
data structure or other information source that includes a set of
preferences and other storage criteria for performing a storage
operation and/or other functions that relate to storage operation.
The preferences and storage criteria may include, but are not
limited to, a storage location, relationships between system
components, network pathway to utilize, retention policies, data
characteristics, compression or encryption requirements, preferred
system components to utilize in a storage operation, and other
criteria relating to a storage operation. For example, a storage
policy may indicate that certain data is to be stored in a specific
storage device, retained for a specified period of time before
being aged to another tier of secondary storage, copied to
secondary storage using a specified number of streams, etc. In one
embodiment, a storage policy may be stored in a storage manager
database 111. Alternatively, certain data may be stored to archive
media as metadata for use in restore operations or other storage
operations. In other embodiments, the data may be stored to other
locations or components of the system.
[0039] A schedule policy specifies when and how often to perform
storage operations and may also specify performing certain storage
operations (i.e. replicating certain data) on sub-clients of data
including how to handle those sub-clients. A sub-client may
represent static or dynamic associations of portions of data of a
volume and are generally mutually exclusive. Thus, a portion of
data may be given a label and the association is stored as a static
entity in an index, database or other storage location used by the
system. Sub-clients may also be used as an effective administrative
scheme of organizing data according to data type, department within
the enterprise, storage preferences, etc. For example, an
administrator may find it preferable to separate e-mail data from
financial data using two different sub-clients having different
storage preferences, retention criteria, etc.
[0040] Storage operation cells may contain not only physical
devices, but also may represent logical concepts, organizations,
and hierarchies. For example, a first storage operation cell 50 may
be configured to perform HSM operations, such as data backup or
other types of data migration, and may include a variety of
physical components including a storage manager 100 (or management
agent 130), a media agent 105, a client component 85, and other
components as described herein. A second storage operation cell may
contain the same or similar physical components, however, it may be
configured to perform storage resource management ("SRM")
operations, such as monitoring a primary data copy or performing
other known SRM operations.
[0041] In one embodiment a data agent 95 may be a software module
or part of a software module that is generally responsible for
archiving, migrating, and recovering data from client computer 85
stored in an information store 90 or other memory location. Each
computer 85 may have at least one data agent 95 and a
resynchronization agent 133. Storage operation cell 50 may also
support computers 85 having multiple clients (e.g., each computer
may have multiple applications, with each application considered as
either a client or sub-client).
[0042] In some embodiments, the data agents 95 may be distributed
between computer 85 and the storage manager 100 (and any other
intermediate components (not explicitly shown)) or may be deployed
from a remote location or its functions approximated by a remote
process that performs some or all of the functions of the data
agent 95. The data agent 95 may also generate metadata associated
with the data that it is generally responsible for replicating,
archiving, migrating, and recovering from client computer 85. This
metadata may be appended or embedded within the client data as it
is transferred to a backup or secondary storage location, such as a
replication storage device, under the direction of storage manager
100.
[0043] One embodiment may also include multiple data agents 95,
each of which may be used to backup, migrate, and recover data
associated with a different application. For example, different
individual data agents 95 may be designed to handle MICROSOFT
EXCHANGE.RTM. data, MICROSOFT SHAREPOINT data or other
collaborative project and document management data, LOTUS
NOTES.RTM. data, MICROSOFT WINDOWS 2000.RTM. file system data,
MICROSOFT Active Directory Objects data, and other types of data
known in the art. Alternatively, one or more generic data agents 95
may be used to handle and process multiple data types rather than
using the specialized data agents described above.
[0044] In an embodiment utilizing a computer 85 having two or more
types of data, one data agent 95 may be used for each data type to
archive, migrate, and restore the client computer 85 data. For
example, to backup, migrate, and restore all of the data on a
MICROSOFT EXCHANGE 2000.RTM. server, the computer 85 may use one
MICROSOFT EXCHANGE 2000.RTM. Mailbox data agent to backup the
EXCHANGE 2000.RTM. mailboxes, one MICROSOFT EXCHANGE 2000.RTM.
Database data agent to backup the EXCHANGE 2000.RTM. databases, one
MICROSOFT EXCHANGE 2000.RTM. Public Folder data agent to backup the
EXCHANGE 2000.RTM. Public Folders, and one MICROSOFT WINDOWS
2000.RTM. File System data agent to backup the file system of the
computer 85. These data agents 95 would be treated as four separate
data agents 95 by the system even though they reside on the same
computer 85.
[0045] In an alternative embodiment, one or more generic data
agents 95 may be used, each of which may be capable of handling two
or more data types. For example, one generic data agent 95 may be
used to back up, migrate and restore MICROSOFT EXCHANGE 2000.RTM.
Mailbox data and MICROSOFT EXCHANGE 2000.RTM. Database data while
another generic data agent may handle MICROSOFT EXCHANGE 2000.RTM.
Public Folder data and MICROSOFT WINDOWS 2000.RTM. File System
data.
[0046] While the illustrative embodiments described herein detail
data agents implemented, specifically or generically, for Microsoft
applications, one skilled in the art should recognize that other
application types (i.e. Oracle data, SQL data, Lotus Notes, etc.)
may be implemented without deviating from the scope of the present
invention.
[0047] Resynchronization agent 133 may initiate and manage system
backups, migrations, and data recovery. Although resynchronization
agent 133 is shown as being part of each client computer 85, it may
exist within the storage operation cell 50 as a separate module or
may be integrated with or part of a data agent (not shown). In
other embodiments, resynchronization agent 133 may be resident on a
separate host. As a separate module, resynchronization agent 133
may communicate with all or some of the software modules in storage
operation cell 50. For example, resynchronization agent 133 may
communicate with storage manager 100, other data agents 95, media
agents 105, and/or storage devices 115.
[0048] In one embodiment, the storage manager 100 may include a
software module (not shown) or other application that may
coordinate and control storage operations performed by storage
operation cell 50. The storage manager 100 may communicate with the
elements of storage operation cell 50 including computers 85, data
agents 95, media agents 105, and storage devices 115.
[0049] In one embodiment the storage manager 100 may include a jobs
agent 120 that monitors the status of some or all storage
operations previously performed, currently being performed, or
scheduled to be performed by the storage operation cell 50. The
jobs agent 120 may be linked with an interface module 125
(typically a software module or application). The interface module
125 may include information processing and display software, such
as a graphical user interface ("GUI"), an application program
interface ("API"), or other interactive interface through which
users and system processes can retrieve information about the
status of storage operations. Through the interface module 125,
users may optionally issue instructions to various storage
operation cells 50 regarding performance of the storage operations
as described and contemplated by embodiment of the present
invention. For example, a user may modify a schedule concerning the
number of pending snapshot copies or other types of copies
scheduled as needed to suit particular needs or requirements. As
another example, a user may utilize the GUI to view the status of
pending storage operations in some or all of the storage operation
cells in a given network or to monitor the status of certain
components in a particular storage operation cell (e.g., the amount
of storage capacity left in a particular storage device). As a
further example, the interface module 125 may display the cost
metrics associated with a particular type of data storage and may
allow a user to determine the overall and target cost metrics
associated with a particular data type. This determination may also
be done for specific storage operation cells 50 or any other
storage operation as predefined or user-defined (discussed in more
detail below).
[0050] One embodiment of the storage manager 100 may also include a
management agent 130 that is typically implemented as a software
module or application program. The management agent 130 may provide
an interface that allows various management components in other
storage operation cells 50 to communicate with one another. For
example, one embodiment of a network configuration may include
multiple cells adjacent to one another or otherwise logically
related in a WAN or LAN configuration (not explicitly shown). With
this arrangement, each cell 50 may be connected to the other
through each respective management agent 130. This allows each cell
50 to send and receive certain pertinent information from other
cells 50 including status information, routing information,
information regarding capacity and utilization, etc. These
communication paths may also be used to convey information and
instructions regarding storage operations.
[0051] In an illustrative embodiment, the management agent 130 in
the first storage operation cell 50 may communicate with a
management agent 130 in a second storage operation cell regarding
the status of storage operations in the second storage operation
cell. Another illustrative example may include a first management
agent 130 in a first storage operation cell 50 that may communicate
with a second management agent in a second storage operation cell
to control the storage manager (and other components) of the second
storage operation cell via the first management agent 130 contained
in the storage manager 100 of the first storage operation cell.
[0052] Another illustrative example may include the management
agent 130 in the first storage operation cell 50 communicating
directly with and controlling the components in the second storage
management cell 50, bypassing the storage manager 100 in the second
storage management cell. In an alternative embodiment, the storage
operation cells may also be organized hierarchically such that
hierarchically superior cells control or pass information to
hierarchically subordinate cells or vice versa.
[0053] The storage manager 100 may also maintain, in an embodiment,
an index cache, a database, or other data structure 111. The data
stored in the database 111 may be used to indicate logical
associations between components of the system, user preferences,
management tasks, Storage Resource Management (SRM) data,
Hierarchical Storage Management (HSM) data or other useful data.
The SRM data may, for example, include information that relates to
monitoring the health and status of the primary copies of data
(e.g., live or production line copies). HSM data may, for example,
be related to information associated with migrating and storing
secondary data copies including archival volumes to various storage
devices in the storage system. As further described herein, some of
this information may be stored in a media agent database 110 or
other local data store. For example, the storage manager 100 may
use data from the database 111 to track logical associations
between the media agents 105 and the storage devices 115.
[0054] From the client computer 85, resynchronization agent 133 may
maintain and manage the synchronization of data both within the
storage operation cell 50, and between the storage operation cell
50 and other storage operation cells. For example,
resynchronization agent 133 may initiate and manage a data
synchronization operation between data store 90 and one or more of
storage devices 115. Resynchronization agent 133 may also initiate
and manage a storage operation between two data stores 90 and
associated storage devices, each in a separate storage operation
cell implemented as primary storage. Alternatively,
resynchronization agent 133 may be implemented as a separate
software module that communicates with the client 85 for
maintaining and managing resynchronization operations.
[0055] In one embodiment, a media agent 105 may be implemented as a
software module that conveys data, as directed by the storage
manager 100, between computer 85 and one or more storage devices
115 such as a tape library, a magnetic media storage device, an
optical media storage device, or any other suitable storage device.
Media agents 105 may be linked with and control a storage device
115 associated with a particular media agent. In some embodiments,
a media agent 105 may be considered to be associated with a
particular storage device 115 if that media agent 105 is capable of
routing and storing data to particular storage device 115.
[0056] In operation, a media agent 105 associated with a particular
storage device 115 may instruct the storage device to use a robotic
arm or other retrieval means to load or eject a certain storage
media, and to subsequently archive, migrate, or restore data to or
from that media. The media agents 105 may communicate with the
storage device 115 via a suitable communications path such as a
SCSI (Small Computer System Interface), fiber channel or wireless
communications link or other network connections known in the art
such as a WAN or LAN. Storage device 115 may be linked to a data
agent 105 via a Storage Area Network ("SAN").
[0057] Each media agent 105 may maintain an index cache, a
database, or other data structure 110 which may store index data
generated during backup, migration, and restore and other storage
operations as described herein. For example, performing storage
operations on MICROSOFT EXCHANGE.RTM. data may generate index data.
Such index data provides the media agent 105 or other external
device with a fast and efficient mechanism for locating the data
stored or backed up. In some embodiments, storage manager database
111 may store data associating a computer 85 with a particular
media agent 105 or storage device 115 as specified in a storage
policy. The media agent database 110 may indicate where,
specifically, the computer data is stored in the storage device
115, what specific files were stored, and other information
associated with storage of the computer data. In some embodiments,
such index data may be stored along with the data backed up in the
storage device 115, with an additional copy of the index data
written to the index cache 110. The data in the database 110 is
thus readily available for use in storage operations and other
activities without having to be first retrieved from the storage
device 115.
[0058] In some embodiments, certain components may reside and
execute on the same computer. For example, a client computer 85
including a data agent 95, a media agent 105, or a storage manager
100 coordinates and directs local archiving, migration, and
retrieval application functions as further described in U.S. Pat.
No. 7,035,880. Thus, client computer 85 can function independently
or together with other similar client computers 85.
[0059] FIG. 3A illustrates a block diagram of a system 200 of
system storage operation system components that may be utilized
during synchronization operations on electronic data in a computer
network in accordance with an embodiment of the present invention.
The system 200 may comprise CLIENT 1 and CLIENT 2 for, among other
things, replicating data. CLIENT 1 may include a replication
manager 210, a memory device 215, a log filter driver 220, a log
225, a file system 230, and a link to a storage device 235.
Similarly, CLIENT 2 may include a replication manager 245, a memory
device 250, a log filter driver 255, a log 260, a file system 265,
and a storage device. Additional logs 261 may also reside on CLIENT
2 in some embodiments.
[0060] In one embodiment, replication manager 210 may be included
in resynchronization agent 133 (FIG. 2). Replication manager 210,
in one embodiment, may manage and coordinate the replication and
transfer of data files between storage device 235 and a replication
volume. As previously described in relation to FIG. 2,
resynchronization agent 133 may be included in client computer 85.
In such an embodiment, replication manager 210 may reside within
resynchronization agent 133 (FIG. 2) in a client computer. In other
embodiments, the replication manager 210 may be part of a computer
operating system (OS). In such embodiments, for example, client
computer 85 (FIG. 2) may communicate and coordinate the data
replication processes with the OS.
[0061] In the exemplary embodiment of FIG. 3A, the replication
process between CLIENT 1 and CLIENT 2 in system architecture 200
may occur, for example, during a data write operation in which
storage data may be transferred from a memory device 215 to a log
filter driver 220. Log filter driver 220 may, among other things,
filter or select specific application data or other data that may
be parsed as part of the replication process that is received from
the memory device 215. For example, ORACLE data, SQL data, or
MICROSOFT EXCHANGE data may be selected by the log filter driver
220. The log filter driver 220 may, among other things, include a
specific application or module that resides on the input/output
("I/O") stack between the memory device 215 and the storage device
235. Once write data passes through the memory device 215 towards
the file system 230, the write data is intercepted and processed by
the log filter driver 220. As the write data is intercepted by the
log filter driver 220, it is also received by the file system 230.
The file system 230 may be responsible for managing the allocation
of storage space on the storage device 235. Therefore, the file
system 230 may facilitate storing the write data to the storage
device 235 associated with CLIENT 1.
[0062] In order to replicate the filtered write data that is
received from the memory device 215, the log filter driver 220 may
send filtered write data to the log 225. The log 225 may include
metadata in addition to write data, whereby the write data entries
in log 225 may include a data format 300, such as that illustrated
in FIG. 3B. Metadata may include information, or data, about the
data stored on the system. Metadata, while generally not including
the substantive operational data of the network is useful in the
administration, security, maintenance and accessibility of
operational data. Examples of metadata include files size, edit
times, edit dates, locations on storage devices, version numbers,
encryption codes, restrictions on access or uses, and tags of
information that may include an identifier for editors. These are
mere examples of common usages of metadata. Any form of data that
describes or contains attributes or parameters of other data may be
considered metadata.
[0063] As illustrated in FIG. 3B, the data format of the logged
write data entries in the log 225 may include, for example, a file
identifier field(s) 302, an offset 304, a payload region 306, and a
timestamp 309. Identifier 302 may include information associated
with the write data (e.g., file name, path, size, computer device
associations, user information, etc.). Timestamp field 309 may
include a timestamp referring to the time associated with its log
entry, and in some embodiments may include a indicator, which may
be unique, such as USN.
[0064] Offset 304 may indicate the distance from the beginning of
the file to the position of the payload data. For example, as
indicated by the illustrative example 308, the offset may indicate
the distance of the payload 310 from the beginning of the file 312.
Thus, using the offset 314 (e.g., offset=n), only the payload 310
(e.g., payload n) that requires replicating is sent from storage
device 235 (FIG. 3A) to the replication volume storage device.
Thereby replicating only that portion of the data that has changed.
The replication process may be sent over the network, for example,
the communication link 275 (FIG. 3A) to another client, CLIENT
2.
[0065] As indicated in FIG. 3A, at CLIENT 2, write data associated
with the log 225 of CLIENT 1 may be received by the log 260 of
CLIENT 2 via the communication link 275. The write data may then be
received by the file system 265 of CLIENT 2 prior to being stored
on the replication volume at the storage device (the replication
volume).
[0066] Referring to FIG. 3A, changes captured by filter driver 220
on CLIENT 1 may later be used to replicate the write data entries
utilizing the log 225, if, for example, a communication failure
occurs between CLIENT 1 and CLIENT 2 due to a network problem
associated with communication link 275. If the failure is of
limited duration the log 225 will not be overwritten by additional
data being logged. Therefore, provided that during a network
failure, the log 225 has enough storage capacity to store recent
entries associated with the write data, the log 225 may be able to
successfully send the recent write data entries to a replication
volume upon restoration of communication.
[0067] The write data entries in the log 225 of CLIENT 1 may
accumulate over time. Replication manager 210 of CLIENT 1 may
periodically direct the write data entries of the log 225 to be
sent to a storage device having the replication volume. During a
network failure, however, the storage capacity of the log 225 may
be exceeded as a result of recent logged entries associated with
the write data. Upon such an occurrence, the log filter driver 220
may begin to overwrite the oldest entries associated with the write
data. Replication of the write data associated with the overwritten
entries may not be possible. Thus, the present embodiment allows
for a full synchronization of data files between the storage device
235 and a replication volume which may be necessary to ensure the
data volume in the storage device 235 associated with CLIENT 1 is
replicated at the replication volume.
[0068] In one embodiment, the storage manager 100 (FIG. 2) may
monitor and control the network resources utilized in the
replication operations. Through a defined storage policy, or
interactive interfacing with a system administrator, the storage
manager 100 may reallocate network resources (e.g. storage
operation paths, storage devices utilized, etc). Reallocating the
resources of the network may alleviate the concentrated traffic and
bottlenecks created by these types of situations in replication
operations.
[0069] FIG. 4A illustrates a block diagram 280 of storage operation
system components that may be utilized during synchronization
operations on electronic data in a computer network in accordance
with another embodiment of the present invention. System 280 is
similar to system 200 (FIG. 3A) and use like reference numbers to
designate generally like components. As shown, system 280 may
include CLIENT 1 and CLIENT 2 for, among other things, replicating
data. CLIENT 1 may include a replication manager 210, a memory
device 215, a log filter driver 220, one or more log files 225, a
change journal filter 240, a change journal 241, a file system 230,
and a storage device 235. Similarly, CLIENT 2 may include a
replication manager 245, a memory device 250, one or more log files
260, 261, and a file system 265. The one or more log files 260, 261
may be utilized for different application types, such as, SQL data,
MICROSOFT EXCHANGE data, etc.
[0070] In one embodiment, the replication manager 210 may be
included in the resynchronization agent 133 (FIG. 2). The
replication manager 210, in one embodiment may manage and
coordinate the replication of data files between storage device 235
and a replication volume. As previously described in relation to
FIG. 2, resynchronization agent 133 may be included in client
computer 85. In such an embodiment, the replication manager 210 may
reside within resynchronization agent 133, in a client computer. In
other embodiments, replication manager 210 may be part of a
computer operating system (OS). In such embodiments, for example,
the client computer 85 (FIG. 2) may communicate and coordinate the
data replication processes with the OS.
[0071] In the exemplary embodiment of FIG. 4A, the replication
process between CLIENT 1 and CLIENT 2 in the system architecture
280 may occur, for example, during a data write operation in which
storage data may be transferred from the memory device 215 of
CLIENT 1 to a storage device 235 via the file system 230. The write
data from the memory 215 device, however, may be intercepted by the
log filter driver 220. As previously described, the log filter
driver 220 may, among other things, trap, filter or select
intercepted application data received from memory 215. For example,
ORACLE data, SQL data, or MICROSOFT EXCHANGE data may be selected
by the log filter driver 220. Once the write data passes through
and is captured by the log filter driver 220, the write data may be
received by the change journal filter driver 240.
[0072] Change journal filter driver 240 may also create data
records that reflect changes made to the data files (e.g., write
activity associated with new file creation, existing file updates,
file deletion, etc.) stored on the storage device 235. These data
records, once selected by the change journal filter driver 240, may
be stored as records in the change journal 241. The replication
manager 210 may then utilize these change journal 241 record
entries during replication operations if access to the log file 225
entries, which may have ordinarily facilitated the replication
process as further described herein, is unavailable (e.g.,
corrupted, deleted, or overwritten entries). Write data may then be
received at the file system 230 from the change journal filter
driver 240, whereby the file system 230 may be responsible for
managing the allocation of storage space and storage operations on
the storage device 235, and copying/transferring data to the
storage device 235.
[0073] In order to replicate the filtered write data that is
received from the memory device 215, the log filter driver 220 may
send write data filtered by the log filter driver 220 to the log
225. The log 225 may include metadata in addition to write data
payloads, whereby the write data entries in the log 225 may include
the data format 300, previously described and illustrated in
relation to FIG. 3B.
[0074] As previously described in relation to the embodiments of
FIGS. 3A and 3B, the present invention provides for replication
operations during both normal and failure occurrences between
CLIENT 1 and CLIENT 2 due to network problems (e.g., failure in
communication link 275). In one embodiment, the filter driver 220
captures changes in the write data that may later be used to
replicate write data entries utilizing the log 225, provided the
failure is of limited duration and the log 225 goes not get
overwritten. Therefore, provided that during a network failure, the
log 220 has enough storage capacity to store recent entries
associated with the write data, the log filter driver 220 may be
able to successfully send the recent write data entries to the
replication upon restoration of communication.
[0075] The write data entries in the log 225 of CLIENT 1 may
accumulate over time. The replication manager 210 of CLIENT 1 may
periodically direct the write data entries of the log 225 to be
sent to the replication volume. During a network failure, however,
the storage capacity of the log 225 may be exceeded as a result of
recent logged entries associated with the write data. Replication
of write data associated with the overwritten entries may not be
possible. Thus, under these conditions, the change journal 241
entries captured by the change journal filter driver 240 may enable
the replication of write data without the need for a full
synchronization of data files between the storage devices 235 and a
replication volume. As previously described, full synchronization
may require a transfer of the entire storage volume stored at the
storage device 235 linked to CLIENT 1 to the replication volume of
CLIENT 2. The present embodiment is advantageous as a full
synchronization operations may place a heavy burden on network
resources, especially considering the large data volume that may
reside on the storage device 235. In addition to the large data
transfer requirement during this operation, other data transfer
activities within the storage operation system may also create
further network bottlenecks.
[0076] With the implementation of the change journal filter driver
240 and the change journal 241, the requirement for a full
synchronization may be obviated. The changed data entries in change
journal 241 may allow for the replication manager to selectively
update the replicated data instead of requiring a full
synchronization that may occupy valuable network resources better
suited for other operations.
[0077] FIG. 4B illustrates some of the data fields 400 associated
with entries within the change journal log 241 according to an
embodiment of the invention. The data fields 400 may include, for
example, a record identifier 402 such as an Update Sequence Number
(USN), metadata 404, and a data object identifier 406 such as a
File Reference Number (FRN). The data object identifier 406 may
include additional information associated with the write data
(e.g., file name, path size, etc.). Each record logged or entered
in change journal 241 via change journal filter driver 240 may have
a unique record identifier number that may be located in the record
identifier field 402. For example, this identifier may be a 64-bit
identifier such as a USN number used in the MICROSOFT Windows.RTM.
OS change journal system. Each of the records that are created and
entered into the change journal 241 is assigned such a record
identifier. In one embodiment, each of the assigned identifiers is
sequentially incremented with the creation of a newly created
record reflecting a change to the data of the client. For example,
an assigned identifier (e.g., USN) associated with the most recent
change to a file on the storage device may include the numerically
greatest record identifier with respect to all previously created
records, thereby indicating the most recent change. The metadata
field 404 may include, among other things, a time stamp of the
record, information associated with the sort of changes that have
occurred to a file or directory (e.g., a Reason member), etc. In
some embodiments, a FRN associated with the data object identifier
406 may include a 64-bit ID that uniquely identifies any file or
directory on a storage volume such as that of the storage device
235 (FIG. 4A).
[0078] In accordance with an embodiment of the invention, as
further described herein, the record identifier fields 402 (FIG.
4B) of each logged record entered in change journal 241 may be
utilized to resynchronize replication operations in conjunction
with replication managers 210, 245 and one or more of the log files
260, 261. Based on the recorded entries in change journal 241, the
replication manager 210 of CLIENT 1 may coordinate the transfer of
files that are to be replicated with replication manager 245 of
CLIENT 2. This may be accomplished as follows. Change journal 241
logs all changes and assigns a USN or FRN to each log entry in log
242 (FIG. 4A). Each log entry may include a timestamp indicating
its recordation in log 242. Periodically, replication manager 210
may send the most recent USN copied to log 242 to the destination.
Next, change journal 241 may be queried for changes since the last
USN copied, which indicates the difference between the log at the
source and the log at the destination, and only those log entries
are replicated. This may be thought of as "resynchronizing" CLIENT
1 and CLIENT 2.
[0079] Once the transfer of files has been coordinated by
replication managers 210, 245, the designated files may be sent
over communication link 275 to the one or more log files 260, 261.
The files received are then forwarded from the one or more log
files 260, 261 to the replication volume.
[0080] FIG. 5 is a flowchart 500 illustrating some of the steps
involved in a replication process in a storage operation system
under substantially normal operating conditions according to an
embodiment of the invention. The replication process of FIG. 5 may
be described with reference to system architecture 280 illustrated
in FIG. 4A to facilitate comprehension. However, it will be
understood this merely represents one possible embodiment of the
invention and should not be construed to be limited to this
exemplary architecture.
[0081] As shown, at step 502, it may be determined whether any
write data (e.g., application specific data) is available for
transfer to the storage device 235 of a first client, whereby the
write data may require replication at the replication volume of a
second client. If the write data (e.g., application data) requiring
replication exists, it may be captured by the log filter driver 220
and logged in the log 225 (step 504). Additionally, through the use
of another data volume filter driver, such as a MICROSOFT Change
Journal filter driver, records identifying any changes to files or
directories (e.g., change journal records) on the storage device
235 of the first client may be captured and stored in the change
journal 241 (step 506).
[0082] In some embodiments, under the direction of the replication
manager 210, the write data stored and maintained in the log 225
may be periodically (e.g., every 5 minutes) sent via a
communications link 275, to the replication volume of the second
client. In an alternative embodiment, under the direction of the
replication manager 210, the write data stored in the log 225 may
be sent via the communications link 275, to the replication volume
when the quantity of data stored in the log 225 exceeds a given
threshold. For example, when write data stored to the log 225
reaches a five megabyte (MB) capacity, all write data entries in
the log 225 may be replicated to the second client.
[0083] Also, in some embodiments, under the direction of the
replication manager 210, record identifiers (e.g., USN numbers)
stored in the change journal 241 may also be periodically (e.g.,
every 5 minutes) sent via the communications link 275 to the
replication manager 245 of the second client. The replication
manager 245 may store these record identifiers in a log file at
CLIENT 2, or at another memory index, or data structure (step 508).
In other embodiments, under the direction of the replication
manager 210, each record written to the change journal 241 may be
directly sent via the communications link 275 to the replication
manager 245.
[0084] At step 510, the record identifiers (e.g., USN numbers) sent
via the communications link 275 and stored in the log file 260 may
be compared with existing record identifiers. Based on a comparison
between the greatest numerical value of a record identifier
received at the log 260 and other record identifiers, replication
data may be identified and replicated to the data volume of the
second client.
[0085] FIG. 6 is a flowchart 600 illustrating some of the steps
involved in a replication resynchronization process in a storage
operation system according to an embodiment of the invention. The
replication process of FIG. 6 may be described with reference to
system architecture 280 illustrated in FIG. 4A to facilitate
comprehension. However, it will be understood this merely
represents one possible embodiment of the invention and should not
be construed to be limited to this exemplary architecture.
[0086] At step 604, if a communication failure affecting
replication or other event criteria, such as log file corruption,
power failure, loss of network, for example, is detected or found
and then restored, the most recent record identifier field (e.g.,
USN number) in the destination log may be accessed and compared
with the last record identifier received from the change journal
log 241. The replication managers 210, 245 may coordinate and
manage the comparison of these record identifier fields, which may
include, in one embodiment, comparing identifier values such as
USNs used in the MICROSOFT change journal (step 606).
[0087] As previously described, write operations or other
activities (e.g., file deletions) associated with each file are
logged in the change journal records having unique identification
numbers (i.e., record identifier) such as a USN number. At step
606, an identification number (e.g., USN number) associated with
the last record identifier field stored at the change journal 241
may be compared with an identification number (e.g., USN number)
associated with the most recent record identifier stored in the log
260 upon restoration of the communication failure or other event.
If it is determined that these identification numbers (e.g., USN
numbers) are not the same (step 608), this may indicate that
additional file activities (e.g., data write to file operations)
may have occurred at the source location (i.e., CLIENT 1), during
the failure. These changes may not have been replicated to the
second client due to the failure. For example, this may be
determined by the last record identifier field's USN number from
the change journal 241 at the source having a larger numerical
value than the USN number associated with the most recent record
identifier field accessed from the log 260. In one embodiment, this
may occur as a result of a log filter driver 220 not capturing an
event (e.g., a data write operation) or overwriting an event. This
may, therefore, lead to a record identifier such as a USN number
not being sent to log file 260 associated with the replication data
volume of the second client.
[0088] Since USN numbers are assigned sequentially, in an
embodiment, the numerical comparison between the last record
identifier field's USN number stored at the log 260 and the most
recent record identifier field's USN number accessed from the
change journal 241 may be used to identify any files that may not
have been replicated at the replication volume (step 610) of the
second client. For example, if the last record identifier field's
USN number (i.e., at log 241) is "5" and the most recently sent
record identifier field's USN number (i.e., at log 260) is "2," it
may be determined that the data objects associated with USN numbers
"3, 4, and 5" have not yet be replicated to the second client. Once
these data files have been identified (e.g., by data object
identifiers such as FRNs in the change journal entries) (step 610),
they may be copied from the storage device 235 of the first client
and sent over the communication link 275 to the second client (step
612). Thus, the data volumes associated with storage devices 235
and the replication volume may be brought back into sync without
the need for resending (or re-copying) all the data files between
the two storage devices.
[0089] In the exemplary embodiments discussed above, a
communication failure may generate an over-flow in the log 225,
which in turn may cause a loss of logged entries. As, previously
described, these lost entries inhibit the replication process upon
restoration of the communication failure. Other failures may also
lead to a loss of logged entries in log 225. For example, these
failures may include, but are not limited to, corrupted entries in
log 225 and/or the inadvertent deletion or loss of entries in log
225.
[0090] FIG. 7 is a flowchart 700 illustrating a replication process
in a storage operation system according to another embodiment of
the invention. The replication process of FIG. 7 may also be
described with reference to system architecture 280 illustrated in
FIG. 4A to facilitate comprehension. However, it will be understood
this merely represents one possible embodiment of the invention and
should not be construed to be limited to this exemplary
architecture.
[0091] The replication process 700 may, in one embodiment, be based
on ensuring that electronic data files at a source storage device
are synchronized with electronic data files at a destination or
target storage device without the need to perform full
synchronization operations over the storage operation network.
[0092] At step 702, the data files stored on a first storage device
235 and the record identifiers associated with the data records at
the first storage device logged in change journal 241 may undergo a
data transfer. Examples of certain data transfers include, but are
not limited to, a block level copy, storage to a first destination
storage medium/media such as magnetic media storage, tape media
storage, optical media storage, or any other storage means having
sufficient retention and storage capacity.
[0093] At step 704, the first destination medium/media, holding
data from the first storage device, may be transferred (e.g., by
vehicle) to a second destination storage device of the second
client in FIG. 4A. At step 706, the data stored on a first
destination medium/media may be loaded onto the second destination
storage device.
[0094] Since copying the data from the first storage device 235 and
journal log 241 onto the first destination medium/media and
transporting the first destination medium/media to the second
destination storage device (e.g., a storage device of the second
client, (not shown)), the data files at the first storage device
235 may have undergone changes during this transit period. For
example, one or more existing data files may have been modified
(e.g., a data write operation), deleted or augmented at the first
storage device 235. In order to ensure that an up-to-date
replication of the data files is copied to the destination storage
device, particularly in light of such changes, a synchronization of
data between the data files residing on both the first storage
device 235 and the destination storage device may be required.
[0095] At step 708, record identifiers such as the USN numbers
associated with each data record logged within the change journal
241 are compared with the record identifiers associated with data
loaded onto the second destination storage device. This process may
be performed, as during the time period between the first storage
device 235 data files and the record identifiers being copied to
the first destination medium/media and being transferred to the
second destination storage device, the data files at the first
storage device 235 may have undergone changes (e.g., modify, write,
delete etc.). Based on these changes to the data files at the first
storage device 235, additional data record entries (e.g., the
change journal entries) may have been created in change journal
241.
[0096] At step 710, the process determines whether data files at
the first storage device 235 have changed compared to their copies
stored at the destination storage device. As previously described
(step 708), this is achieved by comparing the record identifiers
(e.g., USN numbers) associated with each data record logged within
the change journal 241 with the record identifiers associated with
data loaded onto the second destination storage device. For
example, if the USN numbers are the same, at step 712 it may be
determined that no synchronization of data is required as the data
has not changed. Thus, there is an indication that the data files
at the first storage device 235 have not changed since being copied
to the second destination storage device. However, for example, if
at step 710 it is determined that the USN numbers associated with
each data record logged within the change journal 241 are not the
same as the USN numbers loaded onto the second destination storage
device, the data files associated with the USN numbers that were
not loaded onto the second destination storage device may be sent
via a communication pathway from the first storage device 235 to
the second destination storage device. Thus, the data files
associated with the first storage device 235 (source location) are
synchronized with the data files at second destination storage
device (target location).
[0097] Systems and modules described herein may comprise software,
firmware, hardware, or any combination(s) of software, firmware, or
hardware suitable for the purposes described herein. Software and
other modules may reside on servers, workstations, personal
computers, computerized tablets, PDAs, and other devices suitable
for the purposes described herein. Software and other modules may
be accessible via local memory, via a network, via a browser or
other application in an ASP context or via other means suitable for
the purposes described herein. Data structures described herein may
comprise computer files, variables, programming arrays, programming
structures, or any electronic information storage schemes or
methods, or any combinations thereof, suitable for the purposes
described herein. User interface elements described herein may
comprise elements from graphical user interfaces, command line
interfaces, and other interfaces suitable for the purposes
described herein. Screenshots presented and described herein can be
displayed differently as known in the art to input, access, change,
manipulate, modify, alter, and work with information.
[0098] While the invention has been described and illustrated in
connection with preferred embodiments, many variations and
modifications as will be evident to those skilled in this art may
be made without departing from the spirit and scope of the
invention, and the invention is thus not to be limited to the
precise details of methodology or construction set forth above as
such variations and modification are intended to be included within
the scope of the invention.
* * * * *