U.S. patent application number 14/673683 was filed with the patent office on 2016-10-06 for database encryption to provide write protection.
The applicant listed for this patent is Airbnb, Inc.. Invention is credited to Nelson Aurel Gauthier, Kevin Nguyen, Ismail Cem Paya.
Application Number | 20160292427 14/673683 |
Document ID | / |
Family ID | 57015955 |
Filed Date | 2016-10-06 |
United States Patent
Application |
20160292427 |
Kind Code |
A1 |
Paya; Ismail Cem ; et
al. |
October 6, 2016 |
Database Encryption to Provide Write Protection
Abstract
An online computer system including a database uses an encrypted
table that allows for write protection its contents. Middleware
logic operating on the system acts as an interface for access to
the database, so that any business logic on the system accesses the
database through simple procedural calls to the middleware rather
than directly to the database itself. The middleware logic
abstracts logic that helps implement write protection with the
encrypted table. Data to be encrypted that has been traditionally
written to other tables is migrated to the encrypted table, where
the data encrypted using an authenticated encryption with
additional data (AEAD) algorithm. To implement AEAD, the original
table, column, and primary key indicating where the data would have
otherwise been stored are together used as additional authenticated
data (AAD). This tuple of information is also stored in the
encrypted table.
Inventors: |
Paya; Ismail Cem; (San
Francisco, CA) ; Gauthier; Nelson Aurel; (Los
Angeles, CA) ; Nguyen; Kevin; (San Francisco,
CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Airbnb, Inc. |
San Francisco |
CA |
US |
|
|
Family ID: |
57015955 |
Appl. No.: |
14/673683 |
Filed: |
March 30, 2015 |
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06F 21/6227 20130101;
G06F 2221/2107 20130101; G06F 21/6218 20130101; H04L 63/061
20130101; G06F 21/602 20130101; G06F 16/2282 20190101 |
International
Class: |
G06F 21/60 20060101
G06F021/60; G06F 21/62 20060101 G06F021/62; G06F 17/30 20060101
G06F017/30 |
Claims
1. A computer implemented method comprising: receiving a write
request to a database, the write request specifying an original
table (OT) name of one of a plurality of tables in the database, a
column name of one of a plurality of columns in the OT, and a data
entry to be written; accessing an OT primary key where the data
entry is to be written; writing the data entry to a first location
in the OT corresponding to the column name and the OT primary key;
preparing an additional authenticated data (AAD) comprising the OT
name, the column name, and the OT primary key; obtaining a cipher
text based on the data entry and the AAD; and writing the cipher
text to a second location in a cipher text column of an encrypted
table (ET) different than the OT, the second location being in a
row of the ET where the row also stores the OT name, the column
name, and the OT primary key.
2. The computer implemented method of claim 1, wherein the write
request further comprises the OT primary key, and accessing the
primary key of the original table comprises: extracting the OT
primary key from the write request.
3. The computer implemented method of claim 1, wherein accessing
the primary key of the original table comprises: initiating a
transaction state with the database creating a new primary key in
the original table and preventing the new primary key from being
used outside of the transaction; responsive to the data entry being
written to the first location and the cipher text being written to
the second location, exiting the transaction state, thereby
allowing free use of the new primary key.
4. The computer implemented method of claim 1, wherein obtaining
the cipher text comprises: communicating the data entry, the AAD,
and an identification of a cryptographic key to a cryptography
service; and responsive to the communication, receiving a cipher
text from the cryptography service.
5. The computer implemented method of claim 1, wherein the AAD
further comprises additional information stored in the database but
not in the encrypted table.
6. The computer implemented method of claim 1, further comprising:
accessing the row of the ET corresponding to the OT primary key;
creating a second AAD based on at least some of the contents of the
row other than the cipher text; communicating the cipher text and
the second AAD to a cryptography service; responsive to the
cryptography service returning the data entry, validating the
encryption as successful; and responsive to the cryptography
service returning a decryption error, reporting that the encryption
was unsuccessful or that a write attack has occurred.
7. The computer implemented method of claim 1, further comprising:
deleting the data entry from the original table.
8. The computer implemented method of claim 1, wherein writing the
cipher text to the second location further comprises: writing the
original table name to an original table column in the row of the
ET; writing the column name to a column column in the row of the
ET; and writing the OT primary key to an OT primary key column in
the row of the ET.
9. The computer implemented method of claim 1, wherein writing the
cipher text to the second location further comprises: writing to a
key revision column at least one of an identifier for the
cryptographic key and an identifier of a generation of a
cryptographic key.
10. A computer implemented method comprising: receiving a write
request to a database, the write request including an original
table name of one of a plurality of tables in the database, a
column name of one of a plurality of columns in the original table,
and a data entry to be written; accessing an original table (OT)
primary key; preparing an additional authenticated data (AAD)
comprising the OT name, the column name, and the OT primary key;
obtaining a cipher text based on the data entry and the AAD; and
writing the cipher text to a location in a cipher text column of an
encrypted table (ET) different than the OT, the second location
being in a row of the ET where the row also stores the OT name, the
column name, and the OT primary key, and wherein the data entry is
not written to the OT.
11. A computer implemented method comprising: receiving a read
request to a database, the read request including an original table
(OT) name of one of a plurality of tables in the database, a column
name of one of a plurality of columns in the OT; accessing an OT
primary key where the data entry is to be read from; reading a data
entry from a first location in the OT corresponding to the column
name and the OT primary key; reading a cipher text from a first
location in a cipher text column of an encrypted table (ET)
different than the OT, the second location being in a row of the ET
where the row also stores the OT name, the column name, and the OT
primary key; preparing, as read from the ET, an additional
authenticated data (AAD) comprising the OT name, the column name,
and the OT primary key; sending a decryption request to a
cryptographic service, the request comprising the cipher text and
the AAD; responsive to receiving a response that the decryption was
successful, receiving a plaintext version of the cipher text and
validating the encryption as successful if the plaintext version of
the cipher text matches the data entry accessed from the OT.
Description
BACKGROUND
[0001] This description generally relates to database cryptography,
and more particularly to database write protection.
[0002] Online computer databases are vulnerable to a variety of
different types of attacks by third parties. Often, the goal of
these attacks is to be able to obtain read access, and therefore
access to the substantive contents of the database. To prevent such
breaches, online systems have grown more robust over time. One tool
in breach prevention is using one or more cryptographic primitives
to securely store passwords that permit read access. Cryptographic
primitives include using salts, one way functions (e.g., hash
functions), cryptographic operations (e.g., encryption,
decryption), symmetric cryptography schemes, and asymmetric (or
public key) cryptography schemes. Another major tool in breach
prevention is using a separate cryptography service to perform
various tasks associated with database access, including applying
cryptographic primitives as mentioned above. For example, one type
of cryptography service involves using a physically separate
computing device, one example of which is referred to as a hardware
security module (HSM), that is specially designed to securely store
cryptographic keys and perform cryptographic processing.
[0003] For many online databases, merely having the database
breached and read by an intruder is a worst case scenario that
merits most if not all of the attention of the online system's
security staff. As a result, many online systems do not implement
database write protection on top of read protection because all the
harm has been done once the database has been read.
[0004] For some online systems, however, write protection is at
least as important as read protection, and thus merits attention.
For example, for an online payments system, breaches that write
data to a database can have catastrophic financial implications of
their own. FIGS. 2A-2C contains several examples of database write
attacks that can cause problems for an online system.
[0005] FIG. 2A illustrates an example bank account table containing
bank account numbers for a number of users in an online payments
system that provides payouts to users. While a read attack that
obtains user bank account numbers would be problematic on its own,
also problematic would be a write attack that replaces the bank
account number of a user with the bank account number of an
attacker, for example. As an illustration, if Alice's account
number abc123 were overwritten with the attacker's account number
def456, any payments that were intended to go to Alice would
instead go to the attacker. Such a write attack is dangerous not
only due to losses by Alice and the online system, but also because
no read attack is needed to cause harm. Alternatively, Alice's
account number could be written to another row in the table
accessible by the attacker (e.g., Bob's row), who could then read
the account number without needing to perform a read attack.
[0006] FIG. 2B illustrates a similar example bank account table and
an example credit card table. In this example, rather than
overwriting or copying data within a single table, a write attack
could write data between tables. Some of these attacks may result
in erroneous or non-functional behavior in the online system, for
example account numbers and credit card numbers may not be able to
be written between tables based on the sizes and data types of
those columns. These attacks may also result in errors in the
business logic implemented on top of the database by the online
system, depending upon how these different columns of data are used
once read out of the database. However, some of these attacks may
result in data that was previously protected by rigorous business
logic to become openly visible to the attacker. For example, a
column data that was previously protected by a password system may
instead end up being visible through an unprotected portion of the
web interface provided by the online system. Here, write attacks
provide a way to circumvent read protection or other security
mechanisms implemented by the online system.
[0007] FIG. 2C illustrates another example bank account table. In
this example, rather than overwriting or copying elsewhere the data
item of interest, a write attack may instead change the data in the
surrounding cells in the same row of the table. For example, a
write attack may change the bank routing number associated with a
user's account, as well as the contact email address for that user.
If the attacker controls the account number at the bank
corresponding to the new routing number, then any payments made to
that user would be redirected to the attacker. Further, emails
directed to the account holder may be redirected to an email
account associated with the attacker, thus preventing the real
account holder from receiving valuable information from the online
system about the breach.
SUMMARY
[0008] An online computer system including a database uses an
encrypted table that allows for both read and write protection of
the contents of the database. Middleware software logic operating
on the online computer system acts as an interface for access to
the database, so that any business logic on the online system that
wants to access the database does it through simple procedural
calls to the middleware rather than directly to the database
itself. The middleware logic abstracts database protection logic
that helps implement read/write protection.
[0009] To protect against read and write attacks, data to be
encrypted that has been traditionally written to other tables is
migrated on write (or on read) to an encrypted table (ET), where
the data is stored as cipher text. An authenticated encryption with
additional data (AEAD) algorithm is used to encrypt or validate the
encryption of cipher text. To implement AEAD, the original table
(OT) name, column name, and OT primary key (pk) where the data
would have been stored prior to the existence of the ET are
together used as additional authenticated data (AAD) as a parameter
passed in parallel with the OT data or cipher text during
encryption or decryption, respectively. The use of AEAD encryption,
along with a cryptography service to perform cryptographic
operations, provides read protection. If a write attack occurs that
changes any element of the AAD, decryption will fail and the
failure can be detected and reported accordingly. The OT, OT
column, and OT pk are also stored in the row of the ET along with
the corresponding cipher text. The AAD may also be augmented with
additional information stored elsewhere in the database, as known
to the middleware. Once all of the data from a column in an OT has
been encrypted in the ET and the encryption has been sufficiently
validated for accuracy, the column whose data is now written to the
ET can be deleted from the OT, thus preventing a write attack from
accessing or modifying the encrypted data.
[0010] Various processes are also described, including "safe"
(dual)-write and safe read processes for ensuring that data is
accessible during the migration of data in a column from the OT to
the ET, during migration between generations of cryptographic keys,
as well as longer term processes that describe both the general
data migration process into the ET, as well as the general key
revision process between generations of cryptographic keys.
[0011] The features and advantages described in this summary and
the following detailed description are not all-inclusive. Many
additional features and advantages will be apparent to one of
ordinary skill in the art in view of the drawings, specification,
and claims.
BRIEF DESCRIPTION OF THE DRAWINGS
[0012] FIG. 1 is a block diagram of a computing environment for an
online system that includes a database and which implements read
and write protection for the content of the database, according to
one embodiment.
[0013] FIGS. 2A-2C illustrates example tables showing example write
attacks that can affect a database that does not have write
protection, according to one embodiment.
[0014] FIGS. 3A-3D illustrate a pair of example original tables
(OTs) and a pair of example encryption tables (ETs) that highlight
aspects of the write protection offered by the online system,
according to one embodiment.
[0015] FIG. 4 is a flow diagram of a migration process for the
online system, according to one embodiment.
[0016] FIG. 5 is a flow diagram of a protected (dual) write
operation for use by the online system during migration, according
to one embodiment.
[0017] FIG. 6 is a flow diagram of a protected read operation for
use by the online system during migration, according to one
embodiment.
[0018] FIG. 7 is a flow diagram of a non-protected (non-dual) write
operation for use by the online system after migration, according
to one embodiment.
[0019] FIG. 8 is a flow diagram of a key revision process for the
online system, according to one embodiment.
[0020] FIG. 9 is a flow diagram of a protected write during a key
revision operation for the online system, according to one
embodiment.
[0021] The figures depict various embodiments of the present
invention for purposes of illustration only. One skilled in the art
will readily recognize from the following discussion that
alternative embodiments of the structures and methods illustrated
herein may be employed without departing from the principles of the
invention described herein
DETAILED DESCRIPTION
I. Online System Overview
[0022] FIG. 1 is a block diagram of a computing environment for an
online system 130 that includes a database and which implements
read and write protection for the content of the database,
according to one embodiment. The online system 130 may be any kind
of system which stores data that is to be protected from potential
attackers. Examples include payments systems that pay users or
exchange payments between users, accounts systems that maintain
private information about users, booking systems that maintain
private itinerary or accommodation information about users,
etc.
[0023] Users 101 use client devices 110 to interact with the online
system 130. A client device 110 can be any device that is or
incorporates a computer such as a personal computer (PC), a desktop
computer, a laptop computer, a notebook, a smartphone, or the like.
A computer is a device having one or more general or special
purpose processors, memory, storage, and networking components
(either wired or wireless). The device executes an operating
system, for example, a Microsoft Windows-compatible operating
system (OS), Apple OS X or iOS, a Linux distribution, or Google's
Android OS. In some embodiments, the client device 110 may use a
web browser 115, such as Microsoft Internet Explorer, Mozilla
Firefox, Google Chrome, Apple Safari and/or Opera, as an interface
to interact with the online system. In other embodiments, the
client device 110 can execute a dedicated application for accessing
the online system 130.
[0024] The network 120 represents the communication pathways
between client devices 110 (e.g., users) and the online system. In
one embodiment, the network is the Internet. The network can also
utilize dedicated or private communication links (e.g. wide area
network (WAN), local area network (LAN)) that are not necessarily
part of the Internet. The network uses standard communications
technologies and/or protocols (e.g., TCP-IP).
[0025] The online system 130 includes a web server 140, a feature
software application, database access middleware 160, and a
database 170. The web server 140 communicates with the client
devices 110 over the network, for example by providing web page
content for display in a browser 115. The web server 140 further
provides an interface allowing the user to control their
interaction with the web server 140, including both sending
information from the client device 110 to the web server 140, and
receiving data at the client device 140 from the web server.
[0026] For example, if the online system 140 is a payment system
for sending and receiving payments between users, the web server
140 may provide graphical user interface allowing the user to
create and manage their account, add bank routing and bank account
numbers to send and receive money from, add credit card numbers to
send and receive payments from, add a contact email address where
they can be contacted regarding payments processed by the online
system 140, view payment histories and dollar amounts of payments,
etc.
[0027] The web server 140 processes and communicates data to a
feature software application 150 (referred to as application 150)
that executes the features provided by the online system 140.
Application 150 carries out tasks based on information (e.g.,
requests) received externally and based on software code that
determines what actions to take in any given situation. Generally,
application 150 includes the business logic of the online system.
Continuing with the payments system example above, application 150
may include software logic for processing payments by instructing
money to be transferred between accounts, provide notice via email
when payments have been made, and so on.
[0028] During the normal course of operation, the online system 130
will create, read, update, and delete (perform CRUD operations on)
data in a database 170. To provide a clean software abstraction for
performing these operations, a separate database access middleware
160 (which is also software) controls database access. Any other
portion of the system that wants to perform a CRUD operation
(generally functions within application 150) with respect to the
database 170 instead makes the appropriate function call to
middleware 160 rather than performing an operation directly with
the database 170. Middleware 160 makes available to application 150
or any other piece of software seeking to access database 170 a
relatively simple set of function calls for performing CRUD
operations. These include read and write requests to the database
170. This assists the online system in reducing the complexity of
software code in application 150 as any operation the application
150 wants to perform on data stored in the database is encapsulated
by one of the functions made available by the middleware 160.
[0029] Through this abstraction, the database access middleware 160
is also able to control encryption of data in the database in order
to provide read and write protection. Because middleware 160 is
logically abstracted and separated from the 150, the details of how
the encryption processes are managed does not need to be known by
application 150, further reducing the complexity 150. In addition
to providing function calls for CRUD operations, middleware 160
also provides function calls and parameters allowing the
application 150 or an external online system administrator or
software developer to specify what elements of data in database 170
(e.g., particular columns of particular tables) are to be
encrypted.
[0030] The online system 130 also communicates with a cryptography
service 180, which is also referred to as an encryption/decryption
oracle. The cryptography service 180 safeguards cryptographic keys
(e.g., private keys in a public key cryptography scheme) and
performs at least some (though not necessarily all) of the
cryptographic primitives (e.g., encryptions, decryptions, hashing,
truncation, and masking) involved in safeguarding data against read
attacks. In one embodiment, the cryptography service 180 is a HSM
that includes a crypto processor. A crypto processor is a computing
device such as a dedicated computer, system-on-a-chip, or
microprocessor that can be attached to another computer or server
as, for example, a plug-in card or an external device. In one
embodiment, the cryptography service 180 can be accessed (e.g.,
rented) through paid systems such as AMAZON WEB SERVICES (AWS)
CLOUDHSM. The cryptography service 180 can use a variety of
cryptography schemes including symmetric key cryptography schemes
and public key cryptography schemes.
[0031] In between the cryptography service 180 and the online
system 130 may be one or more intermediate servers. These
intermediate servers may have varying levels of security. In some
implementations, some cryptoprocessing functions may be performed
in addition to the final encryption or decryption performed by the
cryptography service 180, in attempt to guard against attacks that
try to take advantage of the relative insecurity of intermediate
servers. For example the middleware 160 may use masks on
cryptographic keys or data to be encrypted or decrypted prior to
passing data to the cryptography service 180. These types of
security help prevent read attacks that either attempt to access
data in transit between the online system 130 and 180, or attempt
to emulate either the online system 130 or cryptography service 180
to cause data to be transmitted when it should not be. Further
detail regarding securing the online system 130 and cryptography
service 180 against read attacks can be found in co-pending U.S.
patent application Ser. No. 14/515,499, which is incorporate d by
reference herein in its entirety. The computing systems and
techniques described in that application are compatible for use
alongside the computing systems and techniques described in this
application.
[0032] Database 170 may be a relational database or a
non-relational database that uses structured logic (for example in
the middleware 160) to enforce proper database behavior. The
database 170 may be accessed using a structured query language
(SQL). In the case of a relational database, the database will
generally consist of a number of tables, each having a number of
columns, where the rows of each table are indexed according to a
primary key that is unique for each row.
[0033] Although the components of the online system 130 and
cryptography service 180 have been described as single elements, in
practice there may be many of each element in order to implement
the online system 130. For example, the online system 130 may
consist of a number of server-class computers existing in several
different geographic locations, such as data centers. The web
server 140 and each software module such as application 150 and
middleware 160 may be replicated across each data center, with the
different data centers in communication to provide additional
redundancy and data security. Database 170 may be similarly
replicated. Further, there may be more than one intermediate
servers between the online system 130 and the cryptography service
180, and the cryptography service's 180 load may be split among
several computing devices.
II. Original Table and Encrypted Table Database Schemas
[0034] FIGS. 3A-3D illustrates a pair of example original tables
and a pair of example encryption tables that highlight aspects of
the write protection offered by the online system, according to one
embodiment. To implement write protection at least one new table,
herein referred to as the encrypted tables (ET) is added to the
database 170. Generally, the ET stores data that would have
otherwise been stored in another existing table, generally referred
to as the original table (OT), instead. The ET stores data in a
cipher text format.
[0035] The ET also stores sufficient information to fully specify
the original location of where the data would have been stored in
the OT, which includes the table name where the data would have
been stored, the column name where the data would have been stored,
and primary key (pk) (or row) of the OT where the data would have
been stored. Each row in the ET corresponds to a row/column
combination in one of the OTs of the database 170. A single ET may
store data from multiple OTs easily, as the table/column/OT pk
information (herein referred to as the table/column/OT pk tuple)
stored in each ET row allows data from different OTs to be
distinguished from each other.
[0036] In some instances, no OT will exist for new data that is to
be encrypted and stored in the ET right from the start. In this
instance, the OT name and OT column name for that data are chosen
by the system administrator based on whatever database schema is
desired. However, no actual OT corresponding to the chosen OT name
needs to actually exist, these names simply serve as a mechanism by
which the middleware 160 can identify this data versus other data
from other OTs. The middleware 160 can also keep track of the OT pk
as new data is added for this chosen OT/OT column name combination,
and increment it as new rows are added. Additionally, the
middleware 160 uses the OT name and OT column name chosen for this
data in CRUD operations on behalf of applications 150.
[0037] One schema for the ET specifies that the ET includes a
primary key (pk) column, a table (or table name) column, a column
(or column name) column, a column for the pk from the OT from which
the data for that row in the ET originated (also referred to as the
OT pk column), and a cipher text column. The ET may also have
additional columns.
[0038] The primary key column of the ET is simply the row of the
ET. To avoid confusion between the pk of the ET and the pk of the
OT, herein the pk of the ET is referred to as the row of the ET,
and the pk of the OT is referred to as the OT pk. Note also that
the term "primary key" refers to the row of a table, whereas the
previously introduced term "cryptographic key" refers to a key used
to perform cryptographic primitives, such as the
encryption/decryption of data into/out of cipher text.
[0039] The table column of the ET contains the name of the OT from
which the data in that row of the ET originated. Likewise, the
column column of the ET contains the name of the column in the OT
from which the data in that row of the ET originated. Likewise, the
OT pk column is the row in the OT from which the data in that row
of the ET originated.
[0040] The cipher text column stores the encrypted data from the
OT. Generally, an OT's schema will specify, for any given column of
data, the maximum bit size, format, and/or type of data that may be
stored in that column. The cipher text column does likewise,
however the maximum size, format, and/or type of the encrypted
version of the data that is stored in the cipher text column will
almost always differ from how the data was originally specified to
be stored in the OT. For example, an OT data column may specify
that that an account number column is to store data as an 32-bit
unsigned integer. However, in encrypted form in the ET, the cipher
text column may specify that data is to be stored as a 128 bit
string instead. Further, as the ET may store data from multiple
OTs, the change of size/format/type between the OT and the ET
standardizes data storage so that all data may be encrypted with
the same process, and so that all data can be stored in the cipher
text column alone.
[0041] In order to secure the ET against write attacks, the
middleware 160 is configured to make use of the data in the other
rows of the ET (e.g., the pk column, the table column, and the
column column, as well as additional columns in some
implementations) to execute encryption/decryption using an
authenticated encryption with additional data (AEAD) algorithm
carried out in conjunction with the cryptography service 180. In
one implementation, the additional authenticated data (AAD) for
implementing the AEAD algorithm includes at least the following
tuple of information from the ET row of the OT data to be
encrypted: {table name, column name, OT pk}. The AAD is not part of
the cipher text, but is instead used to influence the calculation
of the cipher text during encryption and decryption.
[0042] To encrypt OT data, the middleware 160 passes OT data and
the AAD to the cryptography service 180, which returns a cipher
text that the middleware 160 writes in the cipher text column of
the ET. When the middleware 160 requests decryption, the middleware
160 passes the cipher text and the AAD to the cryptography service
180. Assuming neither the cipher text nor the AAD have been
modified by a write attack, the cryptography service 180 will
return the OT data in plaintext. However, if a write attack has
altered any of the three items of information in the AAD or the
cipher text, the decryption will fail.
[0043] FIGS. 3A-3D illustrate example OTs and ETs. FIGS. 3A and 3B
illustrate example OTs, with an example accounts table having
routing number, account number, email, and User ID columns and
corresponding data, and an example payments table having payment
and User ID columns. Example encryption table 1 (ET1) illustrates
several rows of the OTs having been written to ET1. For example, in
row 2 of ET1, the table name "Payments", column name "Payment" and
primary key of the Payments table "3" specify the originating
location of the OT data. The cipher text for row 2 of ET1 stores an
encrypted version of the payment, 98.02, and the AAD may, for
example, be the string "PaymentsPayment3".
[0044] The AAD may also include additional information beyond just
the tuple. The additional information may be one or more items of
data from any OT in the database 170 that is immutable over the
timescale during which the OT data to be encrypted will be read and
written. Despite being added to the AAD, this additional
information is not added as a column in the ET. This prevents write
attacks that affect the ET from also changing the additional
information, as unless the attacker has knowledge of which other
data the middleware 160 is using as additional information in the
AAD, they will not know what data to change to cause decryption to
be successful during a write attack.
[0045] Using FIGS. 3A-3C as an example, the database 170 may
include an OT having a column for a user's email address (e.g., the
Accounts table), that may be different from the OT having the data
to be encrypted (e.g., the Payments table). Further, in this
example the middleware 160 is programmed to know that the email
address column from the Accounts table is to be used as additional
information in the AAD. In some instances other columns, such as
the User ID columns in both the Accounts and Payments table, may be
used to identify which row to obtain the additional information for
the AAD from. When the ET row is created, the middleware 160
inserts the tuple information into the table name, column name, and
OT pk columns of the ET for that row. To encrypt the OT data in the
ET row, the middleware 160 accesses the tuple and OT data from the
ET, and separately access the additional information (e.g., the
email address) from the OT (e.g., the Accounts table), combines the
tuple and the additional information to create the AAD (e.g.,
"PaymentsPayment3bob@isp.net"), and passes the AAD and the OT data
to the cryptography service 180 for encryption. To perform the
combination, a number of operations may be performed such as
concatenation or by prefixing each item of information with its
length prior to concatenation to avoid ambiguity regarding which
portion of the AAD is related to each contributing item of
information. When the cipher text is received back from the
cryptography service 180, the data is stored in the cipher text
column of the ET. Thus, the data in encrypted without storing the
additional information in the AAD in the ET.
[0046] The type of encryption used by the online system 130 and
cryptography service 180 to generate the cipher text may vary. In
one embodiment, symmetric block ciphers such as the Advanced
Encryption Standard (AES) are used to create the AEAD algorithm
that is used by the cryptography service 180 to encrypt OT data
using the AAD and cryptographic key. One example of a fully
specified AEAD algorithm is AES-GCM (Galois counter-mode).
[0047] In some implementations, the ET may also include a key
revision (key rev) column. Encrypted table 2 in FIG. 3D illustrates
such a table. The key rev column specifies what generation of
cryptographic key was used to encrypt the data in that row of the
ET. For example, in FIG. 3D, "v2" indicates a second generation
cryptographic key, and "v3" indicates a third generation
cryptographic key newer than the second generation cryptographic
key. The key rev column generally does not store the cryptographic
key itself In this way, the middleware 160 and cryptography service
180 knows which generation of cryptographic key needs to be used to
encrypt or decrypt the data for performing a write or read
operation on the column, and also so that the middleware 160 and
cryptography service 180 can migrate the row to a new generation of
cryptographic key, as circumstances dictate. It is possible that
the ET may store the same piece of data in multiple rows, where
each such row stores the data with a different generation of key.
This may be used, for example, during verification of the new
cryptographic key, so that data encrypted with the old
cryptographic key (old key) is not lost by accident during the
migration to the new cryptographic key (new key). This is further
described below with respect to FIGS. 8 and 9.
III. Migration Process
[0048] FIG. 4 is a flow diagram of a data migration process for the
online system, according to one embodiment. The data migration
process (or simply migration or migration process) refer s to the
processes carried by the application 150, middleware 160, and/or in
some cases administrator to transition a column of OT data that is
either stored in plaintext 405 or stored using a legacy encryption
405 to instead be stored in ET, thereby providing write protection
to the column's data. Migration of data can be implemented in the
online system 130 without altering the schema of an existing
database 170.
[0049] To begin encrypting data stored in OTs, an administrator
will modify application 150 to specify that middleware 160 should
begin encrypting the data of a column of an OT (or more than one
column of an OT or more than one OT at a time). The middleware 160
will flag that OT column as having to be stored in the ET instead.
This initiates a process 410 in the middleware 160, whereby each
time data is to be written to the selected column of the selected
OT, the data is written to new (and then subsequently existing)
rows in the ET.
[0050] In order to verify that the encryption is working properly
(e.g., to guard against cryptography system failure), and in order
to ensure smooth operation of the online system 130 for the
application 150 and users, the activation of the write migration
process does not immediately deprecate the OT column where the data
used to be written to. Instead, during the initial stage of
migration, a "safe" (dual) write process 415 is used whereby not
only is data written to the ET as described above, but data is also
written to the OT column as well for some period of until 1) all
rows of the OT column have been written to the ET, and 2) the
middleware 160 or an administrator is sufficiently confident that
the encryption is working properly. The safe write process is
described with respect to FIG. 5 below. The process for performing
a read while data is initially being migrated (i.e., while safe
writes are being performed) is described with respect to FIG. 6
below, and is herein referred to as a "safe read" for clarity.
[0051] Once these conditions are met, safe write can be deactivated
420 and going forward the middleware 160 will write to only the ET
and not the OT column. The non-safe write process (i.e.,
non-migration or normal write process) is described with respect to
FIG. 7 below. At this point, the OT column data can be deleted 425.
Normal reads may also be performed as well, which are also
discussed below. In an instance where data is encrypted from the
start and no corresponding OT or OT column exists or previously
existed prior to the data being added, safe write and safe read are
not used as there is no fall back OT data to rely on. For this kind
of data, the normal write process is used to write data to the ET
and the normal read process is used to read data from the ET.
IV. Safe Write
[0052] FIG. 5 is a flow diagram of a safe write operation for use
by the online system during migration, according to one embodiment.
The middleware 160 receives 505 a request to write to the database
170. The write request either includes or causes a database query
to determine the OT name, the column in the OT, and the data to be
written. This request may be a create operation to create a new row
in the OT, or it may be an update operation to update an existing
row in the OT. In the case of an update write request 505, the
write request will also include or determine the pk in the OT for
the row corresponding to that data.
[0053] If 510 the OT pk exists in the OT, i.e. if it is known or
can be derived from the request, then the middleware 160 carries
out a first set of operations, and if the OT pk is not known such
that a new row needs to be created in the OT to store the data,
then the middleware 160 carries out a second set of operations.
[0054] Assuming the OT pk is known or derived, a safe write is
performed including a write operation to the OT, as well as a write
operation to the ET as introduced above. To perform both write
operations, a transaction state is initiated with the database 170
that is exited when the middleware can confirm that both the ET and
OT writes were performed successfully. The write operation to the
OT 515 may, for example, be a simple SQL operation to
overwrite/update the provided data to the appropriate OT/column/pk.
If the data in the OT is encrypted with a legacy encryption, the
middleware 160 or the cryptography service 180 performs the legacy
encryption before writing the data to the OT.
[0055] For the write operation to the ET, the OT name, column name,
OT pk tuple, and any additional information are combined to form
the AAD. The middleware 160 passes to the cryptography service 180
the OT data to be encrypted, the AAD and an identification of which
cryptographic key is to be used for encryption. The cryptography
service 180 returns the cipher text to the middleware 160 which in
turn writes 525 a new row to the ET. Specifically, the middleware
160 writes each column of the tuple in the appropriate column of
the ET, and writes the cipher text to the cipher text column of the
ET. Like the write to the OT, this may be a simple SQL operation.
Any subsequent safe writes for that ET row (i.e., that tuple) are
similar, except the same ET row can be reused and overwritten,
rather newly adding a row to the ET for each update operation.
[0056] Alternatively, the OT pk may not exist. This will most
commonly occur for new data that does not have a corresponding row
in the OT. If the OT pk is not known, the transaction state
introduced above also enforces atomicity in the creation and use of
rows of the OT. The middleware 160 reserves 550 a new row/pk in the
OT and preventing (or "locking") the OT and/or database 170 from
re-using the reserved row/pk or duplicating the row/pk and/or write
different data from another database operation to the reserved row
or another row having the same pk. In one embodiment, the
transaction state may include locking the entire OT and/or database
170 from allowing any new rows/pks to be created in the OT until
the transaction is complete and the OT and/or database 170 is
unlocked. However, in other embodiments, less onerous safeguards
may be implemented to prevent any other database operations from
affecting the cells of the reserved row/pk in the OT other than the
write request that caused the reserved row/pk to be created in the
first place.
[0057] The middleware 160 writes 515 the data to the OT as
specified by the OT/column/new (reserved) pk tuple. This is the
same step as the overwrite 515 mentioned previously, except that
data is added rather than being updated because the row is new.
Also as described above, the middleware 160 has the data encrypted
520 by the cryptography service 180 and writes 525 the data to the
ET 525. Although the reservation 550 and write 515 steps are
described here as separate steps, in practice these steps may occur
simultaneously, for example the database operation of adding a new
row may be the triggering action that causes a pk to be assigned to
the new row, which may then initiate the transaction state causing
the pk to be reserved until the write to the ET is complete.
[0058] Once the data has been written to both the new pk of the OT
and the new row of the ET, the transaction is completed such that
the reserved row/pk is no longer reserved, thereby allowing other
database operations to interact with the previously reserved
row/pk. However, in the event that the encryption fails and/or the
write 525 to the ET was not successful, having the pk/row of the OT
reserved allows the entire write process to be rolled back. This
may be accomplishing by deleting the data from the reserved pk/row
of the OT and sending an indication to the middleware 160 that the
write operation failed. The transaction state may then be exited,
thereby allowing other operations to use the previously reserved
pk/row of the OT due to the rollback.
V. Safe Reads
[0059] FIG. 6 is a flow diagram of a safe read operation for use by
the online system during migration, according to one embodiment.
The middleware 160 receives 605 a request to read from the database
170. The read request includes a query that uniquely identifies the
row, column, and table of the OT to be read. The middleware 160
determines 610 whether the data can be read from ET. To make this
determination, the middleware 160 checks 610 whether a row exists
in the ET that matches the OT/column/OT pk tuple received and/or
derived from the read request. If such a row does exist in the ET,
the middleware 160 constructs an AAD based at least on the tuple
from the corresponding row in the ET and any additional information
used by the middleware 160 and sends 615 the AAD, the cipher text
from that row of the ET, and a reference to the cryptographic key
to be used for decryption to the cryptography service 180 for
decryption.
[0060] The middleware 160 also access the data 620 from the OT
using the tuple. If the data in the OT is encrypted with a legacy
encryption, the middleware 160 or the cryptography service 180
performs decryption of the legacy encryption as well. Although FIG.
6 illustrates steps 615 and 620 as occurring one after the other,
in practice these steps may be performed in parallel, as neither
requires the outcome of the other in order to proceed.
[0061] To validate that the encryption with the ET is working
properly, the middleware 160 compares 625 the returned, decrypted
ET data from cryptography service 180 against the data from the OT.
If the encryption is working properly and no write attack has been
performed that alters the AAD or ciphertext, the decrypted ET data
will match the corresponding data from the OT. The middleware 160
can then return 630 the data to the application 150. If the
decryption failed, either something went wrong with encryption or
decryption, or a write attack affected some item of data used in
the AAD. The middleware 160 can then report 635 an error and/or
carry out other fallback instructions. In this instance, to ensure
smooth operation of the application 150, the middleware 160 returns
the OT's version of the data, because the error in the decryption
means that the decrypted ET data is not reliable.
[0062] If a row corresponding to the received OT/column/OT pk tuple
does not exist in the ET, then the middleware 160 accesses 650 the
data from the OT instead. Depending upon the implementation, the
middleware 160 may also include logic to carry out a write
operation to the ET. This helps speed the migration process, as the
ET will be added to on-read as well as on-write. In this case, the
OT data, the AAD, and an identification of the cryptographic key to
be used for encryption are sent 655 to the cryptography service
180. This step is similar to step 520, described above with respect
to safe writes. The cryptography service 180 passes back cipher
text which the middleware 160 receives and in turn writes to a new
row in the ET. This step is similar to step 525, also described
above. The middleware 160 returns 655 the data from the OT to the
application 150.
VI. Normal Writes and Reads
[0063] FIG. 7 is a flow diagram of a non-safe (i.e., normal) write
operation for use by the online system after migration, according
to one embodiment. The middleware 160 receives 705 a request to
write to the database 170. The write request is similar to the
request from step 505 described above. Similarly to step 510, if
710 the OT pk exists in the OT, i.e. if it is known or can be
derived 510, then the middleware 160 carries out a first set of
operations, and if the OT pk is not known such that a new row needs
to be created in the OT to store the data in the ET, then the
middleware 160 carries out a second set of operations.
[0064] Assuming the OT pk is known or derived, a normal write is
performed including a write operation to the ET. However, different
from a safe write, a normal write does not include a second write
to the OT. To perform the write operation, a transaction state is
initiated with the database 170 that is exited when the middleware
can confirm that the ET write was performed successfully. The write
operation to the ET is the same as steps 520 and 525 described
above. Generally, the middleware passes the cryptography service
180 the OT data, the AAD, and an identification of the
cryptographic key to be used for encryption 715, and the
cryptography service 180 returns the cipher text. The cipher text
is then written 720 to the row in the ET corresponding to the
tuple.
[0065] If the OT pk is not known (e.g., new row to be written), the
transaction begins 750 with the middleware 160 reserving a new
row/pk in the OT, and locking the OT and/or database 170 so that no
other database operations can be carried out on the cells of the
reserved row or using the reserved PK until the transaction is
complete and the row/pk is unlocked.
[0066] With the new pk for the OT now having been created, the
middleware 160 sends 715 to the cryptography service 180 the data,
the AAD, and an identification of the cryptographic key to be used
for encryption, and writes 720 the returned cipher text to the ET.
These steps are similar to steps 520 and 525, respectively. Once
the data has been written to the ET, the transaction is completed
and the reserved row/pk are unlocked 755.
[0067] Although not separately illustrated, after migration
completes and normal writes are being used to write data to the ET,
normal reads may be performed by performing steps 605, 610, 615,
630 described with respect to FIG. 6 above. Once the decrypted
cipher text from an ET row is received 615 from the ET, it can be
directly returned 630 to the requesting application 150 without
needing to perform the extra access 620 and comparison 625 steps
described above.
VII. Key Revision Process
[0068] FIG. 8 is a flow diagram of a key revision process for the
online system, according to one embodiment. Key revision is a
process whereby the online system 130 and cryptography system 180
officially discontinue use of an old cryptographic key and begins
using a new cryptographic key in order to increase security of the
system. Key revision is not instantaneous, as any data encrypted
with the old key needs to be decrypted with the old key and
re-encrypted with the new key before the old key can be said to be
no longer in use
[0069] Prior to implementation of the key revision process, it is
assumed that all of the data in the ET has been encrypted 805 using
the same, existing key to be replaced. In one embodiment, the ET
has a schema as illustrated in FIG. 3D, including a key revision
column listing the generation of cryptographic key used to encrypt
the data of that row of the ET.
[0070] The key revision process is activated 810 either in the
middleware 160, by a system administrator, or by the cryptography
service 180. The key revision process may be a periodic process
that occurs automatically, or responsive to an external event such
as an administrator action. Activation of the key revision process
causes the cryptography service 180 to begin encrypting OT data
stored in the ET using the new key rather than the old key. Once
key revision is activated, each time that data is to be encrypted
for storage in the ET, for example upon write operation and/or upon
read operation, the data is encrypted with the new key rather than
the old key.
[0071] In order to verify that encryption using the new key is
working properly, and in order to ensure smooth operation of the
online system 130 for the application 150 and users, the activation
of the key revision process does not immediately deprecate the old
key. The old key and the rows of the ET storing data encrypted with
the old key is stored until 1) all of the data in the ET has been
written to a new row in the ET and encrypted with the new key, and
2) the middleware 160 or an administrator is sufficiently confident
that the encryption is working properly. As a result, after any
decryption with the new key (e.g., on read operation), the
encryption can be verified by comparing against the decrypted
version of the data stored with the old key. Similarly to the
process for initial migration of data to the ET described above
with respect to FIG. 5 above, this process is also referred to as a
"safe write" process, and is described below with respect to FIG. 9
below.
[0072] Once these conditions are met, the safe write process can be
deactivated 820 and going forward the middleware 160 will write to
only the ET rows corresponding to the new key. At this point, the
rows in the ET corresponding to the old key can be deleted 825.
VIII. Key Revision Safe Write
[0073] FIG. 9 is a flow diagram of a safe write operation during a
key revision process for the online system, according to one
embodiment. The middleware 160 receives 905 a request to write to
the database 170. Step 905 and the contents of the write request in
step 905 are similar to step 505 and the contents of that step's
write request, described above with respect to FIG. 5. If 910 the
OT pk exists in the OT, i.e. if it is known or can be derived from
the request, then the middleware 160 carries out a first set of
operations, and if the OT pk is not known such that a new row needs
to be created in the OT to store the data, then the middleware 160
carries out a second set of operations.
[0074] As key revision only needs to be performed on items of data
that have been encrypted using an old cryptographic key, the key
revision process is generally only applicable in this situation.
New items of data can simply be written to the ET with the new
cryptographic key (not shown). For data encrypted using an old key,
once the data is identified in the ET and the pk of the OT is
known, the middleware 160 determines 915 whether the key used to
encrypt the OT data in the ET is a current-generation (i.e., new)
key for encryption, or whether the key is out of date (i.e., old
key). In an implementation where the ET has a column dedicated to
storing the key revision (key generation) used to encrypt the data,
for example as illustrated in FIG. 3D, the middleware 160 may make
this determination by performing a SQL query to read out the key
rev value from that column based on the OT/column/OT pk tuple from
the write request.
[0075] If the old (existing) key that was used to encrypt the data
is still valid, then a normal write can be performed and no
additional steps are performed. Specifically, the middleware 160
passes to the cryptography service 180 the OT data, the AAD, and an
identification that the new key (same as the existing key) is to be
used for encryption 920, and the cryptography service 180 returns
the cipher text. The middleware 160 then overwrites 925 the cipher
text in the cipher text column and row corresponding to the tuple
from the write request. These steps are similar to steps 520 and
525, respectively, as described above.
[0076] If the existing key that was used to encrypt the data is out
of date, the middleware 160 passes to the cryptography service 180
the OT data, the AAD, and an identifier signifying the new key is
to be used for a first encryption 950. In this case, the new key is
different from the existing (i.e., old) key listed in the ET for
that tuple and ET row combination. The middleware 160 also passes
to the cryptography service 180 another identifier signifying that
the old key is also to be used for a second encryption 950
operation.
[0077] The cryptography service 180 returns both the data encrypted
with the new key as well as the data encrypted with the old key to
the middleware 160. The middleware 160 performs two separate
writes. The new data encrypted with the old key overwrites 955 the
cipher text column of the existing row in the ET where older data
was previously written. The new data encrypted with the new key is
also written 960 to the cipher text column of a new row in the ET,
along with the tuple and the generation identifier of the new
key.
[0078] Writing two different versions of the data allows the
middleware 160 to validate whether or not the encryption with the
new key is working properly. Either upon each read request or in
batch on an event-driven basis, the middleware 160 performs a
validation (not shown) to determine whether the encryption of the
data with the new key is working properly. This may include
decrypting both rows, one containing the data encrypted with the
new key and one containing the data encrypted with the old key, and
comparing the decrypted results. Once all of the rows of the OT
have been encrypted with the new key and a sufficient number of
validations indicate that the encryption is working, the rows of
the ET encrypting the data with the old key can be deleted from the
ET.
IX. Advantages
[0079] The encryption system described herein has a number of
advantages. First, database schemas can be exceptionally expensive
to changeover. For a large scale system that has anywhere from
thousands to billions of users, and based upon the amount of data
stored for each user, even minor changes to a database schema might
involve storage capacity issues, legacy software based on the
schema, developer time to work out software bugs, etc. Using an ET
and the encryption processes described herein avoids these costs
and complexities as it does not require an adjustment to existing
database schema. The ET operates in conjunction with existing
databases, as a separate repository during migration, and
ultimately as the only repository over time.
[0080] Encryption can be added incrementally, at the convenience of
the developer. Activation can take place on a column by column
basis allowing encryption to be scaled gradually in place without
the need to interrupt regular operation of the system or the
database.
[0081] During migration of a column to be encrypted to the ET, safe
write allows the administrator to ensure that the encryption is
working before write protection is disabled. If the encryption is
not working and data is failing to be decrypted correctly, with
safe write the middleware can fall back to the OT to obtain the
data, to ensure seamless operation from the point of view of the
application 150 and the user 101. While safe write activation of
the ET for a particular column of data in a table initially results
in data duplication for that column, once migration is complete
duplication can be stopped, and the original data can be deleted
from the OT. Thus, the ET results in only a small net increase in
the size of the database.
[0082] The encryption system can also be added on top of an
existing encryption system where some data is encrypted with a
legacy encryption in an OT. The write protection system is equally
able to handle unencrypted data as well already encrypted data.
Once migration is complete, the both the originally encrypted data
and the unencrypted data can be deleted, thus phasing out legacy
encryption systems, allowing for a simpler encryption system that
is centrally managed by the middleware 160.
[0083] The migration system can be used not only to migrate
unencrypted or legacy encrypted from an OT into the ET, but it can
also be used to migrate between generations of cryptographic keys.
Similarly to the initial migration process, this can occur on a
column by column basis or on an individual write by write basis,
allowing key migration to be a seamless, low overhead process that
occurs as gradually or as quickly as the administrator desires.
Also similarly to the initial migration, safe write may be
temporarily used to store the data encrypted with the old key to
prevent data loss in the event that a new key causes encryption
issues.
X. Additional Considerations
[0084] Although the ET has been introduced in terms of language
associated with SQL databases (e.g., with reference to primary keys
and rows), the ET and OTs may be structured to operate in other
types of database systems that use different technology and
terminology (e.g., key/value as opposed to primary key/row).
[0085] The figures use like reference numerals to identify like
elements. A letter after a reference numeral, such as "110A,"
indicates that the text refers specifically to the element having
that particular reference numeral. A reference numeral in the text
without a following letter, such as "110," refers to any or all of
the elements in the figures bearing that reference numeral (e.g.
"110" in the text refers to reference numerals "110A" and/or "110B"
in the figures).
[0086] Some portions of this description describe the embodiments
of the invention in terms of algorithms and symbolic
representations of operations on information (also referred to as
data). Examples include encryption, decryption, primary keys,
passwords, and cipher texts. These algorithmic descriptions and
representations are commonly used by those skilled in the data
processing arts to convey the substance of their work effectively
to others skilled in the art. These operations, while described
functionally, computationally, or logically, are understood to be
implemented by computer programs or equivalent electrical circuits,
microcode, or the like. Furthermore, it has also proven convenient
at times, to refer to various arrangements of operations as
"business logic", "modules", "software", "software instructions",
"computer program instructions", and "middleware" without loss of
generality. The described operations may be embodied in software,
firmware, hardware, or any combinations thereof.
[0087] Generally, the operations described herein are sufficiently
numerous in quantity, and individually are sufficiently complex so
in order to achieve the desired outcome in a reasonable amount of
time (e.g., performing many such operations in the space of
seconds, minutes, or hours) that consequently these operations can
only carried out by a computing device including a processor. Thus,
these operations cannot be performed by the human mind in any
reasonable amount of time, particularly for business applications
where expediency is important.
[0088] In one embodiment, the operations described herein are
implemented in a software module as computer program instructions
(also referred to as computer program code) that is stored using a
persistent computer-readable storage medium, where the software
module can be executed by a computer processor.
[0089] Finally, the language used in the specification has been
principally selected for readability and instructional purposes,
and it may not have been selected to delineate or circumscribe the
inventive subject matter. Persons skilled in the relevant art can
appreciate that many modifications and variations are possible in
light of the above disclosure. It is therefore intended that the
scope of the invention be limited not by this detailed description,
but rather by any claims that issue on an application based hereon.
Accordingly, the disclosure of the embodiments of the invention is
intended to be illustrative, but not limiting, of the scope of the
invention, which is set forth at least in part by the following
claims.
* * * * *