Scribd
Upload
120 views23 pages

Ota Slides

OTA (Over The Air) allows large database updates to be applied efficiently by writing changes in a sorted order to avoid write amplification. It also supports resuming interrupted updates. The OTA database contains the changes, and an application uses the OTA API to copy changes from OTA to the target database in multiple passes to update each index in order. This makes updates efficient and resumable.

Uploaded by

mahesh_rampalli
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
120 views23 pages

Ota Slides

OTA (Over The Air) allows large database updates to be applied efficiently by writing changes in a sorted order to avoid write amplification. It also supports resuming interrupted updates. The OTA database contains the changes, and an application uses the OTA API to copy changes from OTA to the target database in multiple passes to update each index in order. This makes updates efficient and resumable.

Uploaded by

mahesh_rampalli
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 23

Overview: What is OTA?

● A system for applying large updates to field


databases that
– May be more efficient than writing the db using
other methods, and
– Allows an update to be suspended and resumed.
● An acronym for “Over The Air”
Overview: How is OTA used
● The user creates an “OTA database” containing
the changes to apply to the “target database”
● Then somehow sends it to the device
● Where an application uses the OTA module API
to copy changes from the OTA to the target
database.
Overview: OTA Deployment

System
Applications OTA Application

target.db
ota.db
state.db

Read-only access Read/write access Can be the same db


Overview: OTA Database Example
Target Database Schema:

CREATE TABLE t1(a TEXT PRIMARY KEY, b INTEGER, c INTEGER);

Update in SQL:

INSERT INTO t1 VALUES('one', 1, 4);


UPDATE t1 SET b=5 WHERE a = 'two';
DELETE FROM t1 WHERE a = 'three';

OTA Database Schema

CREATE TABLE data_t1(a, b, c, ota_control);

OTA Database Contents:

a b c ota_control
'one' 1 4 0 If the target table
'three' NULL NULL 1 actually has a column
'two' 5 NULL '.x.' named “ota_control”,
you're out of luck
Overview: Application Code
int apply_ota(){
sqlite3ota *pOta;
int rc;

/* Open an OTA handle */


pOta = sqlite3ota_open("target.db", "ota.db", "state.db");

/* Call sqlite3ota_step() on the OTA handle until it returns


** something other than SQLITE_OK. Either SQLITE_DONE to
** indicate that the update has been fully applied, or else
** an SQLite error code. */
do {
rc = sqlite3ota_step(pOta);
}while( rc==SQLITE_OK );

/* Clean up the OTA handle. sqlite3ota_close() always returns


** the same value as the last call to sqlite3ota_step(). */
rc = sqlite3ota_close(pOta);
assert( rc!=SQLITE_OK );
return rc==SQLITE_DONE ? SQLITE_OK : rc;
}
What does this do again?
● Why are we using OTA instead of regular SQL?
– Avoids the “write amplification”/“cache
thrashing” problem.
– Regular SQL transactions are not resumable.

● If neither of these seem applicable...


Write Amplification
● When writing X bytes of data to the database
results in (N * X) bytes being written to disk. For
inconveniently large values of N.
● In the worst cases, (N * X) bytes might be many
times larger than the target database (!!).
● The worst cases occur when the working set is
larger than the cache.
● For b-trees, the solution is to sort the keys before
they are inserted.
Write Amplification: Illustration

B-Tree:

SQLite Cache:

ZipVFS

Operating System
Write Amplification
● For a non-trivial schema:
CREATE TABLE t1(a PRIMARY KEY, b, c UNIQUE) WITHOUT ROWID;
CREATE INDEX t1b ON t1(b);

Each SQL operation writes up to 3 b-trees.


● So it is not possible to simply sort the SQL
statements:
INSERT INTO t1 VALUES(0, 'A', 'iii');
INSERT INTO t1 VALUES(1, 'C', 'i');
INSERT INTO t1 VALUES(2, 'B', 'ii');
...
Write Amplification
● OTA makes all required updates to each b-tree
before moving on to the next.
● By making multiple passes of the data_xxx
table:
a b c ota_control
0 'A' 'iii' 0
1 'C' 'i' 0
2 'B' 'ii' 0

SELECT … FROM data_t1; -- For the PK b-tree


SELECT … FROM data_t1 ORDER BY c; -- For the UNIQUE b-tree
SELECT … FROM data_t1 ORDER BY b; -- For index t1b
How UPDATE/DELETE ops work
● UPDATE and DELETE operations use the state
database.
● An OTA update containing a DELETE:
DELETE FROM t1 WHERE a=4;

a b c ota_control
4 NULL NULL 1

● The first pass can delete the entry from the PK


b-tree, but how about the other two?
● Values required to delete index entries are
saved in the state db as part of the first pass.
Write Amplification Example
● POI Database Schema:
CREATE TABLE poiServiceLocationTable(
serviceLocationId IPK, poiId, catId, mortonCode, iconSetId
);
-- ...and 4 indexes…\

● 985,000 entries on 14356 pages of 8192 bytes


each. Zipvfs. File size of 56MB.
● 69500 inserts, 39352 deletes, 7075 updates.
● Directly: 28582 page writes, 40000 reads, 38
seconds.
● With OTA: 14343 page reads/writes, 9 seconds.
...So that was “write amplification”. Next is
“resumable transactions”...
Resumable Transaction Support
● Means this scenario is supported:
– Applications reading from a database in state A.
– OTA Application begins applying an update (to
move the db to state B).
– The system is rebooted.
– Applications resume, still reading the database in
state A
– The OTA Application also picks up where it left off –
midway through applying the update.
Application Code II
int apply_ota(int *pbFinished){
sqlite3ota *pOta;
int rc;

/* Open an OTA handle */


pOta = sqlite3ota_open("target.db", "ota.db", "state.db");

/* Call sqlite3ota_step() until either the entire update is


** applied or the application needs to shutdown. */
do {
rc = sqlite3ota_step(pOta);
if( <app needs to shutdown> ) break;
}while( rc==SQLITE_OK );

/* Clean up the OTA handle. sqlite3ota_close() always returns


** the same value as the last call to sqlite3ota_step(). */
rc = sqlite3ota_close(pOta);
*pbFinished = (rc==SQLITE_DONE);
return rc==SQLITE_DONE ? SQLITE_OK : rc;
}
How SQLite Wal Mode Works
● How “wal mode” works:
– Writers append data to the *-wal file
– Readers read from both the db file and the *-wal file
– Eventually data is copied from the *-wal file into the
db file. This is called a checkpoint.

target.db

Database Applications

target.db-wal
How Resumable Transactions Work
● Target database must be in “rollback mode”
● OTA writes into the *-oal file, which is like a *-
wal file but nobody else knows it's there
● Once the entire update is in the *-oal file, OTA
moves it to *-wal
● Then runs a checkpoint (incrementally – which
regular clients cannot do).
How Resumable Transactions Work
Database Database
Applications Applications

target.db target.db
target.db-oal target.db-wal

OTA Application OTA Application


...And that was “resumable transactions”. Now
more about the OTA database schema...
Details: PRIMARY KEY-less Tables
● Add an “ota_rowid” column to data_xxx table:
Target Database Schema:
CREATE TABLE t1(a TEXT, b INTEGER, c INTEGER);

Update in SQL:
INSERT INTO t1 VALUES('x', 'y', 'z');
INSERT INTO t1(rowid, a, b, c) VALUES(13, 'a', 'b', 'c');
DELETE FROM t1 WHERE rowid = 5;

OTA Database Schema


CREATE TABLE data_t1(ota_rowid, a, b, c, ota_control);

OTA Database Contents:

ota_rowid a b c ota_control
5 NULL NULL NULL 1
13 'a' 'b' 'c' 0
NULL 'x' 'y' 'z' 0
Details: Virtual Tables
● Virtual tables must have a “rowid” column that
works like a PRIMARY KEY (most do)
● Add the “ota_rowid” column to the data_xxx
table
● Any hidden columns (i.e. languageid columns)
are optional
Target Database Schema:
CREATE VIRTUAL TABLE x1 USING fts4(txt, languageid=lid);
OTA Database Schema:
CREATE TABLE data_x1(ota_rowid, txt, lid, ota_control);
Details: Limitations
● INSERT, UPDATE and DELETE operations only,
● INSERT may not use default values,
● UPDATE and DELETE must identify their target rows by
primary key,
● UPDATE statements may not modify primary key columns,
● No triggers are fired,
● No foreign key constraint handling is performed,
● CHECK constraints are not enforced
● Constraint handling (for UNIQUE and NOT NULL constraints) is
always “OR ROLLBACK”.
And Finally...
● It does large updates to databases:
– With b-tree writes in sorted order to avoid “write
amplfication”, and
– With support for “resumable transactions”

● Sort the contents of the data_xxx tables by PK.

● Canonical documentation is comments in


“sqlite3ota.h” - part of devkit.

You might also like