Using VoltDB

Table of Contents

Preface

1. Overview

1.1. What is VoltDB?

1.2. Who Should Use VoltDB

1.3. How VoltDB Works

1.3.1. Partitioning

1.3.2. Serialized (Single-Threaded) Processing

1.3.3. Partitioned vs. Replicated Tables

1.3.4. Ease of Scaling to Meet Application Needs

2. Installing VoltDB

2.1. Operating System and Software Requirements

2.2. Installing VoltDB

2.2.1. Upgrading From Older Versions

2.2.2. Installing Standard System Packages

2.2.3. Building a New VoltDB Distribution Kit

2.3. Setting Up Your Environment

2.4. What is Included in the VoltDB Distribution

2.5. VoltDB in Action: Running the Sample Applications

3. Designing Your VoltDB Application

3.1. Designing the Database

3.1.1. Partitioning Database Tables

3.1.2. Replicating Tables

3.2. Designing the Data Access (Stored Procedures)

3.2.1. Writing VoltDB Stored Procedures

3.2.2. VoltDB Stored Procedures and Determinism

3.2.3. The Anatomy of a VoltDB Stored Procedure

3.2.4. Partitioning Stored Procedures

3.3. Designing the Application Logic

3.3.1. Connecting to the VoltDB Database

3.3.2. Invoking Stored Procedures

3.3.3. Invoking Stored Procedures Asynchronously

3.3.4. Closing the Connection

3.4. Handling Errors

3.4.1. Interpreting Execution Errors

3.4.2. Handling Timeouts

3.4.3. Interpreting Other Errors

4. Simplifying Application Development

4.1. Default Procedures

4.2. Shortcut for Defining Simple Stored Procedures

4.3. Writing Stored Procedures Inline Using Groovy

4.4. Verifying Expected Query Results

5. Building Your VoltDB Application

5.1. Compiling the Client Application and Stored Procedures

5.2. Declaring the Stored Procedures

5.3. Building the Application Catalog

6. Running Your VoltDB Application

6.1. Defining the Cluster Configuration

6.1.1. Determining How Many Partitions to Use

6.1.2. Configuring Paths for Runtime Features

6.1.3. Verifying your Hardware Configuration

6.2. Starting a VoltDB Database for the First Time

6.2.1. Simplifying Startup on a Cluster

6.2.2. How VoltDB Database Startup Works

6.3. Starting VoltDB Client Applications

6.4. Shutting Down a VoltDB Database

6.5. Stopping and Restarting a VoltDB Database

6.5.1. Save and Restore

6.5.2. Command Logging and Recovery

6.6. Modes of Operation

6.6.1. Admin Mode

6.6.2. Starting the Database in Admin Mode

7. Updating Your VoltDB Database

7.1. Planning Your Application Updates

7.2. Updating the Database Schema on a Running Database

7.2.1. Validating the Updated Catalog

7.2.2. Managing the Update Process

7.3. Updating the Database Using Save and Restore

7.4. Updating the Hardware Configuration

7.4.1. Adding Nodes with Elastic Scaling

7.4.2. Configuring How VoltDB Rebalances New Nodes

8. Security

8.1. How Security Works in VoltDB

8.2. Enabling Authentication and Authorization

8.3. Defining Users and Roles

8.4. Assigning Access to Stored Procedures

8.5. Assigning Access by Function (System Procedures, SQL Queries, and Default Procedures)

8.6. Using Default Roles

8.7. Integrating Kerberos Security with VoltDB

8.7.1. Installing and Configuring Kerberos

8.7.2. Installing and Configuring the JAVA Security Extensions

8.7.3. Configuring the VoltDB Servers and Clients

9. Saving & Restoring a VoltDB Database

9.1. Performing a Manual Save and Restore of a VoltDB Cluster

9.1.1. How to Save the Contents of a VoltDB Database

9.1.2. How to Restore the Contents of a VoltDB Database

9.1.3. Changing the Database Schema or Cluster Configuration Using Save and Restore

9.2. Scheduling Automated Snapshots

9.3. Managing Snapshots

9.4. Special Notes Concerning Save and Restore

10. Command Logging and Recovery

10.1. How Command Logging Works

10.2. Controlling Command Logging

10.3. Configuring Command Logging for Optimal Performance

10.3.1. Log Size

10.3.2. Log Frequency

10.3.3. Synchronous vs. Asynchronous Logging

10.3.4. Hardware Considerations

11. Availability

11.1. How K-Safety Works

11.2. Enabling K-Safety

11.2.1. What Happens When You Enable K-Safety

11.2.2. Calculating the Appropriate Number of Nodes for K-Safety

11.3. Recovering from System Failures

11.3.1. What Happens When a Node Rejoins the Cluster

11.3.2. Where and When Recovery May Fail

11.4. Avoiding Network Partitions

11.4.1. K-Safety and Network Partitions

11.4.2. Using Network Fault Protection

12. Database Replication

12.1. How Database Replication Works

12.1.1. Starting Replication

12.1.2. Replication and Existing Databases

12.1.3. Database Replication and Disaster Recovery

12.1.4. Database Replication and Completeness

12.1.5. Database Replication and Read-only Clients

12.2. Database Replication in Action

12.2.1. Starting Replication

12.2.2. Stopping Replication

12.2.3. Promoting the Replica When the Master Becomes Unavailable

12.2.4. Managing Database Replication

12.3. Using the Sample Applications to Demonstrate Replication

12.3.1. Replicating the Voter Sample Using the Enterprise Manager

12.3.2. Replicating the Voter Sample Using the Command Line

13. Exporting Live Data

13.1. Understanding Export

13.2. Planning your Export Strategy

13.3. Identifying Export Tables in the Schema

13.4. Configuring Export in the Deployment File

13.5. The File Connector

13.6. The HTTP Connector

13.6.1. Understanding HTTP Properties

13.6.2. Exporting to Hadoop via WebHDFS

13.7. The JDBC Connector

13.8. The Kafka Connector

13.9. The RabbitMQ Connector

13.10. How Export Works

13.10.1. Export Overflow

13.10.2. Persistence Across Database Sessions

14. Logging and Analyzing Activity in a VoltDB Database

14.1. Introduction to Logging

14.2. Creating the Logging Configuration File

14.3. Enabling Logging for VoltDB

14.4. Customizing Logging in the VoltDB Enterprise Manager

14.5. Changing the Timezone of Log Messages

14.6. Changing the Configuration on the Fly

15. Using VoltDB with Other Programming Languages

15.1. C++ Client Interface

15.1.1. Writing VoltDB Client Applications in C++

15.1.2. Creating a Connection to the Database Cluster

15.1.3. Invoking Stored Procedures

15.1.4. Invoking Stored Procedures Asynchronously

15.1.5. Interpreting the Results

15.2. JSON HTTP Interface

15.2.1. How the JSON Interface Works

15.2.2. Using the JSON Interface from Client Applications

15.2.3. How Parameters Are Interpreted

15.2.4. Interpreting the JSON Results

15.2.5. Error Handling using the JSON Interface

15.3. JDBC Interface

15.3.1. Using JDBC to Connect to a VoltDB Database

15.3.2. Using JDBC to Query a VoltDB Database

A. Supported SQL DDL Statements

CREATE INDEX — Creates an index for faster access to a table.

CREATE PROCEDURE AS — Defines a stored procedure composed of a SQL query.

CREATE PROCEDURE FROM CLASS — Defines a stored procedure associated with a Java class.

CREATE ROLE — Defines a role and the permissions associated with that role.

CREATE TABLE — Creates a table in the database.

CREATE VIEW — Creates a view into a table, used to optimize access to specific columns within a table.

EXPORT TABLE — Specifies that a table is for export only.

IMPORT CLASS — Specifies additional Java classes to include in the application catalog.

PARTITION PROCEDURE — Specifies that a stored procedure is partitioned.

PARTITION TABLE — Specifies that a table is partitioned and which is the partitioning column.

B. Supported SQL Statements

DELETE — Deletes one or more records from the database.

INSERT — Creates new rows in the database, using the specified values for the columns.

SELECT — Fetches the specified rows and columns from the database.

TRUNCATE TABLE — Deletes all records from the specified table.

UPDATE — Updates the values within the specified columns and rows of the database.

UPSERT — Either inserts new rows or updates existing rows depending on the primary key value.

C. SQL Functions

ABS() — Returns the absolute value of a numeric expression.

ARRAY_ELEMENT() — Returns the element at the specified location in a JSON array.

ARRAY_LENGTH() — Returns the number of elements in a JSON array.

AVG() — Returns the average of a range of numeric column values.

CAST() — Explicitly converts an expression to the specified datatype.

CEILING() — Returns the smallest integer value greater than or equal to a numeric expression.

CHAR() — Returns a string with a single UTF-8 character associated with the specified character code.

CHAR_LENGTH() — Returns the number of characters in a string.

COALESCE() — Returns the first non-null argument, or null.

CONCAT() — Concatenates two or more strings and returns the result.

COUNT() — Returns the number of rows selected containing the specified column.

CURRENT_TIMESTAMP — Returns the current time as a timestamp value.

DAY(), DAYOFMONTH() — Returns the day of the month as an integer value.

DAYOFWEEK() — Returns the day of the week as an integer between 1 and 7.

DAYOFYEAR() — Returns the day of the year as an integer between 1 and 366.

DECODE() — Evaluates an expression against one or more alternatives and returns the matching response.

EXP() — Returns the exponential of the specified numeric expression.

EXTRACT() — Returns the value of a selected portion of a timestamp.

FIELD() — Extracts a field value from a JSON-encoded string column.

FLOOR() — Returns the largest integer value less than or equal to a numeric expression.

FORMAT_CURRENCY() — Converts a DECIMAL to a text string as a monetary value.

FROM_UNIXTIME() — Converts a UNIX time value to a VoltDB timestamp.

HOUR() — Returns the hour of the day as an integer value.

LEFT() — Returns a substring from the beginning of a string.

LOWER() — Returns a string converted to all lowercase characters.

MAX() — Returns the maximum value from a range of column values.

MIN() — Returns the minimum value from a range of column values.

MINUTE() — Returns the minute of the hour as an integer value.

MONTH() — Returns the month of the year as an integer value.

NOW — Returns the current time as a timestamp value.

OCTET_LENGTH() — Returns the number of bytes in a string.

OVERLAY() — Returns a string overwriting a portion of the original string with the specified replacement.

POSITION() — Returns the starting position of a substring in another string.

POWER() — Returns the value of the first argument raised to the power of the second argument.

QUARTER() — Returns the quarter of the year as an integer value

REPEAT() — Returns a string composed of a substring repeated the specified number of times.

REPLACE() — Returns a string replacing the specified substring of the original string with new text.

RIGHT() — Returns a substring from the end of a string.

SECOND() — Returns the seconds of the minute as a floating point value.

SET_FIELD() — Returns a copy of a JSON-encoded string, replacing the specified field value.

SINCE_EPOCH() — Converts a VoltDB timestamp to an integer number of time units since the POSIX epoch.

SPACE() — Returns a string of spaces of the specified length.

SQRT() — Returns the square root of a numeric expression.

SUBSTRING() — Returns the specified portion of a string expression.

SUM() — Returns the sum of a range of numeric column values.

TO_TIMESTAMP() — Converts an integer value to a VoltDB timestamp based on the time unit specified.

TRIM() — Returns a string with leading and/or training spaces removed.

TRUNCATE() — Truncates a VoltDB timestamp to the specified time unit.

UPPER() — Returns a string converted to all uppercase characters.

WEEK(), WEEKOFYEAR() — Returns the week of the year as an integer value.

WEEKDAY() — Returns the day of the week as an integer between 0 and 6.

YEAR() — Returns the year as an integer value.

D. VoltDB CLI Commands

csvloader — Imports the contents of a CSV file and inserts it into a VoltDB table.

dragent — Starts the database replication agent.

jdbcloader — Extracts a table from another database via JDBC and inserts it into a VoltDB table.

kafkaloader — Imports data from a Kafka message queue into the specified database table.

sqlcmd — Starts an interactive command prompt for issuing SQL queries to a running VoltDB database

voltadmin — Performs administrative functions on a VoltDB database.

voltdb — Performs management tasks on the current server, such as compiling the application catalog and starting the database.

E. Deployment File (deployment.xml)

E.1. Understanding XML Syntax

E.2. The Structure of the Deployment File

F. VoltDB Datatype Compatibility

F.1. Java and VoltDB Datatype Compatibility

G. System Procedures

@AdHoc — Executes an SQL statement specified at runtime.

@Explain — Returns the execution plan for the specified SQL query.

@ExplainProc — Returns the execution plans for all SQL queries in the specified stored procedure.

@GetPartitionKeys — Returns a list of partition values, one for every partition in the database.

@Pause — Initiates admin mode on the cluster.

@Promote — Promotes a replica database to normal operation.

@Quiesce — Waits for all queued export data to be written to the connector.

@Resume — Returns a paused database to normal operating mode.

@Shutdown — Shuts down the database.

@SnapshotDelete — Deletes one or more native snapshots.

@SnapshotRestore — Restores a database from disk using a native format snapshot.

@SnapshotSave — Saves the current database contents to disk.

@SnapshotScan — Lists information about existing native snapshots in a given directory path.

@SnapshotStatus — Lists information about the most recent snapshots created from the current database.

@Statistics — Returns statistics about the usage of the VoltDB database.

@StopNode — Stops a VoltDB server process, removing the node from the cluster.

@SystemCatalog — Returns metadata about the database schema.

@SystemInformation — Returns configuration information about VoltDB and the individual nodes of the database cluster.

@UpdateApplicationCatalog — Reconfigures the database by replacing the application catalog and/or deployment configuration.

@UpdateLogging — Changes the logging configuration for a running database.