With V4.0, the durability and availability features associated with production usage move from the VoltDB community edition to the enterprise edition. This includes K-safety and snapshot save and restore. We encourage anyone who has been using the community edition to try the enterprise edition, which comes with a 30 day trial license. The community edition is still freely available for research and application development. However, the Enterprise Edition is recommended for all production environments.

To avoid possible mismatches between catalogs and server software, VoltDB now requires that you recompile your application catalog whenever you upgrade. This change was introduced in VoltDB 3.7.

The Java client library implements client affinity, allowing the client to route procedure calls to the most effective database server. VoltDB 4.0 changes the hashing algorithms for partitioning. As result, attempting to call a VoltDB v4.0 database using a VoltDB 3.x Java client library can result in spurious errors and decreased performance. We strongly recommend upgrading both server and client systems when upgrading to VoltDB V4.0. (This change does not affect non-Java clients.)

With 4.0. VoltDB has updated system requirements. CentOS 5.8 and OS X 10.6 are no longer supported. Also, VoltDB now requires Java 7. If you have not already, be sure to upgrade your server software to meet the new requirements:

VoltDB 4.0 introduces a new command line interface (CLI) for starting server nodes. The commands for voltdb create, recover, rejoin, and add now use standard LINUX command line syntax, including argument switches, consistent with the other VoltDB command line utilities voltadmin, csvloader, and sqlcmd. As a result, you will need to update any scripts you have for starting VoltDB servers from the command line. (Note that the REST interface has not changed.) For example, the command to start a new database cluster looks like this:

$ voltdb create mycatalog.jar \
     --deployment=deployment.xml \
     --host=voltsvr1 \
     --license=~/license.xml

There are also short forms for the most common switches:

$ voltdb create mycatalog.jar \
     -d deployment.xml \
     -H voltsvr1 \
     -l ~/license.xml

If you would prefer to temporarily continue using the old syntax for the voltdb command, you can either:

With 4.0, elasticity (the ability to expand a running database) is enabled by default. There are several consequences of this change:

With 4.0, support for remote clients has been dropped. All export clients must run on the database server. This change provides both more reliability and better performance for the export process.

As part of this change, the configuration of export in the deployment file has changed to simplify configuration. Specifically, the <onserver> element has been removed and the exportto attribute is replaced by the target attribute to the parent <export> element. The following example illustrates the new syntax you must use:

<export enabled="true" target="file">
   <configuration>
     <property name="type">csv</property>
     <property name="nonce">MyExport</property>
  </configuration>
</export>

In previous releases, VoltDB allowed you to define unique indexes on partitioned tables, although the index could not be guaranteed to be unique. If an index on a partitioned table does not include the partitioning column, there is no way for VoltDB to ensure the index is unique except within the partition where the query is executed.

To ensure global uniqueness for UNIQUE and PRIMARY KEY indexes, VoltDB no longer allows you to define such indexes on partitioned tables if the index does not include the partitioning column. As a result, schema that compiled successfully in the past may fail to compile under VoltDB 4.0.

However, there is an alternative. A new keyword, ASSUMEUNIQUE, has been added. You can use ASSUMEUNIQUE anywhere you use UNIQUE. The difference is that an ASSUMEUNIQUE index can be defined on a partitioned table where the index does not contain the partitioning column. VoltDB treats ASSUMEUNIQUE indexes as if they were unique, providing all the performance advantages of unique indexes. However, it is your responsibility to ensure the index is actually globally unique.

Although VoltDB verifies the index entry is unique within the current partition when a record is inserted, the danger is that if the index is not actually unique, any operation that repartitions the data could generate a constraint violation when two matching ASSUMEUNIQUE entries fall within the same partition. Operations that could generate such violations include restoring a snapshot to a different cluster configuration or rebalancing as a result of adding nodes. When this happens the repartitioning operation is likely to fail and you must correct the conflict before continuing.

Therefore, although there are performance advantages to using ASSUMEUNIQUE indexes, you use them at your own risk. You should ensure that the indexes are actually globally unique, either by design or by external verification, before inserting ASSUMEUNIQUE records into the database itself.

Swapping is an operating system feature that optimizes memory usage when running multiple processes. However, memory is a critical component of the VoltDB server process. Any contention for memory, including swapping, will have a very negative impact on performance and functionality.

We recommend using dedicated servers and disabling swapping when running the VoltDB database server process. Use the swapoff command to disable swapping on Linux systems. If swapping cannot be disabled for any reason, you can reduce the likelihood of VoltDB being swapped out by setting the kernel parameter vm.swappiness to zero.

There is an issue where, under certain conditions, the use of TCP segmentation offload (TSO) and generic receive offload (GRO) can cause nodes to randomly drop out of a cluster. The symptoms of this problem are that nodes timeout — that is, the rest of the cluster thinks they have failed — although the node is still running and no other network issues (such as a network partition) are the cause.

Disabling TSO and GRO is recommended for any VoltDB clusters that experience such instability. The commands to disable offloading are the following, where N is replaced by the number of the ethernet card:

ethtool -K ethN tso off
ethtool -K ethN gro off

Note that these commands disable offloading temporarily. You must issue these commands every time the node reboots.

A number of issues have been discovered and resolved in the elastic cluster functionality that is introduced in version 4.0. In particular, edge cases related to error conditions when nodes fail during elastic scaling have been identified and corrected.

This release includes several improvements to the Java client, including:

As a consequence of this last change, the method ClientResponse.getException() has been removed from the client API. Also, the causedBy property of ProcCallExceptions no longer returns an exception, All underlining exception information is returned as text by the getStatusString() method.

A number of improvements have been made to the JDBC interface as well. Similar to the Java API, the JDBC interface is provided as a single JAR file with no external dependencies. This means that if you use Guava and depended on the Guava library provided by VoltDB, you must either provide your own Guava JAR file or change the dependency to "com.google_voltpatches.common.*".

Other improvements to the JDBC interface include:

VoltDB now supports the CASE-WHEN-THEN-ELSE-END syntax in queries. For example:

SELECT Prod_name, 
    CASE WHEN price > 100.00 
          THEN `Expensive`
          ELSE `Cheap`
    END 
FROM products ORDER BY Prod_name;                      

VoltDB now supports the use of aggregate functions in the HAVING clause. For example:

SELECT game_id, count(*) FROM games
  GROUP BY game_id
  HAVING count(*) > 1;

The default Java heap size for the VoltDB server process has been increased from 1GB to 2GB. The new default more closely matches recommended settings for general purpose usage, More detailed recommendations can be found in the revised "Server Process Memory Usage" section of the VoltDB Planning Guide.

In previous versions there was an issue where, if a cluster was reduced in size and then restored from snapshots, future command logs of the cluster could not be recovered. One symptom of this issue is the fatal error "No viable snapshots to restore" during recovery.

The problem only occurs if the number of unique partitions in the cluster was reduced, either by reducing the number of servers, reducing the sites per host, or increasing the K-safety factor. With this release, the issue has been corrected. The issue is resolved for any affected databases by following the instructions for upgrading in the previous section; specifically, saving a snapshot, upgrading the software to VoltDB 4.0.1 or later, then restoring the snapshot.

In addition to the new features and enhancements listed above, VoltDB V4.0 includes fixes to a number of limitations in previous versions, including the following:

It is not currently possible to add new servers "on the fly" if the database schema contains only replicated tables. The workaround until this is fixed is to perform a catalog update to add a partitioned table (it does not need to contain any data) before adding new servers to the running database.

To ensure complete and accurate restoration of a database, recovery using command logs can only be performed to a cluster with the same number of unique partitions as the cluster that created the logs. If you restart and recover to the same cluster with the same deployment options, there is no problem. But if you change the deployment options for number of nodes, sites per host, or K-safety, recovery may not be possible.

For example, if a four node cluster is running with four sites per host and a K-safety value of one, the cluster has two copies of eight unique partitions (4 X 4 / 2). If one server fails, you cannot recover the command logs from the original cluster to a new cluster made up of the remaining three nodes, because the new cluster only has six unique partitions (3 X 4 / 2). You must either replace the failed server to reinstate the original hardware configuration or otherwise change the deployment options to match the number of unique partitions. (For example, increasing the site per host to eight and K-safety to two.)

VoltDB reserves the subfolder "segments" under the command log directory for storing the actual command log files. Do not add, remove, or modify any files in this directory. In particular, do not set the command log snapshot directory to a subfolder "segments" of the command log directory, or else the server will hang on startup.

Using the VoltDB Enterprise Manager, if a replica database was started with command logging, then stopped (intentionally or by accident), the Enterprise Manager cannot restart the database as a normal database using the recover action to reinstate the database's previous state. The Enterprise Manager can restore from a snapshot.

If you want to use the Enterprise Manager to stop a replica and restart it as a normal database, the recommended procedure is:

Note that this limitation is specific to the Enterprise Manager. Failed replica databases can be recovered manually using the command line.

It was possible that if all servers in the cluster fail and rejoin without stopping the database itself, data waiting in the export queue can be lost. This only happens if all servers in the cluster are cycled (stopped and rejoined). The workaround until this problem is resolved is to manually call the @Quiesce system procedure to make sure the export queue is cleared before all servers are stopped and restarted.

Use of SELECT DISTINCT is supported for a single column (such as SELECT DISTINCT Price FROM Inventory). However, using DISTINCT with multiple columns or arithmetic expressions is not currently supported. For example, the following SELECT DISTINCT statements should not be used:

SELECT DISTINCT Price, Discount FROM Inventory
SELECT DISTINCT (Price - Discount) FROM Inventory

VoltDB currently intercepts assertions as part of its handling of stored procedures. Attempts to use assertions in stored procedures for debugging or to find programmatic errors will not work as expected.

There is a problem with how the math library used to build the C++ client library handles large decimal values on 32-bit operating systems. As a result, the C++ library cannot serialize and pass Decimal datatypes reliably on these systems.

Note that the C++ client interface can send and receive Decimal values properly on 64-bit platforms.

To ensure proper recovery on startup, either from command logs or the last database snapshot, make sure all snapshot files — or at least complete subsets of the snapshot files — are available on the nodes of the cluster. If you delete or move snapshot files (for example, copying all snapshot files to a single node) be sure to keep all of the files for each node together. Do not selectively delete or move individual files or else the recovery may fail.

If the HTTP port is enabled (which it is by default) but the process does not have execute privileges for the /tmp directory, VoltDB throws a fatal exception on startup. The error message indicates that the process could not load the native library for the Snappy web server.

The workaround is to either use an account that has execute permission for the /tmp directory or specify an alternate directory that the account can access by assigning the environment variable VOLTDB_OPTS = "-Djava.io.tmpdir={alternate-tmpdir}".

Automatic snapshots (enabled through the deployment file) are intended to create periodic snapshots on a scheduled basis while the database is running. However, there are instances when the automatic snapshots stop occurring. This problem is sporadic and only happens on K-safe clusters where nodes fail and rejoin.

Once automatic snapshots stop, they will not restart until the cluster stops and restarts. The workaround is, after a node fails and rejoins, to check the logs to make sure automatic snapshots are still occurring. If not you can create a cron job to periodically take snapshots manually using the voltadmin save command until the next time the cluster reboots.

Normally, manual snapshots (those created with the Take a Snapshot button) are copied to the management server. However, if automated snapshots are also being created and copied to the management server, it is possible for an automated snapshot to override the manual snapshot.

If this happens, the workaround is to turn off automated snapshots (and their copying) temporarily. To do this, uncheck the box for copying snapshots, set the frequency to zero, and click OK. Then re-open the Edit Snapshots dialog and take the manual snapshot. Once the snapshot is complete and copied to the management server (that is, the manual snapshot appears in the list on the dialog box), you can re-enable copying and automated snapshots.

When the Enterprise Manager starts, it unpacks files that the web server uses into a subfolder of the /tmp directory. It does not delete these files when it stops. Under normal operation, this is not a problem. However, if you upgrade to a new version of the Enterprise Edition, files for the new version become intermixed with the older files and can result in the Enterprise Manager starting databases using the wrong version of VoltDB. To avoid this situation, make sure these temporary files are deleted before starting a new version of VoltDB Enterprise Manager.

The /tmp directory is emptied every time the server reboots. So the simplest workaround is to reboot your management server after you upgrade VoltDB. Alternately, you can delete these temporary files manually by deleting the winstone subfolders in the /tmp directory:

$ rm -vr /tmp/winstone*

When upgrading VoltDB Enterprise Edition, please note that the configuration files for the Enterprise Manager are not upwardly compatible. New product features may make existing database and/or deployment definitions unusable. It is always a good idea to delete existing configuration information before upgrading. You can delete the configuration files by deleting the ~/.voltdb directory. For example:

$ rm -vr ~/.voltdb

In the past, it was possible to run two (or more) databases on a single physical server by defining two logical servers with the same IP address and making the ports for each database unique. However, as a result of internal optimizations introduced in VoltDB 2.7, this technique no longer works when using the Enterprise Manager.

We expect to correct this limitation in a future release. Note that it is still possible to start multiple databases on a single server manually using the VoltDB shell commands.

For partitioned tables, the value of the column used to partition the table determines what partition the row belongs to. If you use UPDATE to change this value and the new value belongs in a different partition, the UPDATE request will fail and the stored procedure will be rolled back.

Updating the partition column value may or may not cause the record to be repartitioned (depending on the old and new values). However, since you cannot determine if the update will succeed or fail, you should not use UPDATE to change the value of partitioning columns.

The workaround, if you must change the value of the partitioning column, is to use both a DELETE and an INSERT statement to explicitly remove and then re-insert the desired rows.

If you refer to a table or column name that does not exist, the VoltDB compiler issues the error message "user lacks privilege or object not found". This can happen, for example, if you misspell a table or column name.

Another situation where this occurs is if you mistakenly use double quotation marks to enclose a string literal (such as WHERE ColumnA="True"). ANSI SQL requires single quotes for string literals and reserves double quotes for object names. In the preceding example, VoltDB interprets "True" as an object name, cannot resolve it, and issues the "user lacks privilege" error.

The workaround is, if you receive this error, to look for misspelled table or columns names or string literals delimited by double quotes in the offending SQL statement.

VoltDB opens a file descriptor for every client connection to the database. In normal operation, this use of file descriptors is transparent to the user. However, if there are an inordinate number of concurrent client connections, or clients open and close many connections in rapid succession, it is possible for VoltDB to exceed the process limit on file descriptors. When this happens, new connections may be rejected or other disk-based activities (such as snapshotting) may be disrupted.

In environments where there are likely to be an extremely large number of connections, you should consider increasing the operating system's per-process limit on file descriptors.

There are several situations where an attempt to recover a database — either from a snapshot or command logs — may fail. For example, restoring a snapshot where a unique index has been added to a table can result in a constraint violation that causes the restore, and the database, to fail. Similarly, a command log may contain a transaction that originally succeeded but fails and raises an exception during playback.

In both of these situations, VoltDB issues a fatal error and stops the database to avoid corrupting the contents.

Although protecting you from an incomplete recovery is the appropriate default behavior, there may be cases where you want to recover as much data as possible, with full knowledge that the resulting data set does not match the original. VoltDB provides two techniques for performing partial recoveries in case of failure:

Logging constraint violations — There are several situations that can cause a snapshot restore to fail because of constraint violations. Rather than have the operation fail as a whole, you can request that constraint violations be logged to a file instead. This way you can review the tuples that were excluded and decide whether to ignore or replace their content manually after the restore completes.

To perform a manual restore that logs constraint violations rather than stopping when they occur, you use a special JSON form of the @SnapshotRestore system procedure. You specify the path of the log files in a JSON attribute, duplicatePaths. For example, the following commands perform a restore of snapshot files in the directory /var/voltdb/snapshots/ with the unique identifier myDB. The restore operation logs constraint violations to the directory /var/voltdb/logs.

$ sqlcmd
1> exec @SnapshotRestore '{ "path":"/https/docs.voltactivedata.com/var/voltdb/snapshots/", 
                            "nonce":"myDB", 
                            "duplicatesPath":"/var/voltdb/logs/" }';
2> exit

Constraint violations are logged as needed, one file per table, to CSV files with the name {table}-duplicates-{timestamp}.csv.

Safe Mode Recovery — On rare occasions, recovering a database from command logs may fail. This can happen, for example, if a stored procedure introduces non-deterministic content. If a recovery fails, the specific error is known. However, there is no way for VoltDB to know the root cause or how to continue. Therefore, the recovery fails and the database stops.

When this happens, VoltDB logs the last successful transaction before the recovery failed. You can then ask VoltDB to recover up to but not including the failing transaction by performing a recovery in safe mode.

You request safe mode by adding the --safemode switch to the command line when starting the recovery operation, like so:

$ voltdb recover --safemode -license ~/license.xml

When VoltDB recovers from command logs in safe mode it enables two distinct behaviors:

This means that if you are recovering using an automated snapshot (rather than command logs), you can recover some data even if there are constraint violations during the snapshot restore. Also, when recovering from command logs, VoltDB will ignore constraint violations in the command log snapshot and replay all transactions that succeeded in the previous attempt.

It is important to note that to successfully use safe mode with command logs, you must perform a regular recovery operation first — and have it fail — so that VoltDB can determine the last valid transaction. Also, if the command log contains both constraint violations and failed transactions, you may need to run recovery in safe mode multiple times to recover as much data as possible.

This is not a problem when looking at VoltDB logs separately. However, you should be aware of this distinction when integrating logging of VoltDB with logging of other system components that use the local time zone (rather than GMT). You may want to convert one or the other log streams so the time zones match.

The file voltdb/log4j.xml lists all of the VoltDB-specific logging categories. It also serves as a useful logging schema. The sample applications and the VoltDB shell commands use this file to configure logging and it is recommended for new application development.

VoltDB now supports export to Apache Kafka, a distributed commit log service that operates like a message queue. Kafka is selectable as the target export client, similar to the existing file and JDBC export clients. Customers interested in trying our new Kafka export capabilities and providing feedback, please contact [email protected] for details.

VoltDB includes a new utility, voltify, that helps you migrate from an existing MySQL database to VoltDB. The utility connects to a running MySQL database and creates a target schema and starter project in VoltDB to match the source database. If you are interested, you can find instructions for volity in the VoltDB github repository at the following URL:

http://github.com/VoltDB/voltdb/blob/master/tools/voltify-README.md

We encourage anyone who tries it to provide feedback in the VoltDB forums, http://forum.voltdb.com. Thank you.