There was an issue where a rejoining node could fail with either a "ParameterSet doesn't support type" or "array out of bounds" error. This condition could occur only when using K-safety and database replication (DR) and a failed node on the master cluster was rejoining. This issue has been resolved.

When using database replication (DR), there was a race condition that could cause the master database to stall and report the error "IllegalStateException: invocation buffer in not compiled". Although the error is real, the database did not stop as it should have. The database now takes the appropriate action and stops when this error occurs.

The following limitation in previous versions have been resolved:

In rare cases, frequent deletes can cause memory compaction that coincides with a simultaneous snapshot. In the past, the result was that the servers would stop processing transactions and fill the log file with multiple messages reporting "tuple count > 0 after streaming". This error has been corrected. The triggering condition, although rare, can still occur. However, now if compaction coincides with an automated snapshot (such as a command log snapshot), the snapshot is requeued. If compaction coincides with a manual snapshot, the snapshot is canceled and reported to the user. If this occurs, simply reinitiate the manual snapshot.

Additional changes have been made to improve memory utilization for database replication (DR) buffering in virtualized environments and snapshot processing. The setting -XX:MaxDirectMemorySize=<size> is no longer required.

There was an issue where, if the master database was under load, the DR agent would timeout attempting to retrieve the initial DR snapshot. This issue has been resolved by extending the timeout period on the DR agent from 5 seconds to 90 seconds and modifying the master database to issue periodic ACKs even when processing snapshots for DR initiation.

Under certain circumstances, a race condition in database replication (DR) was found that could result in a node on the master cluster crashing with the error "java.lang.IllegalStateException: buffer is already compiled". This issue has been resolved.

Changes have been made to DR buffering to use Java-managed DirectByteBuffers instead of unmanaged DirectByteBuffers. This change requires setting -XX:MaxDirectMemorySize=<size> in the VOLTDB_OPTS. The recommended initial setting is a size of 4 Gigabytes, but this should be tested against the actual application and hardware to best determine the setting.

Previously, when inserting values into a VoltDB DECIMAL datatype, any values with a scale greater than 12 (that is, more than 12 decimal places) were rejected and generated an exception in the JDBC interface. This behavior has been changed and VoltDB now rounds higher precision values to the closest 12 decimal digits.

When processing ad hoc queries that use parameters and placeholders (rather than a single SQL statement as a text string), VoltDB now makes use of previously cached queries to significantly improve performance for repeated queries. This can be most notable for JDBC prepared statements that are implemented as ad hoc queries with parameters.

There was an issue in earlier releases where using an INSERT INTO SELECT statement with an export table as the target for the insert either generated a null pointer exception or did not insert the expected data into the export stream. The issue only applies to INSERT INTO SELECT as an ad hoc query or within a multi-partitioned stored procedure.

These issues have now been corrected.

There was an issue in previous versions (starting with VoltDB 4.8), where if a query runs for a significant amount of time, VoltDB attempts to log a warning. However, the warning generates an error (index out of bounds) and stops the database.

This issue is now fixed.

There was an issue in earlier releases where command log recovery would hang if the command logs contained an application catalog update (for example, from a voltadmin update command) followed by a subsequent ad hoc query that writes to the database. The recovery would either stop or silently fail to process the transaction when attempting to replay the ad hoc query.

This issue is now fixed.

In previous releases, a defect in the JSON HTTP server could cause the server to stop handling client requests if the Management Center browser window was left open for a few hours. The more Management Center sessions in use, the sooner the problem would manifest itself.

If you have a server that has experienced this issue, you may stop the server and rejoin it to the cluster. Or you can stop the entire cluster and recover the database.

The problem is resolved in this release. Because this issue impedes normal operation of the database, we recommend any customers using VoltDB 4.9 or 4.9.1 update to VoltDB 4.9.2 as soon as possible.

Under certain conditions, the voltdb collect command could fail displaying a traceback reporting an attribute error. This problem has been resolved.

HTTP export now supports CSV output format when using HttpFS as a destination for WebHDFS export. To export CSV data to HttpFS, add the following properties to the export configuration in the deployment.xml:

<property name="httpfs.enable">true</property>
<property name="type">csv</property>

HTTP export now supports HttpFS as a destination for WebHDFS export using the Avro format. CSV format will be supported in the next major release. To enable exporting to HttpFS add the following properties to the export configuration in the deployment.xml:

<property name="httpfs.enable">true</property>
<property name="type">avro</property>

The following bugs were fixed in this release:

VoltDB 4.9 introduces a new management console that is available from any running VoltDB database. The VoltDB Management Center integrates the catalog report and web studio functionality, plus the monitoring functions of the VoltDB Enterprise Manager, into a single interface. Available at http://server:8080 by default, the Management Center provides real-time monitoring, a schema browser, and interactive SQL queries.

Security has been revamped in this release, providing new permissions organized around functional capabilities. Security still operates on the basic principles of users assigned to roles in the deployment file and roles assigned permissions in the schema. However, the generic permissions (not associated with a specific stored procedure) have been reorganized and renamed to match logical functions. So, for example, there are named permissions for SQLREAD (that is, read-only ad hoc SQL queries), SQL (all SQL queries), ALLPROC (all stored procedures, including the default stored procedures), and ADMIN (full access, including read and modify system procedures).

For backwards compatibility, the previous named permissions ADHOC and SYSPROC are still permitted and map to the new SQL and ADMIN permissions. Note, however, that this mapping increases the permissions of the old permissions slightly, giving ADHOC access to default procedures and giving SYSPROC access to all user-defined stored procedures.

This release also includes two new predefined roles, ADMINISTRATOR and USER, that are always available and can be assigned to users in the deployment file. The ADMINSTRATOR role has ADMIN permissions and the USER role has SQL and ALLPROC permissions. In other words, the ADMINISTRATOR role has full access while the USER role can run any SQL query, stored procedure, or read-only system procedure, but cannot access system procedures that modify the database (such as @Shutdown and @SnapshotRestore).

See the chapter on security in the Using VoltDB manual for details.

The sqlcmd utility for entering interactive SQL queries from a command line is being overhauled. The first major addition is a new command line argument, --stop-on-error, that lets you specify whether a batch of commands — piped to the utility from the shell prompt or executed through the FILE directive — stops on the first error or continues to process subsequent commands. The default for the new argument is --stop-on-error=true, which corresponds to previous behavior.

It is now possible to cancel read queries that take too long based on a predefined query timeout period. The query timeout property, which is a child of <systemsettings> in the deployment file, lets you set a timeout limit for read queries. Any read-only queries (or batch of queries submitted using voltExecuteSQL()) that exceed the limit will be cancelled and an exception returned to the calling application. You set the time out limit in milliseconds. For example, the following deployment file fragment sets the query timeout to 3 seconds:

<systemsettings>
   <query timeout="3000"/>
</systemsettings>

By default, there is no query timeout limit. See the section on database configuration options in the Administrators Guide for details.

Previously, VoltDB supported use of the DISTINCT keyword on a single column only. You can now use DISTINCT with multiple columns.

A new logger has been added to the VoltDB Log4J configuration that logs any ad hoc queries to the database. The new logger, called ADHOC, is not enabled by default, but can be enabled to record the text of the query and, if security is enabled, the user who submitted it.

VoltDB support for CentOS and Red Hat as base operating operating systems has been extended to include version 7.0.

Support for several older technologies, obsolete synonyms for existing commands, and features that were superseded in previous releases will be removed in the next major release of VoltDB. The deprecated features that will be removed in VoltDB v5.0 include:

In addition, use of Java 7 as the JDK will be supported but deprecated starting with VoltDB 5.0. This means Java 7 will be supported for the initial releases of VoltDB V5.0. However, use of Java 8 is recommended. Support for Java 7 will be removed from VoltDB some time after Java 7 reaches end-of-llife, currently scheduled for April 2015.

In addition to the preceding new features and enhancements, a number of known issues have been corrected, including:

The voltdb collect command has a new argument, --days, that lets you selectively collect only the most recent log files. For example, the command voltdb collect --days=2 collects logs files from the last two days. If you do not specify the --days argument, the command uses a default value of 14 days.

The HTTP connector now supports writing the export files in the Apache Avro format. See the chapter on Export in the Using VoltDB manual for details.

A new SQL statement, UPSERT is now supported. UPSERT has syntax similar to the INSERT statement, except it can only be used on tables that have a primary key. If a record with the specified key value doesn't exist, the UPSERT statement performs an insert operation; if such a record does exist, it performs an update operation. See the documentation for UPSERT for details.

The @Statistics system procedure has a new selector, CPU, which returns information on the percent of total available CPU used by the VotlDB server process for each server in the cluster.

A new function has been added that lets you modify JSON fields within a SQL query. The SET_FIELD() function returns a copy of a JSON-encoded string with the specified field replaced. For example, the following SQL statement replaces the "statebird" field in the JSON-encoded column Description when the State column contains "New Hampshire":

UPDATE Region 
   SET Description = SET_FIELD(Description,'statebird','"purple finch"')
   WHERE State = 'New Hampshire';

See the description of the SET_FIELD() function in the Using VoltDB manual for details.

A number of aliases, common in JDBC interfaces, have been added to provide access to existing functions. New aliases include INSERT, LTRIM, RTRIM, SUBSTR, LCASE, and UCASE. These aliases are not currently listed in the documentation, but will be added in a future update.

In addition to the preceding new features and enhancements, a number of known issues have been corrected, including:

The new jdbcloader copies the contents of a database table from a remote database using JDBC and inserts the records into a VoltDB table. Similar to the csvloader and kafkaloader utilities, jdbcloader provides yet another way to move data from other sources into VoltDB. See the jdbcloader reference page in the Using VoltDB manual for details.

VoltDB now includes an export connector that lets you export data to Hadoop via the WebHDFS API. The HTTP connector can be configured to export data to WebHDFS URLs, including the table name and other metadata as part of the URL. See the chapter on Export in the Using VoltDB manual for details.

The FIELD() function has been improved to provide easier access to lower-level fields within JSON strings. Originally, the second argument to the FIELD() function specified the name of a top-level field to be returned. Starting with this release, you can specify a field name path, separating the levels with periods, so you can select sub fields. For example, if the Film column contains the following the JSON-encoded string:

{"title":"Mark of Zorro","year":"1920",
  "cast":{"Zorro":"Douglas Fairbanks",
          "Lolita":"Marguerite De La Motte",
          "Pedro":"Noah Beery"},
  "alternateTitles:["La marca del Zorro",
                    "Das Zeichen des Zorro",
                    "Zorros Kendemaerke",
                    "Il segno di Zorro"]
}

You can extract the top-level field containing "1920" using FIELD(film,'year') and the second-level field containing "Douglas Fairbanks" using FIELD(film,'cast.Zorro'). You can also use array notation, so the function FIELD(film,'alternateTitles[3]') returns the Italian title of the film. See the description of the FIELD() function in the Using VoltDB manual for details.

It is now possible to use the INSERT INTO... SELECT statement in multi-partitioned procedures and queries. This means you can also use the statement in ad hoc queries, including interactive queries run through the sqlcmd utility. See the description of INSERT in the Using VoltDB manual for further information.

New information has been added to the results of the MEMORY selector for the @Statistics system procedure. A new column (PHYSICALMEMORY) has been added reporting the total physical memory for each server in the cluster. See the description of the @Statistics system procedure in the Using VoltDB manual for details.

In addition to the preceding new features and enhancements, a number of known issues have been corrected, including:

VoltDB has been tested and validated on Ubuntu 14.04. The VoltDB server software now supports the Ubuntu long-term support releases 10.04, 12.04, and 14.04.

When a native format snapshot completes, either for a snapshot initiated manually with voltadmin SAVE or periodic snapshots created by the system automatically , a flag file is now created to indicate the snapshot is done. The flag file is created in the same directory as the snapshot using the snapshot's unique identifier as the file name and .finished as the file extension. You can use the flag file to identify and backup or otherwise process completed snapshots.

This release contains both enhancements to existing SQL functions as well as new functions, many provided for compatibility with other SQL implementations. Changes include:.

See the appendix on SQL functions in Using VoltDB for more information.

VoltDB now supports the INSERT INTO... SELECT statement. For the initial release, this statement is limited to partitioned procedures only (it cannot be used in multi-partition stored procedures or as an ad hoc statement from the sqlcmd prompt). Broader usage will follow in future releases. See the documentation of INSERT in Using VoltDB for more information.

This release includes support in the Java client API for enabling automatic reconnection of the client. By default, if a connection to a database server is lost, it is up to the client application to detect the lost connection and reconnect as appropriate. Now you can enable automatic reconnection as part of the client configuration. When enabled, auto-reconnect periodically attempts to reconnect to servers whose connection was lost. To enable auto-reconnect, you set a property on the client configuration before creating the client object. For example:

ClientConfig config = new ClientConfig("", "");
config.setReconnectOnConnectionLoss(true);
client = ClientFactory.createClient(config);

The kafkaloader is a tool for bulk loading data into a VolltDB database from a Kafka message queue. By default, kafkaloader uses a custom procedure to batch multiple rows into a single insert operation. You can now specify an alternate stored procedure to use for loading the data into the table. To specify an alternate stored procedure, use the -p or --procedure flag on the command line, just as you would for the csvloader.I'm

Low-cardinality indexes — that is, indexes with very few unique values — can be problematic because finding any single row requires scanning all the rows with the same index value. Deleting records from such a table amplifies the problem because the database must not only find and delete the data but also the index entry itself.

In previous versions of VoltDB, deleting large volumes of data from a table with a low-cardinality index could take a long time. This release adds a performance optimization to dramatically improve the latency of delete operations on low-cardinality indexes. This optimization will also benefit any index with a commonly recurring value, such as the null value.

A new Java property, VOLTDB_DISABLE_DR, has been added that allows you to disable the DR port listening ports when you start the VoltDB database process. This property is intended to help protect VoltDB against port scanners. See the implementation note concerning port hardening for details.

A error was introduced in version 4.5 where use of the MIN() and MAX() functions on small VARCHAR columns could produce incorrect answers. This issue only occurred for VARCHAR columns of 63 bytes or less — that is, VARCHAR(15) or VARCHAR(63 BYTES) or smaller.

This bug is now fixed. Because the issue introduced in v4.5 involves wrong answers, anyone using v4.5 is urged to upgrade to v4.6 at their earliest possible convenience.

A timing issue related to command logging in earlier versions could result in unnecessary segment files being created. Under the worst conditions, unused segment files could continue to be created, filling up the /command_log/segments directory and using up available disk space.

This bug has now been fixed. Note that when the database restarts (for example, when you upgrade to 4.6) any extra segment files created by an earlier version as a result of this bug issue will automatically be deleted.

VoltDB has been tested and validated on Java 8. The VoltDB server software now supports both the Java 7 and Java 8 JDKs. The VoltDB Java client library continues to support Java 6 as well as 7 and 8.

An export connector for the RabbitMQ messaging service has been added to the VoltDB server software. See the chapter on "Exporting Live Data" in the Using VoltDB manual for details.

The Kafka export connector has been extended and improved to allow more control over the data being exported, including support for VARBINARY columns and new properties to control the format of the data and the value used as the Kafka partitioning key. More information is available in the export chapter of the Using VoltDB manual.

This release includes further performance improvements to the query planner, reducing temptable usage for multi-partition stored procedures. These improvements focus specifically on reducing temptable usage for queries involving GROUP BY clauses.

A new SQL function, FORMAT_CURRENCY(), converts a DECIMAL value to a text string. The function takes two arguments: the DECIMAL value and an integer specifying the decimal place to which the value is rounded. For example FORMAT_CURRENCY(1234.567, 2) = "1,234.57". See the Using VoltDB manual for details.

The graphs in VoltDB Web Studio have been enhanced, improving the accuracy of the latency graph, fixing a bug where the partition starvation graph was not available on K-safe clusters, and adding a new graph: the transaction breakdown graph. The transaction breakdown graph is a pie chart showing the weighted average of execution time for stored procedures running on the database cluster. The new graph type is selectable from the pulldown menu above the graphs in the Web Studio interface.

Previously a query joining six or more tables required the user to explicitly list the join order as an additional argument to the Java SQLStmt declaration. VoltDB now assumes that joins of six or more tables, without an explicit join order, are joined in the order the tables are listed in the query. This change allows many large joins to compile and run, even as ad hoc queries. Note, however, that the order of the tables in the query may not be the optimal join order, so explicitly defining a join order is still recommended when performance is a consideration. See the VoltDB Performance Guide for more information on join order.

When casting between TIMESTAMP and VARCHAR datatypes, VoltDB uses the string format YYYY-MM-DD hh.mm.ss.nnnnnn to represent the TIMESTAMP value. Previously, when entering a text string as input to a TIMESTAMP value, you had to provide the complete string, including both date and time (including fractional microseconds). You can now use just the date portion of the string as input, YYYY-MM-DD, and VoltDB assumes 00.00.00.000000 as the time portion.

In addition to the preceding new features and enhancements, a number of known issues have been corrected, including:

It was possible for pending export data to get stuck in the overflow queue if nodes fail unexpectedly. The consequences of this bug were that the pending data was not written to the export target after the node rejoined and export would eventually stop altogether after a restart. This bug has been fixed.

There was a problem where certain operations, such as command log recovery, could result in an error warning of "transactions moving backwards". Most instances of this error have been eliminated. However, if you encounter this error after upgrading to V4.4.1, please contact [email protected].

A race condition in the export-to-file client could result in a null pointer exception (NPE), causing export to stop. This bug has been fixed.

VoltDB recently added the ability to start the database using a catalog compiled with an older version of VoltDB. In this situation, VoltDB recompiled the catalog before starting. However, the recompiled catalog was not correctly saved as part of the command logs. As a consequence, the resulting command logs could not be recovered when the database restarted. This bug has been fixed.

A new property added to the export-to-kafka connector in 4.4 could cause the database to fail with a Java out-of-memory error. This bug has been fixed.

VoltDB now includes the kafkaloader utility, which is similar to csvloader, in that it imports from an Apache Kafka message queue and writes the records into the specified VoltDB database table. One difference with the kafkaloader is that, rather than the load process being a one-time event, kafkaloader continues to run, monitoring the queue for additional records and acting as a persistent import function. See the Using VoltDB manual for details.

Previously, if CSV input contained missing fields, the default behavior for the csvloader was to generate an error. The default behavior has been changed so csvloader now treats missing values as null, to be more in keeping with other applications such as spreadsheets. You can revert to the previous behavior, if you wish, by including the flag --blank=error when running csvloader.

The voltadmin update command originally required both a catalog and a deployment file. You can now specify either or both on the command line. If you specify just a catalog or just a deployment file, the other is assumed to remain the same. The file extensions .jar and .xml are used to identify which type of file is being updated. For example, all of the following are valid commands:

$ voltadmin update mycatalog.jar mydeployment.xml
$ voltadmin update mycatalog.jar
$ voltadmin update mydeployment.xml

When a schema is compiled into an application catalog, several simple stored procedures are created automatically for each table, including create for all tables and select, delete, and insert for tables with a primary key. A new automatic procedure, upsert, has been added for tables with primary keys. The upsert procedure inserts a new record if one does not already exist for the specified primary key, or updates the existing record if it does. See the Using VoltDB manual for details.

VoltDB has a limit on the amount of memory, known as temp tables, that can be used when processing SQL queries. It is possible to exceed this limit when performing very complex queries or queries with large intermediate result sets. In this release, the management of temp tables has been significantly improved, allowing more complex queries and combinations of queries to operate effectively within the limit. Users who have previously used the <temptables> setting in the deployment file to increase the temp table limit may find that this is no longer necessary,

Earlier releases limited use of subqueries in ad hoc and multi-partition transactions to replicated tables only. This limitation is now removed; ad hoc and multi-partition SELECT queries can include subqueries involving both replicated and partitioned tables.

Prior to VoltDB V3.0, a project definition file was required to compile an application catalog. With V3.0, support for compiling schema files directly was added and use of the project definition file was deprecated and removed from the documentation. However, support for project definition files has remained in the product up to the current release. However, that support will be removed in an upcoming version. Anyone still using a project definition file should migrate to current usage to avoid problems in the future.

In addition to the preceding new features and enhancements, a number of known issues have been corrected, including:

VoltDB now includes export to Apache Kafka as a standard export client. Kafka export was added as a software preview several releases ago. You can now select and configure Kafka export in the deployment file or through the REST interface. (Kafka export is not accessible from the VoltDB Enterprise Manager at this time.) See the chapter on export in the Using VoltDB manual for details.

VoltDB now allows you to use Kerberos to authenticate Java clients to the VoltDB server. Kerberos security in VoltDB supports the same users, roles, and permissions defined in the deployment file and schema as with normal security, but uses the Kerberos authentication protocols to identify authorized clients to the database servers. Kerberos authentication is limited to Java clients only.

An explanation of how to implement Kerberos security in VoltDB will be added to the security chapter of the Using VoltDB manual shortly after version 4.3 releases.

This release adds a new example application, windowing, to the VoltDB kit. This new application demonstrates how to perform two common tasks for real-time, big data applications:

The sample code demonstrates techniques for performing these common tasks using VoltDB's builtin functions and capabilities to achieve maximum throughput with minimal interruption to ongoing data ingestion.

In previous versions, the length of a VARCHAR column was defined in bytes. Starting with 4.3, the length of a VARCHAR column is now declared in characters rather than bytes. This change is made for compliance with the SQL standard and for improved handling of multi-byte UTF-8 character sets.

Three major effects of this change are:

The impact on strings defined as less than 16 characters results from short VARCHAR columns being stored inline as their maximum possible length. Where previously VARCHAR(15) would consume 15 bytes, now it will consume four bytes for every character, or 60 bytes. For longer VARCHARS, the strings are stored in pooled memory as their actual length, so there is no change to the memory they require.

The increased memory consumption, especially for schemas with many of short VARCHARS, could impact the ability to restore snapshots created in previous versions of the product, if memory usage is an issue.

It is possible to reproduce the previous behavior in VARCHAR declarations by including the keyword BYTES. For example VARCHAR(64 BYTES).

Two changes have been made to simplify the process for configuring and starting VoltDB clusters.

Several new SQL string functions have been added to simplify coding:

The @StopNode system procedure let's you stop the VoltDB process on an individual member node of a VoltDB cluster in an orderly way. You specify the host ID of the node you want to stop as an argument to the @StopNode procedure.

You can use @StopNode to remove a node from the cluster for hardware upgrades or other maintenance, then return the node to the cluster with voltdb rejoin. Note that the @StopNode procedure only works if the cluster is K-safe and stopping the node will not stop the cluster itself. In other words, the cluster must remain viable after the system procedure executes. You cannot use @StopNode to shutdown the cluster.

Improvements continue to be made to the JDBC interface, focusing on reliability and extended functionality. This release fixes an issue where the connection would fail if the connection string included an inaccessible server.

This version also includes a number of changes to SQL parsing to ensure correctness and proper index usage, specifically in edge cases related to complex joins and order by clauses.

The csvloader now provides additional context when reporting errors in the input file, making it easier to debug and correct the errors by identifying the specific input field that causes the error.

In addition to the preceding new features and enhancements, a number of known issues have been corrected, including:

In previous releases, the server process would not start with a catalog compiled by an earlier version of VoltDB. Starting with 4.2, when you create a new database using the voltdb create command and an old catalog, VoltDB automatically recompiles the catalog before starting the server.

This means you can upgrade VoltDB versions without manually recompiling the catalog. The same is true when updating the catalog on a running database with the voltadmin update command or @UpdateApplicationCatalog system procedure. Note, however, using older catalogs does not currently work with the voltdb recover command or the VoltDB Enterprise Manager.

If the catalog is old enough to contain outdated or no longer valid schema syntax, VoltDB reports an error and either stops (in the case of voltdb create) or cancels the update (in the case of voltadmin update). In this case, you must update the schema source file and recompile the catalog yourself.

The VoltDB compiler now supports a new constraint, LIMIT PARTITION ROWS, that lets you limit the size of individual tables. The LIMIT PARTITION ROWS constraint is declared in the CREATE TABLE statement and limits the number of rows that can be inserted into any partition for that table. See the Using VoltDB manual for details.

The SELECT statement now supports subqueries as table references in the FROM clause. For the initial release subqueries have certain constraints:

See the documentation of the SELECT statement in the Using VoltDB manual for details.

Previously, you could specify a port number for each port when starting VoltDB and a separate network interface for internal versus external ports. It is now possible to specify both the interface and the port number for individual ports when starting VoltDB. For example, the following command specifies the network interface 15.16.2.24 and the port number 21212 for the client port but the default external interface and port 31313 for the admin port.

$ voltdb create voter.jar --client=15.16.2.24:21212 --admin=31313

Note that the --internalinterface and --externalinterface flags are still available and set the default interfaces, as before. When you specify both a default interface for a collection of ports and a specific interface for an individual port, the port-specific setting overrides the default setting. It is also now possible to specify the http port (and, optionally, its interface) on the command line using the --http flag.

The interface to control the rebalance operations when nodes are added to an elastic cluster have changed from using Java system properties to elements and attributes in the deployment file. You can now adjust the length and size of rebalance operations using the attributes duration and throughput of the <elastic> element in the deployment file. These attributes replace the Java system properties ELASTIC_TARGET_TRANSFER_TIME_MS and ELASTIC_TARGET_THROUGHPUT. For example:

<deployment>
   . . .
   <systemsettings>
       <elastic duration="15" throughput="1"/>
   </systemsettings>
</deployment>

See the section on "Configuring How VoltDB Rebalances New Nodes" in the Using VoltDB for details.

Previously, elastic rebalancing would occupy all partitions in the cluster. With this release, each rebalance operation only uses those partitions it needs, freeing up any remaining partitions for other database transactions. This change does not improve the performance of the rebalance itself, but can significantly reduce the impact rebalance has on ongoing client transactions.

This release contains a number of improvements to reduce latency spikes during operational activities such as snapshots, rejoin, and export. In addition, Java heap usage during export and rejoin has been reduced.

This release includes a number of improvements to the JDBC interface, including extended support for returning metadata, the SetFloat() method, and automatic conversion of numeric values to strings for VARCHAR columns.

In extreme cases, where the disks used for command logging cannot keep up with the write requests coming from VoltDB, the logging packets start to fill up memory. If this condition persisted, it could result in the server process running out of Java heap space. Command logging now includes a back pressure mechanism that will slow the processing of VoltDB transactions if the command logs begin to back up, as a way to avoid this situation.

Previously, the @Shutdown system procedure (and voltadmin shutdown command) simply stopped the database process. The result was similar to a node failing, generating error messages and crash logs as the nodes stopped. User-requested shutdown is now handled as a synchronized event within the cluster, eliminating misleading error messages and unnecessary log files. This is the first step in an effort to provide a more orderly shutdown behavior.

In addition to cluster shutdown improvements, now when you stop a server process with CTRL-C (or, more specifically, the Unix signal SIGINT), the cluster performs a more orderly removal of that node from the cluster, rather than handling the event as an unexpected crash. In this way, removing a server from a K-safe cluster (for maintenance or replacement) is faster and less disruptive of ongoing transactions.

Previously, the csvloader utility provided good performance on small and medium-sized systems. But performance would drop off on clusters with a large number of partitions. This bug has been corrected and csvloader provides scalable performance for loading partitioned tables into different size clusters.

In addition to the preceding new features and enhancements, a number of known issues have been corrected, including:

There was an issue in earlier releases where VoltDB could miscalculate the outstanding transactions. Two consequences of this situation were that the database server would issue an "admission control" error stating that there was a negative outstanding transaction byte count or client applications encountered connectivity issues. This problem has now been corrected.

In previous releases, if all servers in the cluster failed and rejoined without stopping the database itself, data waiting in the export queue could be lost. This only happened if all servers in the cluster were cycled (stopped and rejoined). The cause of this problem has now been corrected.

When starting the VoltDB server process from the command line (using the voltdb create, add, recover, or rejoin command) you can use the -B or --background flag to specify that the process run in the background.

When the VoltDB server does not receive a response from a client connection for a set amount of time, the server times out and closes the connection. The client timeout period has been extended from 4 to 30 seconds so connections are more resilient to network issues.

In recent releases, performance of the csvloader utility was improved by introducing batch mode. However, batch mode inserts were not immediately recognized or recorded by the command logs. As a result, data loaded using csvloader batch mode did not become durable until the next snapshot occurred. This issue has been resolved and all data loaded with csvloader is now immediately durable.

TRUNCATE TABLE, and its equivalent statement DELETE with no WHERE clause, have been optimized to significantly improve performance and reduce the amount of memory used during execution.

In early releases of V4, when specifying join order for a query, the table names had to be in all uppercase. This issue has been resolved and the table names are no longer case sensitive.

There was an issue where attempting to restore a snapshot on a cluster with a large number of tables and partitions could fail, reporting an error that the "next message length" was too long. This problem has been resolved.

It is now possible to declare complex stored procedures inline in the schema using the CREATE PROCEDURE AS statement and embedded Groovy code. See the Using VoltDB manual for details..

A number of issues have been discovered and resolved in the elastic cluster functionality that is introduced in version 4.0. In particular, edge cases related to error conditions when nodes fail during elastic scaling have been identified and corrected.

This release includes several improvements to the Java client, including:

As a consequence of this last change, the method ClientResponse.getException() has been removed from the client API. Also, the causedBy property of ProcCallExceptions no longer returns an exception, All underlining exception information is returned as text by the getStatusString() method.

A number of improvements have been made to the JDBC interface as well. Similar to the Java API, the JDBC interface is provided as a single JAR file with no external dependencies. This means that if you use Guava and depended on the Guava library provided by VoltDB, you must either provide your own Guava JAR file or change the dependency to "com.google_voltpatches.common.*".

Other improvements to the JDBC interface include:

VoltDB now supports the CASE-WHEN-THEN-ELSE-END syntax in queries. For example:

SELECT Prod_name, 
    CASE WHEN price > 100.00 
          THEN 'Expensive'
          ELSE 'Cheap'
    END 
FROM products ORDER BY Prod_name;                      

VoltDB now supports the use of aggregate functions in the HAVING clause. For example:

SELECT game_id, count(*) FROM games
  GROUP BY game_id
  HAVING count(*) > 1;

The default Java heap size for the VoltDB server process has been increased from 1GB to 2GB. The new default more closely matches recommended settings for general purpose usage, More detailed recommendations can be found in the revised "Server Process Memory Usage" section of the VoltDB Planning Guide.

In previous versions there was an issue where, if a cluster was reduced in size and then restored from snapshots, future command logs of the cluster could not be recovered. One symptom of this issue is the fatal error "No viable snapshots to restore" during recovery.

The problem only occurs if the number of unique partitions in the cluster was reduced, either by reducing the number of servers, reducing the sites per host, or increasing the K-safety factor. With this release, the issue has been corrected. The issue is resolved for any affected databases by following the instructions for upgrading in the previous section; specifically, saving a snapshot, upgrading the software to VoltDB 4.0.1 or later, then restoring the snapshot.

VoltDB 4.0 is a major release. It consolidates and completes many features introduced in preceding releases, including elasticity and changes to the user interface to improve overall ease of use of the product. Benefits of VoltDB 4.0 include:

In addition to the new features and enhancements listed above, VoltDB V4.0 includes fixes to a number of limitations in previous versions, including the following:

To ensure complete and accurate restoration of a database, recovery using command logs can only be performed to a cluster with the same number of unique partitions as the cluster that created the logs. If you restart and recover to the same cluster with the same deployment options, there is no problem. But if you change the deployment options for number of nodes, sites per host, or K-safety, recovery may not be possible.

For example, if a four node cluster is running with four sites per host and a K-safety value of one, the cluster has two copies of eight unique partitions (4 X 4 / 2). If one server fails, you cannot recover the command logs from the original cluster to a new cluster made up of the remaining three nodes, because the new cluster only has six unique partitions (3 X 4 / 2). You must either replace the failed server to reinstate the original hardware configuration or otherwise change the deployment options to match the number of unique partitions. (For example, increasing the site per host to eight and K-safety to two.)

VoltDB reserves the subfolder "segments" under the command log directory for storing the actual command log files. Do not add, remove, or modify any files in this directory. In particular, do not set the command log snapshot directory to a subfolder "segments" of the command log directory, or else the server will hang on startup.

If a node on the replica database fails while the master is loading data with the csvloader (or its associated bulk loading methods), when the node rejoins it is possible data already loaded gets reloaded during the rejoin. This can cause divergence between the master and replica databases.

To be safe until this limitation is corrected, if a node on the replica database fails while the master database is bulk loading data, you should stop the replica and the DR agent and restart replication once the bulk load is complete.

Using the VoltDB Enterprise Manager, if a replica database was started with command logging, then stopped (intentionally or by accident), the Enterprise Manager cannot restart the database as a normal database using the recover action to reinstate the database's previous state. The Enterprise Manager can restore from a snapshot.

If you want to use the Enterprise Manager to stop a replica and restart it as a normal database, the recommended procedure is:

Note that this limitation is specific to the Enterprise Manager. Failed replica databases can be recovered manually using the command line.

A bug in the Apache Kafka client can result in file descriptors being allocated but not released if the producer.type attribute is set to "sync" (which is the default). The consequence is that the system eventually runs out of file descriptors and the VoltDB server process will crash.

Until this bug is fixed, use of synchronous Kafka export is not recommended. The workaround is to set the Kafka producer.type attribute to "async" using the VoltDB export properties.

VoltDB currently intercepts assertions as part of its handling of stored procedures. Attempts to use assertions in stored procedures for debugging or to find programmatic errors will not work as expected.

The UPPER() and LOWER() functions return a string converted to all uppercase or all lowercase letters, respectively. However, for the initial release, these functions only operate on characters in the ASCII character set. Other case-sensitive UTF-8 characters in the string are returned unchanged. Support for all case-sensitive UTF-8 characters will be included in a future release.

There is a problem with how the math library used to build the C++ client library handles large decimal values on 32-bit operating systems. As a result, the C++ library cannot serialize and pass Decimal datatypes reliably on these systems.

Note that the C++ client interface can send and receive Decimal values properly on 64-bit platforms.

To ensure proper recovery on startup, either from command logs or the last database snapshot, make sure all snapshot files — or at least complete subsets of the snapshot files — are available on the nodes of the cluster. If you delete or move snapshot files (for example, copying all snapshot files to a single node) be sure to keep all of the files for each node together. Do not selectively delete or move individual files or else the recovery may fail.

If the HTTP port is enabled (which it is by default) but the process does not have execute privileges for the /tmp directory, VoltDB throws a fatal exception on startup. The error message indicates that the process could not load the native library for the Snappy web server.

The workaround is to either use an account that has execute permission for the /tmp directory or specify an alternate directory that the account can access by assigning the environment variable VOLTDB_OPTS = "-Djava.io.tmpdir={alternate-tmpdir}".

Normally, manual snapshots (those created with the Take a Snapshot button) are copied to the management server. However, if automated snapshots are also being created and copied to the management server, it is possible for an automated snapshot to override the manual snapshot.

If this happens, the workaround is to turn off automated snapshots (and their copying) temporarily. To do this, uncheck the box for copying snapshots, set the frequency to zero, and click OK. Then re-open the Edit Snapshots dialog and take the manual snapshot. Once the snapshot is complete and copied to the management server (that is, the manual snapshot appears in the list on the dialog box), you can re-enable copying and automated snapshots.

When the Enterprise Manager starts, it unpacks files that the web server uses into a subfolder of the /tmp directory. It does not delete these files when it stops. Under normal operation, this is not a problem. However, if you upgrade to a new version of the Enterprise Edition, files for the new version become intermixed with the older files and can result in the Enterprise Manager starting databases using the wrong version of VoltDB. To avoid this situation, make sure these temporary files are deleted before starting a new version of VoltDB Enterprise Manager.

The /tmp directory is emptied every time the server reboots. So the simplest workaround is to reboot your management server after you upgrade VoltDB. Alternately, you can delete these temporary files manually by deleting the winstone subfolders in the /tmp directory:

$ rm -vr /tmp/winstone*

When upgrading VoltDB Enterprise Edition, please note that the configuration files for the Enterprise Manager are not upwardly compatible. New product features may make existing database and/or deployment definitions unusable. It is always a good idea to delete existing configuration information before upgrading. You can delete the configuration files by deleting the ~/.voltdb directory. For example:

$ rm -vr ~/.voltdb

In the past, it was possible to run two (or more) databases on a single physical server by defining two logical servers with the same IP address and making the ports for each database unique. However, as a result of internal optimizations introduced in VoltDB 2.7, this technique no longer works when using the Enterprise Manager.

We expect to correct this limitation in a future release. Note that it is still possible to start multiple databases on a single server manually using the VoltDB shell commands.

For partitioned tables, the value of the column used to partition the table determines what partition the row belongs to. If you use UPDATE to change this value and the new value belongs in a different partition, the UPDATE request will fail and the stored procedure will be rolled back.

Updating the partition column value may or may not cause the record to be repartitioned (depending on the old and new values). However, since you cannot determine if the update will succeed or fail, you should not use UPDATE to change the value of partitioning columns.

The workaround, if you must change the value of the partitioning column, is to use both a DELETE and an INSERT statement to explicitly remove and then re-insert the desired rows.

If you refer to a table or column name that does not exist, the VoltDB compiler issues the error message "user lacks privilege or object not found". This can happen, for example, if you misspell a table or column name.

Another situation where this occurs is if you mistakenly use double quotation marks to enclose a string literal (such as WHERE ColumnA="True"). ANSI SQL requires single quotes for string literals and reserves double quotes for object names. In the preceding example, VoltDB interprets "True" as an object name, cannot resolve it, and issues the "user lacks privilege" error.

The workaround is, if you receive this error, to look for misspelled table or columns names or string literals delimited by double quotes in the offending SQL statement.

VoltDB opens a file descriptor for every client connection to the database. In normal operation, this use of file descriptors is transparent to the user. However, if there are an inordinate number of concurrent client connections, or clients open and close many connections in rapid succession, it is possible for VoltDB to exceed the process limit on file descriptors. When this happens, new connections may be rejected or other disk-based activities (such as snapshotting) may be disrupted.

In environments where there are likely to be an extremely large number of connections, you should consider increasing the operating system's per-process limit on file descriptors.

VoltDB uses a number of different ports for interprocess communication as well as features such as HTTP access, DR, and so on. Port scanning software often interferes with normal operation of such ports by sending bogus data to them in an attempt to identify open ports.

Over the past few releases, VoltDB has hardened its port usage to ignore unexpected or irrelevant data from port scanners. However, the ports used for Database Replication (DR) cannot be protected in this way. So, in V4.6, a Java property was introduced to allow you to disable the DR ports, for situations where port scanning cannot be avoided. To disable the DR ports, set the Java property VOLTDB_DISABLE_DR to true before starting the database process. For example:

$ export VOLTDB_OPTS="-DVOLTDB_DISABLE_DR=true"
$ voltdb create myapplication.jar \
                --deployment=deployment.xml \
                --host=voltsvr1

Note that, if you disable the DR ports, you cannot use the database as a master for database replication.

VoltDB includes a new utility, voltify, that helps you migrate from an existing MySQL database to VoltDB. The utility connects to a running MySQL database and creates a target schema and starter project in VoltDB to match the source database. If you are interested, you can find instructions for voltify in the VoltDB github repository at the following URL:

http://github.com/VoltDB/voltdb/blob/master/tools/voltify-README.md

We encourage anyone who tries it to provide feedback in the VoltDB forums, http://forum.voltdb.com. Thank you.