Getting Started with dbExpress

by

Martin Rudy
One of the many data connectivity options for the Borland RAD products is dbExpress. This set of drivers and components provide connectivity to databases for the Windows, .NET and Linux platforms. This paper covers an introduction to using the dbExpress technology in both Win32 and .NET. NOTE: Not all of the material in this paper will be covered in the Conference session. The intent is to provide more detail in this paper that can be referenced during the session.

Contents
The major topics covered are:

Data Connectivity Overview dbExpress Introduction Basic Data Connectivity Sending Updates to a Database The Basics Reconciling Errors Master / Detail NestedDataSets Setting UpdateMode and ProviderFlags properties DataSetProvider Properties and Events Empty Field List for Update Statement Summary

Data Connectivity Overview

Contents Borland RAD tools provide connectivity to a variety of data file formats. You can connect to virtually any type of data if a driver is available for the data format. dbExpress is a set of lightweight, SQL drivers for InterBase, Microsoft SQL Server, MYSQL, Oracle, SQL AnyWhere and DB2. A basic understanding of the foundational concepts of data connectivity is essential before efficiently using many of the database features. These concepts include the definition of terminology and the basic steps to display data in forms. To view and modify data, you need to define the source of the data you intend to use, how to display

the data at runtime, and how to connect the two. This is done using a DataSet, DataSource, and data-aware controls.

Figure 1: Basic requirements to display data in a form Figure 1 shows the dependencies between a form and the data displayed in the form. The data files are defined using a DataSet component. The data can be in a table, the result of a query, or defined by a stored procedure in a SQL database server. The form contains data-aware components that display the data. A DataSource is the component used to connect data-aware components to DataSet components.

dbExpress Introduction
Contents dbExpress is a cross-platform, database-independent set of drivers and components that provide high-performance and a small footprint. It consists of database connectivity technologies introduced in Delphi 6 and Kylix and is one of the data connectivity options in Delphi 2005 for both Win32 and .NET applications. For .NET applications, it is called dbExpress.NET which is a .NET version of the same capabilities found in Delphi 7 dbExpress. Connecting to data requires the database to be identified, the data table(s) and columns selected and a request for the defined data. The major difference between dbExpress and the other RAD data connectivity options is dbExpress only supports unidirectional, readonly cursors for data retrieval. This means no data is buffered and some of the features available in the other data connectivity options are not applicable. The following are some of the features not available with unidirectional cursors: Forward movement only, cannot use Last and Prior methods No editing of data

No filter support No batch moves No heterogeneous queries No cached updates No lookup fields

At first glance you might think dbExpress is not for you, but wait, it is really a more efficient way to retrieve and update data. The technique used by dbExpress to manipulate data is the same as building multi-tier applications using DataSnap (formerly known as MIDAS) you use ClientDataSets. The light-weight, fast dbExpress drivers quickly retrieve data from the database and ClientDataSets provide the ability to modify data, change sort orders, and maintain a change log, support lookups, filtering, plus much more. Using dbExpress as it is designed requires you to perform a fetch of only the data that users need to immediately use, not the entire table. No longer will a SELECT * FROM tablename be sufficient (unless the result set is not too large). ClientDataSets are in-memory tables, all data retrieved, as the result set is stored in the local workstations memory. This basic architecture is called provide/resolve. In Win32, two ini files are used to store information about drivers and database connections. The installed driver types are contained in dbxdrivers. For each driver installed, the required libraries (DLLs for Windows and shared objects in Linux) are listed along with default connection parameter settings. The second ini file used, dbxconnections contains the named configuration connection sets. The components available in dbExpress and dbExpress.NET are essentially the same. Figures 2 and 3 show the Component/Tool Palettes available in Delphi 7 and Delphi 2005. Table 1 lists the components on the dbExpress component palette page and a brief description.

Figure 2: dbExpress components in Delphi 7

Figure 3: dbExpress components in Delphi 2005 Table 1 dbExpress components Component Name Description SQLConnection Defines a connection to a database, similar to TDatabase SQLDataSet General-purpose unidirectional dataset that executes the SQL statement defined by the CommandText property. This can be a SELECT statement that returns a dataset, a SQL statement that does not return data or executes a stored procedure. SQLQuery Supports SQL statements to be executed that return a unidirectional result set or update data or database schemas. SQLStoredProc Executes a stored procedure. If there is a result set, it is unidirectional. SQLTable Provides unidirectional access to a database table. SQLMonitor Used to intercept and display messages passed between a SQLConnection and database. SimpleClientDataSet Combines a SQLDataSet and DataSetProvider internally in the component to support data cached in memory. The SQLConnection is where the database connectivity is defined. One of the next four components in the list is used to specify the data to be retrieved if you intend to use a unidirectional cursor. If data is to be edited and browsed, the SimpleClientDataSet needs to be used, or a ClientDataSet component which is covered later. In the next section the basics of retrieving data using dbExpress is covered.

Basic Data Connectivity

Contents

With dbExpress you start by using a SQLConnection component. The first step is to define the database connection. This can be done using an existing connection definition, creating a new connection definition, or using the Params property of the SQLConnection to define a connection dynamically. We will start using an existing definition. The ConnectionName property is assigned the name of an existing definition. Connection definitions are stored in an ini file named dbxconnections. This file stores the connection configurations settings and specifies which dbExpress driver to use. The available drivers are maintained in an ini file named drivers. This file contains the DLL or SO name required for connection and the default settings for all the connection parameters. Setting the ConnectionName property can be done by selecting an entry in the drop-down or double-clicking on the SQLConnection component which displays the dbExpress Connection Editor. This editor is essentially the same in both Delphi 7 and 8 as shown in Figures 4 and 5.

Figure 4: dbExpress Connection Editor dialog Delphi 7

Figure 5: dbExpress Connection Editor dialog Delphi 2005 In this dialog you select the database drive, connection name and modify properties used to connect to the database. The Driver Name combo box specifies which connections are displayed in the Connection Name list box. By default, all defined connections are listed. Changing Driver Name value reduces the Connection Name listing to only those for the selected driver. The Connection Settings vary depending on the driver selected. Any of the entries in the Value column can be modified. You cannot enter new Key values or delete any of the existing ones.

Selecting component to retrieve data

There are four main unidirectional dataset components on the dbExpress component palette: SQLDataSet, SQLQuery, SQLStoredProc and SQLTable. The last three are similar to the BDE, ADO and InterBase Express components. The SQLDataSet component is a general-purpose component and is the recommended component to use. All four components have a SQLConnection property that links to a SQLConnection component. The SQL statement to execute for a SQLDataSet is defined in the CommandText property. You can enter or paste an SQL statement into the property or click the ellipses button to display a CommandText Editor dialog to select from the list of tables and fields. Figure 6 shows a completed dialog selecting all rows and columns from the CUSTOMER table.

Figure 6: CommandText Editor in Delphi 7 and Delphi 2005 One of easiest components to use for displaying data is the DBGrid. It is located on the Data Controls tab. If you attempt to connect a DataSource and a DBGrid to a unidirectional dataset, you get an exception raised with the text Operation not allowed on unidirectional dataset. A grid requires the ability to scroll forwards and backwards through the dataset, which cannot be done with a unidirectional dataset. If you connect other data-aware components (e.g. DBEdit), the exception is not raised until you attempt to go to the prior or last record in the dataset.

Using the SimpleClientDataSet component in Delphi

The SimpleClientDataSet component is a single component to use for bi-directional movement through a result set. This component uses an internal SQLDataSet and TDataSetProvider for retrieving data and also supports data modification. The DataSetProvider creates datapackets based on the SQL statement defined by the CommandText property and used by the SQLDataSet. The datapackets are cached in memory. When edits are sent back to the database, the DataSetProvider processes datapackets containing just the modified records; it creates the appropriate SQL statements that are sent to the database. Any error records returned from the database are packaged by the DataSetProvider and made available for review in the SimpleClientDataSet. This provide/resolve cycle continues until the records are accepted by the database or the edits are canceled. The SimpleClientDataSet requires the same two basic pieces of information as the SQLDataSet which database connection and the SQL statement. The Connection property is used to identify the SQLConnection to use. Alternatively, you can use the ConnectionName property to select from the connection list. This will create an internal SQLConnection component. The SQL statement is defined in the same manner as the SQLDataSet component using the CommandText property. Figure 7 shows a basic setup in design mode.

Figure 7: Basic dbExpress app in design mode Retrieving data is done when the Active property of the SimpleClientDataSet is set to True. This can be done at design time or when the form is created. In addition to issuing SimpleClientDataSet.Active := True, the Open method (SimpleClientDataSet.Open) can be used on the SimpleClientDataSet which is the equivalent of setting the Active property to True. When Active is set to True, if the SQLConnection has not opened access to the database, it will initiate a connection, attempt to set its Connected property to True. The SQLConnection component also has a LoginPrompt property. By default this property is True, indicating a login dialog will be displayed where a user name and password can be entered before attempting to connect to the database. You can also include those values in the connection parameters and remove the login dialog by setting the LoginPrompt property to False.

Limitations of SimpleClientDataSet
The SimpleClientDataSet is great for quick demos and basic data retrieval. Beyond these two uses, there are limitations that may prevent you from fully using the power of dbExpress. Properties and events of the internal components are not surfaced. This prevents you from creating multi-tier applications, create nested datasets, utilize any of the DataSetProvider features and if an internal SQLConnection is used it cannot be shared with other datasets. A more appropriate approach is to use a ClientDataSet (CDS) and DataSetProvider.

Connecting to data using a CDS and DSP

Both the CDS and DSP components are located on the Data Access component palette. A SQLDataSet component from the dbExpress palette is used to define the result set. The SQLConnection property of the SQLDataSet is assigned to the SQLConnection component name. The SQLDataSet CommandText property defines the SQL statement. The DataSet property of the DSP is set to the name of the SQLDataSet component. The CDS component has a ProviderName property which is set to the DSP name. The last property value change is to the DataSource DataSet property which should be assigned to the CDS component name. Setting the CDS Active property to True (or using the Open method) retrieves the data. At first glance you might feel this is too much work compared to the SimpleClientDataSet. As we cover more of the CDS and DSP features you gain a better understanding of why you will use a DSP and CDS instead of the SimpleClientDataSet.

Sending Updates to a Database The Basics

Contents Running the application shows you can move through the data, insert, edit and delete data. When you close the application, no data is saved. You need to explicitly save the data using the ApplyUpdates method. This method sends to the provider all inserted, deleted, and modified records in the change log from the CDS. ApplyUpdates is a function that takes a single parameter and returns an integer value. The ApplyUpdates parameter indicates the maximum number of errors allowed before the update process is canceled. A value of -1 indicates any number of errors can occur. Any record that could not be posted is returned to the client in the data packet. When zero is used, the first record that generates an error causes the entire update process to be aborted. No records are removed from the change log; they are all retained by the CDS. Also, no changes are written to the database. Any number greater than zero used for the parameter indicates the maximum number of errors that can occur before the entire update process is aborted. If there are less error records than the value of the parameter, all records that were successfully saved are committed and the error records are still retained by the CDS. The value returned by ApplyUpdates indicates the number of records that could not be posted. This allows you to check the result of an ApplyUpdates statement to conditionally proceed based on the existence or non-existence of any errors. When ApplyUpdates results in records being successfully saved, the Data property is automatically updated. Any error records from the update along with the newly updated records are placed in the updated Data property.

Below is a simple example of how a button can be used to send updates to the database. The value returned by ApplyUpdates is saved in the variable iErrCnt. Since -1 is used as the parameter, there is no limit on the number of errors. All records that can be posted will be committed and all error records will remain in the Data property of the CDS. If the value is > 0 then a message is displayed along with the number of remaining errors.
procedure TfrmMain.pbUpdateClick(Sender: TObject); var iErrCnt: Integer; begin iErrCnt := cdsCustomer.ApplyUpdates(-1); if iErrCnt > 0 then MessageDlg('Problem in apply updates, ' + IntToStr(iErrCnt) + ' error(s).',mtWarning,[mbOK],0); end;

Undoing changes
Changes can be reversed in a CDS to a variety of levels. They include an incremental undo, a record undo, all changes, and refreshing the data currently in the provider. Incremental undo is accomplished using the UndoLastChange method. This method reverts an entire record to it previous set of values. Note, you cannot undo to the fieldlevel, only the record-level. Each call to the method reverts in sequence the previous change to the dataset. There is one parameter used with the UndoLastChange method that is a Boolean data type. When the value is True, the cursor is repositioned on the record that is restored. The cursor remains in its current position when the parameter is False. Below is an example of UndoLastChange where the cursor follows the change.
cdsCustomer.UndoLastChange(True);

Calling RevertRecord can reverse all changes for the current record. This method removes all changes for the current record from the change log. Below is an example for RevertRecord.
cdsCustomer.RevertRecord;

All changes can be canceled using CancelUpdates. When this method is called, every entry in the change log is removed. Before calling CancelUpdates, the ChangeCount property can be used to determine if there are any changes to the dataset. The following is an example of using this property with CancelUpdates.
if cdsCustomer.ChangeCount > 0 then begin if MessageDlg('Are you sure you want to cancel all changes?', mtConfirmation,[mbYes,mbNo],0) = mrYes then begin cdsCustomer.CancelUpdates; cdsCustomer.Refresh; end; end else

MessageDlg('There are no changes to undo.',mtInformation,[mbOK],0);

Refreshing client data

Two methods can be used to refresh the data without closing the dataset: Refresh and RefreshRecord. Refresh returns the current values from the server. An exception is raised if there are any records in the change log. It is best to check for any changes in the log using ChangeCount first, then either cancels all changes or prompt users to okay clearing the change log. RefreshRecord updates the current record to those currently in the provider. RefreshRecord also carries a warning when there have been updates to the record. Update conflicts can exist which means no reconcile error will occur even when a conflict exists. Below is an example of using the Refresh method.
if cdsCustomer.ChangeCount > 0 then begin if MessageDlg('Current change log has not been applied. ' you want to cancel all changes?',mtConfirmation, [mbYes,mbNo],0) = mrYes then begin dmMain.cdsCustomer.CancelUpdates; dmMain.cdsCustomer.Refresh; end else pbUpdate.SetFocus; end else dmMain.cdsCustomer.Refresh; Do ' +

In this example, a preemptive test is done before calling Refresh. If there are any changes in the log, the user is prompted to cancel all changes before refreshing the data.

Limiting record display based on update status

Each record that is modified, inserted and deleted gets marked with a status value. The StatusFilter property can be set to limit the types of records displayed. StatusFilter can be set to one of more of the values in Table 2. Table 2 StatusFilter options and descriptions Value usUnmodified usInserted usModified Include records that have been inserted. Limit records displayed to only those that have been modified. Two records are displayed for each original record modified: the original record along with a second record containing all modifications for each Description Record has not been modified.

posting. usDeleted Show records that have been deleted.

StatusFilter can be an empty set, one of the values, or multiple values. When StatusFilter is an empty set, all records are displayed. If more than one value is used, the combination of the options included in the set determines which records are displayed.

Reconciling Errors Using the Built-in Dialog

Contents Another aspect of data entry is correcting records that cannot be saved. There are many reasons for error records. Application validation, business rules, database constraints, and data modified by another user are some of the main reasons. Reconciling these types of errors can be handled in the application, the database, or a combination of both. This next section introduces the basics of a built-in feature to provide error reconciliation in the application. The Object Repository contains a dialog box that provides the basic reconciliation code already completed. Figure 8 shows the entry on the Dialogs tab in the Object Repository for Delphi 7. The Object Repository for Delphi 2005 is shown in Figure 9.

Figure 8: Reconcile Error Dialog option in the Object Repository for Delphi 7

Figure 9: Reconcile Error Dialog option in the Object Repository for Delphi 2005 .NET projects Adding the Reconcile Error Dialog to a project requires you to do three things. First, the unit for the dialog must be removed from the list of auto-created forms for the project, which is done automatically for you when you add the dialog to a project unless you have turned off the auto create feature. Second, you must modify the OnReconcileError event of the ClientDataSet component. Below is an example of the minimum code for the event. The last step is to add the unit for the dialog to the form or data module where the CDS component resides. In the sample code the uses clause in the data module is updated.
procedure TdmMain.cdsCustomerReconcileError(DataSet: TCustomClientDataSet; E: EReconcileError; UpdateKind: TUpdateKind; var Action: TReconcileAction); begin Action := HandleReconcileError(DataSet, UpdateKind, E); end;

The Reconcile Error Dialog is displayed after a call to ApplyUpdates when one or more error records exist. Each record that cannot be posted is displayed in the dialog box, one error at a time. Figure 10 shows an example when there is an attempt to save a record and another user has already made a modification to the record.

Figure 10: Reconcile Error Dialog at runtime The dialog shows errors for records you have inserted, modified and attempted to delete. The Update Type label indicates which of the three actions you took on the record. The grid has a maximum of three columns. They are the value you are attempting to save (Modified Value), the value of the field as it is currently in the database (Conflicting Value) and the value when the client application originally received the data packet (Original Value). The grid changes display based on the type of problem, the update action, and which of the two check boxes at the bottom of the dialog are checked. The first checkbox, Show conflicting fields only, limits the records in the grid to only those fields that have been changed and are in conflict with current database values. Figure 10 shows an example of this selection. If the Show changed fields only checkbox is checked, the fields in all records that have been changed are listed. If there is no conflicting value, the Conflicting Value column contains <Unchanged>. If you attempt to delete a record that has already been deleted by another user, only the Modified Value column is displayed. The radio buttons in the upper right provide the type of action you can with the error record. Table 3 contains a description that is directly from the Delphi Help system. Table 3 Action options in the Reconcile Error Dialog Value Skip Description Skips updating the record that raised the error condition, and leaves the

unapplied changes in the change log. Abort Merge Correct Cancel Refresh Aborts the entire reconcile operation. Merges the updated record with the record on the server. Replaces the current updated record with the value of the record in the event handler. Backs out all changes for this record, reverting to the original field values. Backs out all changes for this record, replacing it with the current values from the server.

The action for each error can be set before clicking the OK button. Based on the type of error, you can determine to leave the record in the change log, cancel it, merge the updated record, or abort the entire reconcile operation.

Master / Detail - NestedDataSets

Contents There are two options for creating master/detail forms using ClientDataSets. You can use the option of linking the results of two datasets after the data is retrieved using the MasterSource and MasterFields properties or define the master detail relationship with SQLDataSets and use the feature called nested datasets. The second option is what will be described next. The major difference with nested datasets over using the MasterSource and MasterFields properties is a single DSP component encapsulates both the master and detail data. The detail data is actually a column within the master dataset. Another feature of nested datasets is all updates are wrapped in a single transaction. Figure 11 shows the form in design mode.

Figure 11: Nested dataset master / detail setup in design mode The first step to setup up master / detail relationship is to select two unidirectional dataset components. In this example, two SQLDataSet components are used to define which data to retrieve. A DataSource component is also needed to link the two datasets. The first SQLDataSet (named sdsCustomer) retrieves all the records from the Customer table using select * from Customer as the SQL statement. The DataSource component (named dsCust) uses the SQLDataSet for the Customer data as its DataSet. The second dataset (named sdsSales) is used for the detail data from the Sales table. To link the two SQLDataSet components, the DataSource property of the Sales SQLDataSet component is used along with the WHERE clause of the SQL statement. The DataSource property is set to dsCust, the name of the DataSource component for the Customer table. This defines the dataset to use as the master data. The linkage between the two tables is defined in the SQL property using the WHERE clause as follows: SELECT * FROM Orders WHERE CUST_NO = :cust_no In this example, all of the columns from the Sales table are selected. The rows are limited to the value in the cust_no parameter. The key to using this technique is the parameter names must match the names of the linking fields in the master table. In this example, CUST_NO is the field name in both tables. Therefore, :cust_no is used as the parameter in the WHERE clause. Lowercase characters are used to visually seeing a difference between the field name and parameter name, but they are not required. The value of

:cust_no is automatically updated each time the master record changes which in turn generates a new SQL statement and the appropriate detail data is returned. Viewing the data with ClientDataSets only requires a single TDataSetProvider. It is linked to the SQLDataSet for the master table. No provider is required for the detail data. The data for the Sales table will automatically be made available by the nesting of the detail data as a separate field for each master record. The major difference in setting up master / detail using nested datasets is in setting the ClientDataSet properties. Two CDS components are required. The first ClientDataSet, used as the master table, is linked directly to the provider. An additional step is required to create TField descendants for the fields in the master dataset. When this is done, a new field is listed that does not exist in the Customer table. Figure 12 shows the TFields Editor for the Customer CDS. The additional field contains the detail records for the Sales table. It is a TDataSetField descendant. The name of this field is the same as the name of the SQLDataSet.

Figure 12: Fields Editor dialog box A second CDS is used for the detail data. Instead of linking the CDS used for the detail data to a provider, the DataSetField property of the ClientDataSet is used. This property is set to the TDataSetField instantiated in the master data Fields Editor. When both CDS components are opened the data for the Customer and Sales are displayed. Figure 13 shows an example of using nested datasets in a DBGrid.

Figure 13: Visual display for nested datasets In the DBGrid the column for the detail dataset is displayed similar to memo and graphic data. The text (DATASET) represents the values for each record in the dataset. An ellipsis button is displayed with a double-click or F2 keypress. Clicking the button displays another window containing the detail data displayed in a grid. This window will always stay on top. You can also see in Figure 13 that the bottom grid in the form displays the same data. You are not required to include the DataSetField in the master table. The TField Visible property can be set to False and, using a second CDS and DataSource, all detail can be displayed in a grid. Another difference when ClientDataSets are used instead of SQLClientDataSets is in saving the data. The Save Customer button on the form is the same as in example projects using SQLClientDataSets. When changes are made here and the ApplyUpdates is executed, all modifications for both the master and detail data sets are wrapped into a single transaction and committed or rolled back based on the parameter passed in the ApplyUpdates method.

Setting UpdateMode and ProviderFlags properties

Contents Control of the type of optimistic locking that is used in an application is established using the dataset UpdateMode property. The UpdateMode setting dictates the fields in the dataset used to find the original record. Specifically, this property sets what fields are used in the WHERE clause of the UPDATE and DELETE statement generated by the provider. If the record is not found with the same values as when the record was originally read, the update fails and an exception is raised. Where you set this property is based on which components you use. Using ClientDataSets, the UpdateMode property is on the DataSetProvider. If you use one of the combination CDS/DSP components (e.g. SimpleClientDataset), those components have an UpdateMode property because of the internal DataSetProvider. Table 4 lists the three values for UpdateMode and their description. Table 4 UpdateMode property settings Value upWhereAll upWhereChanged upWhereKeyOnly Description Every column in the record is used to find the record. This is the Delphi default. Only the columns in the record being edited that have changed are used to find the record. Only the columns that define the key are used to find the record.

Careful consideration should be given to the value selected for the UpdateMode. Each setting has its good and bad side. The default, upWhereAll, is the most restrictive and has the worst performance. Using all the fields in the WHERE clause to locate the original record ensures the record has not been changed since it was originally read. If you get an error message stating the query is too complex when you attempt an update, the SQL database is indicating it cannot handle a WHERE clause with all the fields in the table being used. The upWhereKeyOnly is the least restrictive. This allows anyone to change any field (except the primary key) without consideration of what the original values where. In between these two property options is the upWhereChanged. This can be a problem where the values in multiple fields taken together have a specific meaning, like in a multi-field primary key where users are allowed to change the values. Having a good database design and a well thought out set of business rules assist in determining the correct UpdateMode setting. Testing this feature can be done by changing the UpdateMode property and running two instances of an application making modifications in each. The ProviderFlags properties for TField components are also available to use in determining how an update is to be processed. Table 5 lists the possible options.

Table 4 UpdateMode property settings Value pfInUpdate pfInWhere pfInKey pfHidden Description Field is included in update Field is used in finding the original record to be updated Field is used in finding the current record after an update fails Field is included in the data packet to ensure uniqueness of the record. The field is used to find the original record to update. The field is not visible to the application.

Each field in the dataset used by the provider can set which of the ProviderFlags are applicable. Changing the ProviderFlags modifies the SQL UPDATE and DELETE statements created by the provider to change and remove records from the database. To demonstrate the impact on updating data and the usage of the UpdateMode and ProviderFlags properties, the following SQL statement will be used with the InterBase Employee database. This example is in the ProviderFlags projects which is using dbExpress.
SELECT CUST_NO, CUSTOMER, CONTACT_FIRST, CONTACT_LAST, PHONE_NO, ADDRESS_LINE1, ADDRESS_LINE2, CITY, STATE_PROVINCE, COUNTRY, POSTAL_CODE, ON_HOLD, ((CITY || ',') || STATE_PROVINCE) AS CITYSTATE FROM CUSTOMER

All of the fields from the Customer table are returned and a calculated field combining the City and State_Province field which is named CityState. Any attempt to modify an existing record or delete a record generates an error with the message Column unknown CITYSTATE. The problem is in the WHERE clause that is generated. Below is the SQL generated for deleting a record. The dbExpress SQLMonitor is used in the project to track SQL statements.
delete from CUSTOMER where CUST_NO = ? and CUSTOMER = ? and CONTACT_FIRST = ? and CONTACT_LAST = ? and PHONE_NO = ? and ADDRESS_LINE1 = ? and ADDRESS_LINE2 is null and CITY = ? and STATE_PROVINCE = ? and COUNTRY = ? and POSTAL_CODE = ? and

ON_HOLD = ? and CITYSTATE = ?

The WHERE clause includes all fields listed in the SELECT statement, including CITYSTATE. This DELETE will always fail because of the calculated field, CITYSTATE is not a field in the base Customer table. Changing the UpdateMode property from the default of upWhereAll to upWhereChanged or upWhereAll prevents this type of error depending on the type of modification performed. Either option will eliminate the column unknown error when modifying an existing record. Below is an example of the SQL generated to update the Contact_First field and UpdateMode is set to upWhereChanged. The WHERE clause includes the primary key (Cust_No) and all fields changed, in this example just Contact_First.
update CUSTOMER set CONTACT_FIRST = ? where CUST_NO = ? and CONTACT_FIRST = ?

Using upWhereChanged when attempting to delete a record still causes the error. This is due to the inclusion of all fields in the WHERE clause. To get a delete to work, you need to set the UpdateMode to upWhereKeyOnly. With this setting, only the Cust_No field will be used in the WHERE clause as shown below.
delete from CUSTOMER where CUST_NO = ?

An alternative to solving the CityState unknown column problem is to use the TField ProviderFlags. The dataset used by the provider are the TFields that need to be changed. Setting all the ProviderFlags to False for CityState prevents its usage in the WHERE clause for any SQL statement. An UPDATE statement where the Contact_First is modified now is as follows:
update CUSTOMER set CONTACT_FIRST = ? where CUST_NO = ? and CUSTOMER = ? and CONTACT_FIRST = ? and CONTACT_LAST = ? and PHONE_NO is null and ADDRESS_LINE1 = ? and ADDRESS_LINE2 is null and CITY = ? and STATE_PROVINCE = ? and COUNTRY = ? and POSTAL_CODE = ? and ON_HOLD is null

The ProviderFlags can also be used to reduce the field used in the WHERE clause, even to the point of specifying just the primary key.

DataSetProvider Properties and Events

Contents The TDataSetProvider (DSP) is the conduit between the database and the client. It connects to a dataset component that provides the data from the database. The DataSetProvider is responsible for packaging the data packets and sending them to the client application when a request is received. It also receives the updates from the client and attempts to post the data. Any error record that is generated is sent back to the client application. The expectation of those attending the session for this paper have a basic understanding of how to use the DataSetProvider with a TDataSet descendant and a TClientDataSet. The remainder of the introduction covers topics that are just beyond the basics of how connect the DSP in an application.

How and Where to Define Constraints

In a multi-tier application there are many places where constraints can be placed client, database server, or the application server. In a two-tier application it has to be at the client or at the server. Each level has its advantages and disadvantages. In you put constraints in the client application, you get instant notification of data problems without server inquiries and you have decentralized the business rules. This means the validation checks are programmed in each application. If you have more than one application accessing the data, the constraints have to be replicated in each client application. This becomes a maintenance nightmare. Most database servers provide some level of constraint definition. This varies from schema definition level to programming domains, rules, triggers and stored procedures that are called to further test data modifications. Constraints defined in the database provide the most centralized set of rules tied directly to the data. No matter how data is modified, these rules are applied. There are two issues that result in the possibility of augmenting server-based rules: cryptic error messages from the servers displayed to users and sometimes business rule logic is more sophisticated than the server programming language can support. In a multi-tier application you have a third option which is to program constraints in the application server. Here you can use the power of DataSnap to implement sophisticated logic for business rules. You can take the errors generated by database server constraints and change the messages sent back to the client, you and also add constraint routines to further validate data sent from the client before committing it to the database server. In a

two-tier application, this same technique can be used but the code resides in the client application. In reality, you will use constraints at all three levels. Some of the database-server level constraints can be passed directly to client applications from the server. You can also have application server constraints available in the client application each time a connection is made. This provides the best of all three levels. For this section, the EMPLOYEE.GDB InterBase database that ships with Delphi is used as the data source. Most of the examples use the SALES table. The server and client projects are found in the BusRulesCnstrnts directory of the code examples.

TField properties automatically applied

Some of the TField properties are automatically passed to the client from the application server. Most of the property values are defined at design time, with one of the properties defined based on the type of TDataSet control used and the tables schema definition. To use the automatic TField property feature, you must create persistent TField objects on the TDataSet component. In this example, a query control for the SALES table has persistent TFields with some of the properties modified as described in Table 5. Table 5 TField properties automatically sent to client application Property ReadOnly Required DefaultExpression Description Indicates the field cannot be modified Value must be placed in the field before the record can be posted Default value for the field for any inserted records Example: San Diego Defines a custom field-level constraint using SQL Where clause type expressions. Any validation rule added here augments the server-level validation rules. Example: x > 10 and x < 1000 The message that is displayed on the client machine when a value does not meet the custom constraint. ConstraintErrorMessage Example: Value must be between 10 and 1,000 All of these TField properties are available in single tier, two-tier, and multi-tier applications, there is nothing built into DataSnap to utilize this capability. This set of

CustomConstraint

TField properties reduces the amount of coding required to support additional constraints not supplied by the database server and allow for ease in setting custom error messages. Error messages displayed based on violation of the properties differ based on the data type and what property is set. The following are a few notes that you should be aware of when using these properties: Required property is set to True when a TTable is used and the database schema specifies a field cannot contain a null. It will be set to False if a TQuery is used as the dataset. Required and ReadOnly properties set on the application server are copied to the client persistent TFields on the initial TField creation in the client. If you change the TField settings for these two fields on the server, you will need to either manually change the values for the CDS TFields or delete and re-add them to the CDS. When the Required property is set to True, you will get one of two error messages displayed when the record is saved and no value exists in the field: 1. Field value required Required property is set to True on the server No persistent TField for CDS or field allows null values and you manually set Required to True 2. Field <fldName> must have a value Field cannot contain null values as defined in table create Unless the Required property is explicitly set to False in the CDS If the Required property is set to True on the CDS, but the table definition allows nulls or the server TField has Required set to False If you want field-level validation, you need to set the CustomConstraint property. The message displayed will be the value of the ConstraintErrorMessage property.

Additional TField properties used for constraints

The Options property on the TDataSetProvider supports the passing of additional TField properties. Setting poIncFieldProps to True adds to the data packet TField properties

specified in the DataSet property of the TDataSetProvider. The following are the TField properties that can be used and are passed to the client based on the TField data type:
o o o o o o o o o o

Alignment Currency DisplayFormat DisplayLabel DisplayWidth EditFormat EditMask MaxValue MinValue Visible

Using TDataSetProvider properties

In addition to including TField properties, the TDataSetProvider Options property also includes the ability to set constraints for the entire dataset. Table 6 lists the options and a brief description. Table 6 TDataSetProvider Options property settings for dataset-level constraints Property poReadOnly poDisableInserts poDisableEdits poDisableDeletes Description Read-only result set Client cannot insert records Client cannot edit records Client cannot delete records

The poReadOnly setting prevents any modifications to the data. No visible indicator or message is given to the user; the data just cannot be modified. When poDisableInserts, poDisableEdits, or poDisableDeletes are set to True, an exception is raised when users attempt to insert, modify, or delete records. The exception message that is displayed contains the name of the CDS and Inserts are not allowed for Inserts, Modifications are not allowed for edits, and Deletes are not allowed for deletes. If you do not want to have the exception dialog displayed, you need to trap for the exception and change how the error is displayed. For example the OnEditError or OnDeleteError events can be used for edits and deletes. In the client application, there

are events for the OnEditError and OnDeleteError that exist in the code but the events have not been assigned. You can assign them yourself to see how you might trap for the error and display a customized message.

Using TDataSetProvider events

Three DataSetProvider events can be used for constraint. These three are listed in Table 7 with a brief description. Table 7 TDataSetProvider events used to program constraints Event BeforeUpdateRecord OnUpdateData OnUpdateError Description Before each record is applied to the remote dataset Before the provider actually applies updates to the database of the entire dataset received from the client Errors on applying updates to database server

In this section each of the three events are covered with some simple examples on how they could be used.

Using BeforeUpdateRecord
The BeforeUpdateRecord event is used when you want to validate data on an individual record basis before changes are applied to the database. It can also be used to modify the client data before being saved. Below are the procedure parameters and a brief description of what is passed to the event:

BeforeUpdateRecord(Sender: TObject; SourceDS: TDataSet; DeltaDS: TClientDataSet; UpdateKind: TUpdateKind; var Applied: Boolean);
Sender SourceDS DeltaDS UpdateKind TDataSetProvider the triggered the event the source data data packet from client type of update: ukInsert, ukModify, ukDelete

Applied indicator that you set if you apply the update in the BeforeUpdateRecord code yourself

Some dos and donts for this event: Dont use Edit and Post methods to modify and save data in the record Use TField NewValue property to change the value of a field Use TField OldValue to get original value Use VarIsNull(Field.NewValue) to determine if the field is NULL Use VarIsEmpty(Field.NewValue) to determine if the field has not changed since it was retrieved

In the demo application, the BeforeUpdateRecord is used to validate the order date field. If the order date is not empty, its value is compared to the ship date to ensure the order date is not after the ship date. To simplify the example, only this condition is checked, there is no check to see if the ship date is modified. The code below shows the procedure used.
procedure TBusRulesContstraints.dspSalesBeforeUpdateRecord(Sender: TObject; SourceDS: TDataSet; DeltaDS: TClientDataSet; UpdateKind: TUpdateKind; var Applied: Boolean); begin { NOTE: This example is only checking for changes to Order Date. } if UpdateKind <> ukDelete then { Check to see if Order Date is NULL using VarIsNull } if ((VarIsNull(DeltaDS.FieldByName('ORDER_DATE').NewValue) = False) and { Check to see if Order Date has changed using VarIsEmpty } (VarIsEmpty(DeltaDS.FieldByName('ORDER_DATE').NewValue) = False)) then if DeltaDS.FieldByName('ORDER_DATE').NewValue > DeltaDS.FieldByName('SHIP_DATE').OldValue then raise Exception.Create('Order Date cannot be after ' + 'Ship Date. <BeforeUpdateRecord>'); end;

The first comparison is to see if the action on the record was either an insert or update, there is no need to validate if the record is to be deleted. The UpdateKind parameter is used to see if the action to be taken is not ukDelete (meaning either an insert or update is to be performed). If this is true, the NewValue property of the ORDER_DATE TField is used. NewValue is unassigned when no modification is made to the field. VarIsNull and VarIsEmpty are used to ensure the value is assigned and not null. If this condition is true, the OldValue property of the SHIP_DATE TField is used to compare to the updated ORDER_DATE value. If the order date is after the ship date, an exception is raised. In this demo, OldValue is used because there is no checking to see if the SHIP_DATE field has also changed. In a complete validation, you would need to check to see if both fields changed. OldValue will always be the value that is currently in the client field. You cannot use the Value property. This property will be null when there is no change to the field, therefore the need for the OldValue property.

Using OnUpdateData event

The OnUpdateData event occurs once in the event stream as updates start to be applied in the application server. You have access to all the records in the data packet that is received from the client. Like the BeforeUpdateRecord, you can validate data, but you have the entire dataset, not just one record. Below are the procedure parameters and a brief description of what is passed to the event:

OnUpdateData(Sender: TObject; DataSet: TClientDataSet);

Sender DataSet TDataSetProvider the triggered the event data packet from client

Unlike the BeforeUpdateRecord event, there is no parameter that indicates the type of update. Since you have the entire dataset, you need to look at the status for each individual record. To determine the status for the record, UpdateStatus, a property of TClientDataSet is used. Table 8 lists the possible values for UpdateStatus with a brief description. Table 8 - UpdateStatus options and their description Value usUnmodified usInserted usModified usDeleted Description Record has not been modified. Record is an insert. Record has been modified. Note: this record is the second of a matching pair. The first will have a status of usUnmodified. Record to be deleted.

Before looking at a code example of the OnUpdateData event, the contents of what is passed should be covered. A change log is maintained by the CDS for each insert, update, and delete. The CDS Delta property contains all records in the change log. A separate record is added to the log for each insert and delete. When an existing record is modified, two records are entered in the log. The first record, with a status of usUnmodified, contains all field values for the record before any modification was made. The second record, with a status of usModified, contains only the field values that have changed. All non-modified fields are null in the second record. The CDS Delta property is what the provider receives as the DataSet property in the OnUpdateData event.

In the demo client project, the second tab displays the contents of the change log. Figure 14 shows the log after performing an edit, insert, delete, and a second edit to a different record. The UpdateStatus field does not exist in the data, it is a calculated field used to display the status of each record.

Figure 14: Client form displaying change log The first two records are a matching pair. The first record contains all the values of the record before any changes were made. The second record, with an UpdateStatus of Modified, contains the values for every field in the record that changed. In this example, the SHIP_DATE field is changed from 3/6/1991 to 3/7/1991. The next two records are for an inserted and deleted record. On an insert, any field where data is entered is placed in the change log. For deleted records, the entire original field values are placed in the log. The last two records are again a matching pair. Here two fields are changed, the SALES_REP and the SHIP_DATE. Displaying the change log requires an extra CDS in the application and a small amount of code. The code that is used in the demo client is as follows:
procedure TfrmMain.PageControl1Change(Sender: TObject); begin if PageControl1.ActivePage = tbsDelta then with dmMain do try cdsSalesDelta.Close; cdsSalesDelta.Data := cdsSales.Delta; cdsSalesDelta.Open; except MessageDlg('No delta records exist.',mtWarning,[mbOK],0); end; end;

The CDS cdsSales contains the data from the provider. The CDS for showing the change log is named cdsSalesDelta. When the second tab is selected, the Delta property of cdsSales is assigned to the Data property of cdsSalesDelta. The try except block is used to display a simple message when no modification has been made to the data. To demonstrate the use of OnUpdateData, a similar validation used in the BeforeUpdateRecord on the ORDER_DATE field is replicated. The code for the demo OnUpdateData event is shown below.
procedure TBusRulesContstraints.dspSalesUpdateData(Sender: TObject; DataSet: TClientDataSet); var Old_ShipDate: TDateTime; begin with DataSet do begin First; while not EOF do begin if UpdateStatus = usUnmodified then { In this demo, only the modified records are being evaluated. There will be two records in the data packet for each modification. The first record is the values of all fields in the record before any changes are made. The second record of the pair contains fields that are modified. Any field that did not change is null. } begin { Save the old Ship_Date field from the first record to compare to the modified Order_Date field. } Old_ShipDate := DataSet.FieldByName('SHIP_DATE').NewValue; Next; { Check to see if the modified record has Order Date modified. If so, then validate the difference and raise an exception if invalid. } if not VarIsEmpty(DataSet.FieldByName('ORDER_DATE').NewValue) then if FieldByName('ORDER_DATE').NewValue > Old_ShipDate then raise Exception.Create('Order Date cannot be after Ship Date. ' + '<OnUpdateData>'); end; Next; // Go the next record in the delta packet end; end; end;

A while loop is required to process the entire dataset. You start at the first record and check its status. If it is a non-modified record, you know that this record is the first of a matching pair. The first record contains the original record values; therefore the SHIP_DATE needs to be saved. To get the updated field values, Next is called for the

dataset to move the modified record. If the ORDER_DATE NewValue is not empty, then compare it to the saved SHIP_DATE value. An exception is raised if the order date is after the ship date. In this example, only modified records are checked. If a record is to be deleted or inserted, no validation is done. The while loop forces all records to be checked until EOF or the exception is raised.

Using OnUpdateError event

The provider triggers the OnUpdateError event if an error condition exists in the update of a record. In this event you can process the error, attempt to fix it or change the type of message that is displayed to the client. Below are the procedure parameters and a brief description of what is passed to the event:

OnUpdateError(Sender: TObject; DataSet: TClientDataSet; E: EUpdateError; UpdateKind: TUpdateKind; var Response: TResolverResponse);
Sender DataSet E UpdateKind Response TDataSetProvider that triggered the event temporary dataset to access error record exception object type of update what action to take on the error when the event exits

Like the previous two events, you use NewValue and OldValue TField properties. You also use the CurValue property, which indicates the current value in the database. This allows you to see the currently stored value, the original value the client received, and the updated value the client wants to apply. The exception parameter has a property named OriginalException. It allows you to get the original exception class. If you are using the BDE and the original exception class is EDBEngineError you can use ErrorCode property to get the error code value, otherwise you have to parse the message. It is important to note that you should not change the current record pointer in the OnUpdateError event.

The Response parameter has a different default value depending on the parameter supplied in the ApplyUpdates. If the maximum number of errors allowed is zero, Response defaults to rrAbort otherwise rrSkip is assigned the default. In the demo server application, OnUpdateError is used to change the error messages returned from the database server. With Interbase, there is no specific error number to evaluate for each individual error. You need to look at the error text and determine from it what error is raised. Two examples are used in the demo: INTEG_65 is the error when the ORDER_STATUS field does not equal new, open, shipped, or waiting and INTEG_67 is the error when the order date is after the ship date. The error message General SQL error. Operation violates CHECK constraint INTEG_67 on view or table SALES is the default error message for an order date that is after the ship date. Most users would not be able to identify the problem with the database error message. To improve the information displayed in the error messages, the code below is used for the OnUpdateError event:
procedure TBusRulesContstraints.dspSalesUpdateError(Sender: TObject; DataSet: TClientDataSet; E: EUpdateError; UpdateKind: TUpdateKind; var Response: TResolverResponse); begin if E.OriginalException is EDatabaseError then begin if Pos('INTEG_65',E.OriginalException.Message) <> 0 then E.Message :=('Order Status must be new, open, ' + 'shipped, or waiting <OnUpdateError>') else if Pos('INTEG_67',E.OriginalException.Message) <> 0 then E.Message := ('Order Date cannot be after Ship Date. ' + '<OnUpdateError>') else E.Message := (E.OriginalException.Message + '<OnUpdateError>'); end; end;

The OriginalExcpetion property of the E parameter is first checked to determine if an EDatabaseError occurred. If this is true, the Message property is searched for a unique text string matching each of the database errors where a different error message is to be displayed. Using the Interbase integrity identifiers (INTEG_65 and INTEG_67) a distinction can be made. If any other database error occurs that database error message is displayed. Replacing the contents of the Message property of the EUpdateError parameter fully supports the reconcile error dialog box from the Object Repository.

Empty field list for UPDATE statements

Contents Some application requirements dictate the need to have fields available for input or assignment on the client but the data in the fields do not update the database. These fields can be created on the client side as internal calculated fields or they can be created as calculated fields in the SQL statement retrieving the data. If these fields are the only data that is modified in the CDS, executing ApplyUpdates results in a datapacket sent to the database but there are no fields to place in the UPDATE statement which results in the following error reconcile dialog message using SQL Server: Incorrect syntax near 'se' Running the InterBase DSP_Demo project generates the following error message in a reconcile error dialog: Token unknown - line 2, char -1 where The real issue is the SQL that is generated by the DSP. The first part of the statement is as follows: update Session se where The DataSetProvider attempts to create an UPDATE statement but the SQL statement generated is incorrect and fails to execute. This empty field list generates an exception and causes the update to fail. Even if you exclude the fields using ProviderFlags, the error still persists. The solution to this situation is to use the BeforeUpdateRecord event and cycle through all the fields in the datapacket for each record and exclude the record from generating an update when no change is made to the actual data fields in the table. Additional tables were added to the DBDEMOS database to be used for this and the next example. These tables represent basic data for sessions at BorCon. Figure 15 shows the tables and data types. Table 5 lists each table and how they are used.

Figure 15: Data tables used for final two topics Table 5 Table name and description for example Table Attendee Speaker AttendeeSession TrackType Session SessionRoom RoomList Description Attendee names and id Conference speaker list Sessions selected for an attendee Tracks offered (e.g. Delphi, JBuilder, StarTeam, CalilberRM, etc) Available sessions, title, speaker, room, time, session type Rooms used by each session List of available rooms at the Convention Center

NOTE: The design of these tables was for demonstration purposes only and do not represent the exact relationships required to fully support all possible session, speaker and attendee relationships. For example, each session only supports a single speaker and is assigned to a single track. Only a subset of the data from the Conference was entered, just enough to use in demonstrating the technique being demonstrated. The code for this example is found on the Empty Update tab of the project shown in Figure 16. The requirement here is to display all the sessions and which room or rooms they are going to use in the Convention Center. The RoomList table has RoomNo and RoomName fields. The actual names used are A1, B1, etc. For BorCon, this is a straightforward naming convention. In some hotels/convention centers the names of the rooms are names like Rainer, Whidbey, or other non-numerical names. The intent of the example is to show how you would support the saving of the RoomNo in the data but display the room name in the form.

Figure 16: Empty Update tab The SQL to retrieve the data from the Sessions table and create the calculated field is as follows: SELECT S.*, CAST(NULL as VARCHAR(100)) as RoomNames FROM Session S Once the data is retrieved, the AfterOpen event on the ClientDataSet is used to translate the room numbers into the full room description. The data is placed into the RoomNames field.

Modifications to the rooms used by a session are done by clicking the ellipses button for the row in the grid. This displays a dialog box where the rooms are selected and room name list is placed into the RoomNames field and the room number list is assigned to the RoomIds field. Clicking the Open button on the Empty Update tab retrieves the data. If you immediately click the ApplyUpdates the reconcile error dialog is displayed with the error message Line 1: Incorrect syntax near 'se'. This is caused by the changes created in the CDS AfterOpen. To prevent this error, code needs to be added to the DSP BeforeUpdateRecord. Clicking the Enable BUR checkbox assigns the following code to the DSP.
procedure TdmEmptyUpdate.dspCustomerBeforeUpdateRecord(Sender: TObject; SourceDS: TDataSet; DeltaDS: TCustomClientDataSet; UpdateKind: TUpdateKind; var Applied: Boolean); var bAllowApply: Boolean; i: Integer; begin if UpdateKind = ukModify then begin bAllowApply := False; for i := 0 to DeltaDS.FieldCount - 1 do bAllowApply := bAllowApply or ((not VarIsClear(DeltaDS.Fields[i].NewValue)) and (pfInUpdate in DeltaDS.Fields[i].ProviderFlags)); Applied := not bAllowApply; end; {NOTE: This is from an example by Jeff Overcash on TeamB } end;

It should be first noted that this technique is based on an example from the Delphi forums by Jeff Overcash. Jeff supplied this idea to a question raised on how to create a numeric field that allows entry in the CDS. The basic technique is to cycle through all the fields in the datapacket (DeltaDS) and determines if the field is to be updated (pfInUpdate in the ProviderFlags property) and the NewValue is assigned. If all fields are checked and there are no fields updated, then there is no need to create an update and the Applied parameter is set to True. An alternative to this method is to turn logging off before setting the RoomNames values, modify the data, then turn logging back on. This is done by setting the CDS LogChanges property to False, make the changes, then set LogChanges to True. This disables all logging while RoomNames is assigned. This technique can also be used when the SQL statement used to retrieve data in a one-toone relationship or a many-to-one relationship. It is possible that fields from the one side of the relationship are updated thus cause the same condition as described above. This same technique can be used to determine the base table has no updates but the joined

data has to be updated with an INSERT, DELETE, or UPDATE statement generated by your code.

Summary
Contents dbExpress and dbExpress.NET allow both Win32 and .NET Borland developers to easily connect to many of the most popular databases used today. The power of both the DataSetProvider and ClientDataSet enable custom control over the data retrieval and data modification to support the needs of todays application requirement.