Firebird Documentation IndexFirebird 1.5.6 Release NotesSQL Language Enhancements → Data Manipulation Language (DML)
Firebird Home Firebird Home Prev: Data Definition Language (DDL)Firebird Documentation IndexUp: SQL Language EnhancementsNext: Stored Procedure and Trigger Language (PSQL)

Data Manipulation Language (DML)

(1.5) Expressions and variables as procedure arguments
(1.5) New constructs for CASE expressions
(1.5) SQL99-compliant Savepoints
(1.5) Explicit locking
(1.5) Improved Aggregate Handling
(1.5) ORDER BY clause can specify expressions and nulls placement
(1.5) SELECT FIRST 0.. is Now Valid
Other Firebird 1.0.x Features

Data manipulation language, or DML is the language of query statements, the commands we use to manipulate data, that is, to SELECT FROM, INSERT, UPDATE and DELETE from tables and to EXECUTE PROCEDUREs.

(1.5) Expressions and variables as procedure arguments

Dmitry Yemanov

Calls to EXECUTE PROCEDURE ProcName(<Argument-list>) and SELECT <Output-list> FROM ProcName(<Argument-list>) can now accept local variables (in PSQL) and expressions (in DSQL and PSQL) as arguments.

(1.5) New constructs for CASE expressions

Arno Brinkman

a) CASE

Allow the result of a column to be determined by the outcome of a group of exclusive conditions.

Syntax Pattern

  <case expression> ::=
      <case abbreviation>  | <case specification>

  <case abbreviation> ::=
      NULLIF <left paren> <value expression> <comma> <value expression> <right paren>
    | COALESCE <left paren> <value expression> { <comma> <value expression> }... <right paren>

  <case specification> ::=
      <simple case>  | <searched case>

  <simple case> ::=
    CASE <value expression>  <simple when clause>...
      [ <else clause> ]
    END

  <searched case> ::=
    CASE <searched when clause>...
      [ <else clause> ]
    END

  <simple when clause> ::= WHEN <when operand> THEN <result>
  <searched when clause> ::= WHEN <search condition> THEN <result>
  <when operand> ::= <value expression>
  <else clause> ::= ELSE <result>
  <result> ::= <result expression>  | NULL
  <result expression> ::= <value expression>
        

Examples

i) simple

  SELECT
      o.ID,
      o.Description,
      CASE o.Status
        WHEN 1 THEN 'confirmed'
        WHEN 2 THEN 'in production'
        WHEN 3 THEN 'ready'
        WHEN 4 THEN 'shipped'
        ELSE 'unknown status ''' || o.Status || ''''
      END
  FROM Orders o;
        

ii) searched

  SELECT
      o.ID,
      o.Description,
      CASE
        WHEN (o.Status IS NULL) THEN 'new'
        WHEN (o.Status = 1) THEN 'confirmed'
        WHEN (o.Status = 3) THEN 'in production'
        WHEN (o.Status = 4) THEN 'ready'
        WHEN (o.Status = 5) THEN 'shipped'
        ELSE 'unknown status ''' || o.Status || ''''
      END
    FROM Orders o;
        

b) COALESCE

Allows a column value to be calculated by a number of expressions, from which the first expression to return a non-NULL value is returned as the output value.

Syntax Pattern

  <case abbreviation> ::=
    | COALESCE <left paren> <value expression>
      { <comma> <value expression> }... <right paren>
        

Syntax Rules

  1. COALESCE (V1, V2) is equivalent to the following <case specification>:

      CASE WHEN V1 IS NOT NULL THEN V1 ELSE V2 END
              

     

  2. COALESCE (V1, V2,..., Vn), for n >= 3, is equivalent to the following <case specification>:

      CASE WHEN V1 IS NOT NULL THEN V1
           ELSE COALESCE (V2,...,Vn) END
              

Examples

  SELECT
      PROJ_NAME AS Projectname,
      COALESCE(e.FULL_NAME,'[< not assigned >]') AS Employeename
  FROM
    PROJECT p
    LEFT JOIN EMPLOYEE e
    ON (e.EMP_NO = p.TEAM_LEADER);

  SELECT
      COALESCE(Phone,MobilePhone,'Unknown') AS "Phonenumber"
  FROM
    Relations;
        

c) NULLIF

Returns NULL for a sub-expression if it has a specific value, otherwise returns the value of the sub-expression.

Syntax Pattern

  <case abbreviation> ::=
    NULLIF <left paren> <value expression> <comma> <value expression> <right paren>
        

Syntax Rules

NULLIF (V1, V2) is equivalent to the following <case specification>:

  CASE WHEN V1 = V2 THEN NULL ELSE V1 END
        

Example

  UPDATE PRODUCTS
     SET STOCK = NULLIF(STOCK,0)
        

(1.5) SQL99-compliant Savepoints

Nickolay Samofatov

User savepoints (alternative name nested transactions) provide a convenient method to handle business logic errors without needing to roll back the transaction. Available only in DSQL.

Use the SAVEPOINT statement to identify a point in a transaction to which you can later roll back.

Syntax Patterns

  SAVEPOINT <identifier>;
      

<identifier> specifies the name of a savepoint to be created. After a savepoint has been created, you can either continue processing, commit your work, roll back the entire transaction, or roll back to the savepoint.

Savepoint names must be distinct within a given transaction. If you create a second savepoint with the same identifer as an earlier savepoint, the earlier savepoint is erased.

  ROLLBACK [WORK] TO [SAVEPOINT] <identifier>;
      

This statement performs the following operations:

  • Rolls back changes performed in the transaction after the savepoint

     

  • Erases all savepoints created after that savepoint. The named savepoint is retained, so you can roll back to the same savepoint multiple times. Prior savepoints are also retained.

     

  • Releases all implicit and explicit record locks acquired since the savepoint. Other transactions that have requested access to rows locked after the savepoint must continue to wait until the transaction is committed or rolled back. Other transactions that have not already requested the rows can request and access the rows immediately.

    Note

    This behaviour may change in future product versions.

Important

The Savepoint undo log may consume significant amounts of server memory, especially if you update the same records in the same transaction multiple times. Use the RELEASE SAVEPOINT statement to release system resources consumed by savepoint maintenance.

  RELEASE SAVEPOINT <identifier> [ONLY];
      

RELEASE SAVEPOINT statement erases the savepoint <identifer> from the transaction context. Unless you specify the ONLY keyword, all savepoints established since the savepoint <identifier> are erased too.

Example using Savepoints

  create table test (id integer);
  commit;
  insert into test values (1);
  commit;
  insert into test values (2);
  savepoint y;
  delete from test;
  select * from test; -- returns no rows
  rollback to y;
  select * from test; -- returns two rows
  rollback;
  select * from test; -- returns one row
      

Internal savepoints

By default, the engine uses an automatic transaction-level system savepoint to perform transaction rollback. When you issue a ROLLBACK statement, all changes performed in this transaction are backed out via a transaction-level savepoint and the transaction is then committed. This logic reduces the amount of garbage collection caused by rolled back transactions.

When the volume of changes performed under a transaction-level savepoint is getting large (10^4-10^6 records affected) the engine releases the transaction-level savepoint and uses the TIP mechanism to roll back the transaction if needed.

Tip

If you expect the volume of changes in your transaction to be large, you can use the TPB flag isc_tpb_no_auto_undo to avoid the transaction-level savepoint being created.

Savepoints and PSQL

Implementing user savepoints in PSQL layer would break the atomicity rule for statements, including procedure call statements. Firebird provides exception handling in PSQL to undo changes performed in stored procedures and triggers. Each SQL/PSQL statement is executed under a system of automatic, internal savepoints, whereby either the entire statement will complete successfully or ALL its changes are rolled back and an exception is raised.

Each PSQL exception handling block is also bounded by automatic system savepoints.

(1.5) Explicit locking

Nickolay Samofatov

The addition of the optional WITH LOCK clause provides a limited explicit pessimistic locking capability for cautious use in conditions where the affected row set is

  1. extremely small (ideally, a singleton) AND

  2. precisely controlled by the application code.

Caution

This is for experts only!

The need for a pessimistic lock in Firebird is very rare indeed and should be well understood before use of this extension is considered.

It is essential to understand the effects of transaction isolation and other transaction attributes before attempting to implement explicit locking in your application.

Syntax Pattern

  SELECT ... FROM <sometable>
    [WHERE ...]
    [FOR UPDATE [OF ...]]
    [WITH LOCK]
  ...;
      

If the WITH LOCK clause succeeds, it will secure a lock on the selected rows and prevent any other transaction from obtaining write access to any of those rows, or their dependants, until your transaction ends.

If the FOR UPDATE clause is included, the lock will be applied to each row, one by one, as it is fetched into the server-side row cache. It becomes possible, then, that a lock which appeared to succeed when requested will nevertheless fail subsequently, when an attempt is made to fetch a row which becomes locked by another transaction.

The SELECT... WITH LOCK construct is available in DSQL and PSQL.

Important

The SELECT... WITH LOCK construct can succeed only in a top-level, single-table SELECT statement. It is not available

  • in a subquery specification

     

  • for joined sets

     

  • with the DISTINCT operator, a GROUP BY clause or any other aggregating operation

     

  • with a view

     

  • with the output of a selectable stored procedure

     

  • with an external table

Understanding the WITH LOCK clause

As the engine considers, in turn, each record falling under an explicit lock statement, it returns either the record version that is the most currently committed, regardless of database state when the statement was submitted, or an exception.

Wait behaviour and conflict reporting depend on the transaction parameters specified in the TPB block:

Table 4.1. How TPB settings affect explicit locking

TPB mode Behaviour

isc_tpb_consistency

Explicit locks are overridden by implicit or explicit table-level locks and are ignored

isc_tpb_concurrency

+ isc_tpb_nowait

If a record is modified by any transaction that was committed since the transaction attempting to get explicit lock started, or an active transaction has performed a modification of this record, an update conflict exception is raised immediately

isc_tpb_concurrency

+ isc_tpb_wait

If the record is modified by any transaction that has committed since the transaction attempting to get explicit lock started, an update conflict exception is raised immediately.

If an active transaction is holding ownership on this record (via explicit locking or by a normal optimistic write-lock) the transaction attempting the explicit lock waits for the outcome of the blocking transaction and, when it finishes, attempts to get the lock on the record again. This means that, if the blocking transaction committed a modified version of this record, an update conflict exception will be raised.

isc_tpb_read_committed

+ isc_tpb_nowait

If there is an active transaction holding ownership on this record (via explicit locking or normal update), an update conflict exception is raised immediately.

isc_tpb_read_committed

+ isc_tpb_wait

If there is an active transaction holding ownership on this record (via explicit locking or by a normal optimistic write-lock), the transaction attempting the explicit lock waits for the outcome of blocking transation and when it finishes, attempts to get the lock on the record again.

Update conflict exceptions can never be raised by an explicit lock statement in this TPB mode.


How the engine deals with WITH LOCK

When an UPDATE statement tries to access a record that is locked by another transaction, it either raises an update conflict exception or waits for the locking transaction to finish, depending on TPB mode. Engine behaviour here is the same as if this record had already been modified by the locking transaction.

No special gdscodes are returned from conflicts involving pessimistic locks.

The engine guarantees that all records returned by an explicit lock statement are actually locked and DO meet the search conditions specified in WHERE clause, as long as the search conditions do not depend on any other tables, via joins, subqueries, etc. It also guarantees that rows not meeting the search conditions will not be locked by the statement. It can NOT guarantee that there are no rows which, though meeting the search conditions, are not locked.

Note

This situation can arise if other, parallel transactions commit their changes during the course of the locking statement's execution.

The engine locks rows at fetch time. This has important consequences if you lock several rows at once. Many access methods for Firebird databases default to fetching output in packets of a few hundred rows ("buffered fetches"). Most data access components cannot bring you the rows contained in the last-fetched packet, where an error occurred.

The optional OF <column-names> sub-clause

The FOR UPDATE clause provides a technique to prevent usage of buffered fetches, optionally with the OF <column-names> to enable positioned updates.

Tip

Alternatively, it may be possible in your access components to set the size of the fetch buffer to 1. This would enable you to process the currently-locked row before the next is fetched and locked, or to handle errors without rolling back your transaction.

Caveats using WITH LOCK

  • Rolling back of an implicit or explicit savepoint releases record locks that were taken under that savepoint, but it doesn't notify waiting transactions. Applications should not depend on this behaviour as it may get changed in the future.

     

  • While explicit locks can be used to prevent and/or handle unusual update conflict errors, the volume of deadlock errors will grow unless you design your locking strategy carefully and control it rigorously.

     

  • Most applications do not need explicit locks at all. The main purposes of explicit locks are (1) to prevent expensive handling of update conflict errors in heavily loaded applications and (2) to maintain integrity of objects mapped to a relational database in a clustered environment. If your use of explicit locking doesn't fall in one of these two categories, then it's the wrong way to do the task in Firebird.

     

  • Explicit locking is an advanced feature, do not misuse it ! While solutions for these kinds of problems may be very important for web sites handling thousands of concurrent writers, or for ERP/CRM systems operating in large corporations, most application programs do not need to work in such conditions.

Examples using Explicit Locking

i) (simple)

  SELECT * FROM DOCUMENT WHERE ID=? WITH LOCK
        

ii) (multiple rows, one-by-one processing with DSQL cursor)

  SELECT * FROM DOCUMENT WHERE PARENT_ID=?
    FOR UPDATE WITH LOCK
        

(1.5) Improved Aggregate Handling

Arno Brinkman

Originally, grouped sets could be grouped only on named columns. In Firebird 1.0, it became possible to group by a UDF expression. In 1.5, several further extensions to the handling of aggregate functions and the GROUP BY clause now allow groupings to be made by the degree of columns in the output specification (their 1-based "ordinal left-to-right position", as in the ORDER BY clause) or by a variety of expressions.

Caution

Not all expressions are currently allowed inside the GROUP BY list. For example, concatenation is not allowed.

Group By syntax

  SELECT ... FROM .... [GROUP BY group_by_list]

    group_by_list : group_by_item [, group_by_list];

    group_by_item : column_name
      | degree (ordinal)
      | udf
      | group_by_function;

  group_by_function : numeric_value_function
      | string_value_function
      | case_expression
      ;

  numeric_value_function : EXTRACT '(' timestamp_part FROM value ')';

  string_value_function	: SUBSTRING '(' value FROM pos_short_integer ')'
      | SUBSTRING '(' value FROM pos_short_integer FOR nonneg_short_integer ')'
      | KW_UPPER '(' value ')'
      ;
      

Important

The group_by_item cannot be a reference to any aggregate-function (including any that are buried inside an expression) from the same context.

HAVING

The having clause only allows aggregate functions or valid expressions that are part of the GROUP BY clause. Previously it was allowed to use columns that were not part of the GROUP BY clause and to use non-valid expressions.

ORDER BY

When the context is an aggregate statement, the ORDER BY clause only allows valid expressions that are aggregate functions or expression parts of the GROUP BY clause.

Previously it was allowed to use non-valid expressions.

Aggregate functions inside subqueries

It is now possible to use an aggregate function or expression contained in the GROUP BY clause inside a subquery.

Examples

  SELECT
    r.RDB$RELATION_NAME,
    MAX(r.RDB$FIELD_POSITION),
    (SELECT
       r2.RDB$FIELD_NAME
     FROM
       RDB$RELATION_FIELDS r2
     WHERE
       r2.RDB$RELATION_NAME = r.RDB$RELATION_NAME and
       r2.RDB$FIELD_POSITION = MAX(r.RDB$FIELD_POSITION))
  FROM
    RDB$RELATION_FIELDS r
  GROUP BY
    1
  /* ************ */
  SELECT
    rf.RDB$RELATION_NAME AS "Relationname",
    (SELECT
       r.RDB$RELATION_ID
     FROM
       RDB$RELATIONS r
     WHERE
       r.RDB$RELATION_NAME = rf.RDB$RELATION_NAME)
    AS "ID",
    COUNT(*) AS "Fields"
  FROM
    RDB$RELATION_FIELDS rf
  GROUP BY
    rf.RDB$RELATION_NAME
        

Mixing aggregate functions from different contexts

Aggregate functions from different contexts can be used inside an expression.

Examples

  SELECT
    r.RDB$RELATION_NAME,
    MAX(i.RDB$STATISTICS) AS "Max1",
    (SELECT
       COUNT(*) || ' - ' || MAX(i.RDB$STATISTICS)
     FROM RDB$RELATION_FIELDS rf
    WHERE
      rf.RDB$RELATION_NAME = r.RDB$RELATION_NAME) AS "Max2"
  FROM
    RDB$RELATIONS r
    JOIN RDB$INDICES i on (i.RDB$RELATION_NAME = r.RDB$RELATION_NAME)
  GROUP BY
    r.RDB$RELATION_NAME
  HAVING
    MIN(i.RDB$STATISTICS) <> MAX(i.RDB$STATISTICS)
        

Note! This query gives results in FB1.0, but they are WRONG!

Subqueries are supported inside an aggregate function

Using a singleton SELECT expression inside an aggregate function is supported.

Example

  SELECT
    r.RDB$RELATION_NAME,
    SUM((SELECT
           COUNT(*)
         FROM
           RDB$RELATION_FIELDS rf
         WHERE
           rf.RDB$RELATION_NAME = r.RDB$RELATION_NAME))
  FROM
    RDB$RELATIONS r
    JOIN RDB$INDICES i
      on (i.RDB$RELATION_NAME = r.RDB$RELATION_NAME)
  GROUP BY
    r.RDB$RELATION_NAME
        

Nested aggregate functions

Using an aggregate function inside another aggregate function is possible if the inner aggregate function is from a lower context (see example above).

Grouping by degree (ordinal number)

Using the degree number of the output column in the GROUP BY clause 'copies' the expression from the select list (as does the ORDER BY clause). This means that, when a degree number refers to a subquery, the subquery is executed at least twice.

(1.5) ORDER BY clause can specify expressions and nulls placement

Nickolay Samofatov

Order by expression

The ORDER BY clause lets you specify any valid expressions to sort query results. If the expression consists of a single number, it is interpreted as column (degree) number, as previously.

Nulls placement

The ordering of nulls in the result set can be controlled using a nulls placement clause.

Results can be sorted so that nulls are placed either above (NULLS FIRST) or below (NULLS LAST) the sorted non-nulls.

Behaviour when nulls placement is unspecified is NULLS LAST.

Syntax Pattern

  SELECT ... FROM .... [ORDER BY order_list]....;

  order_list : order_item [, order_list];
  order_item : <expression> [order_direction] [nulls_placement]
  order_direction : ASC | DESC;
  nulls_placement : NULLS FIRST | NULLS LAST;
      

Restrictions

  • If NULLS FIRST is specified, no index will be used for sorting.

     

  • The results of a sort based on values returned from a UDF or a stored procedure will be unpredictable if the values returned cannot be used to determine a logical sorting sequence.

     

  • The number of procedure invocations from specifying a sort based on a UDF or stored procedure will be unpredictable, regardless of whether the ordering is specified by the expression itself or by an ordinal number representing an expression in the column-list specification.

     

  • An ordering clause for sorting the output of a union query may use only ordinal (degree) numbers to refer to the ordering columns.

Examples

i)

  SELECT * FROM MSG
  ORDER BY PROCESS_TIME DESC NULLS FIRST
      

ii)

  SELECT FIRST 10 * FROM DOCUMENT
  ORDER BY STRLEN(DESCRIPTION) DESC
      

iii)

  SELECT DOC_NUMBER, DOC_DATE FROM PAYORDER
  UNION ALL
  SELECT DOC_NUMBER, DOC_DATA FROM BUDGORDER
  ORDER BY 2 DESC NULLS LAST, 1 ASC NULLS FIRST
      

(1.5) SELECT FIRST 0.. is Now Valid

In Firebird 1.5, zero can be accepted as an argument for FIRST in the SELECT FIRST m .. SKIP n construction. An empty result set will be returned.

(1.0) SELECT FIRST m .. SKIP n

(from the v.1.0 release notes)

Syntax Pattern

  SELECT [FIRST (<integer expr m>)] [SKIP (<integer expr n>)]
        

Retrieves the first m rows of the selected output set. The optional SKIP clause will cause the first n rows to be discarded and return an output set of m rows starting at n + 1. In the simplest form, m and n are integers but any Firebird expression that evaluates to an integer is valid.

Available in SQL and DSQL except where otherwise indicated.

Arguments m and n

Parentheses are required for expression arguments and are optional otherwise.

The arguments can also bind variables, e.g.

SKIP ? * FROM ATABLE returns the remaining dataset after discarding the n rows at the top, where n is passed in the "?" variable.

SELECT FIRST ? COLUMNA, COLUMNB FROM ATABLE returns the first m rows and discards the rest.

An identifier that evaluates to an integer may also be used in GDML, although not in SQL or DSQL.

FIRST and SKIP Elements

The FIRST clause is also optional, i.e. you can include SKIP in a statement without FIRST to get an output set that simply excludes the rows appointed to SKIP.

Example

  SELECT SKIP (5+3*5) * FROM MYTABLE;

  SELECT FIRST (4-2) SKIP ? * FROM MYTABLE;

  SELECT FIRST 5 DISTINCT FIELD FROM MYTABLE;
          
Two Gotchas with SELECT FIRST

1. This:

  delete from TAB1
    where PK1 in (select first 10 PK1 from TAB1);
          

will delete all of the rows in the table. Ouch! the sub-select is evaluating each 10 candidate rows for deletion, deleting them, slipping forward 10 more...ad infinitum, until there are no rows left. Beware!

2. Queries like:

  ...
  WHERE F1 IN ( SELECT FIRST 5 F2 FROM TABLE2
    ORDER BY 1 DESC )
          

won't work as expected, because the optimization performed by the engine transforms the correlated WHERE...IN (SELECT...) predicate to a correlated EXISTS predicate. It's obvious that in this case FIRST N doesn't make any sense:

  ...
  WHERE EXISTS (
     SELECT FIRST 5 TABLE2.F2 FROM TABLE2
     WHERE TABLE2.F2 = TABLE1.F1
     ORDER BY 1 DESC )
          

Other Firebird 1.0.x Features

The remaining items in this section are DML enhancements that were introduced in Firebird 1.0.x, described again for the reader's convenience.

(1.0) GROUP BY UDF

It is now possible to aggregate a SELECT by grouping on the output of a UDF. e.g.

  select
    strlen(rtrim(rdb$relation_name)),
    count(*) from rdb$relations
  group by strlen(rtrim(rdb$relation_name))
  order by 2
        

A side-effect of the changes enabling grouping by UDFs is that, whereas previously you could not call built-in Firebird functions in GROUP BY, now, by creating a dummy UDF wrapper, you can do:

  select count(*)
  from rdb$relations r
  group by bin_or((select count(rdb$field_name)
     from rdb$relation_fields f
     where f.rdb$relation_name = r.rdb$relation_name),1)
        

(1.0) SUBSTRING( <string expr> FROM <pos> [FOR <length>])

Internal function, available in SQL and DSQL, implementing the ANSI SQL SUBSTRING() function. It will return a stream consisting of the byte at <pos> and all subsequent bytes up to the end of the string. If the optional FOR <length> is specified, it will return the lesser of <length> bytes or the number of bytes up to the end of the input stream.

The first argument can be any expression, constant or identifier that evaluates to a string.

<pos> must evaluate to an integer. <pos> in the string starts at 1, like other SQL positional elements.

Neither <pos> nor <length> can be query parameters: they must evaluate to constants.

Because <pos> and <length> are byte positions, the identifier of the input string can be a binary blob, or a sub_type 1 text blob with an underlying one-byte-per-character charset. The function currently does not handle text blobs with Chinese (2 byte/char maximum) or Unicode (3 byte/char maximum) character sets.

For a string argument (as opposed to a blob), the function will tackle ANY charset.

Example

  UPDATE ATABLE
    SET COLUMNB = SUBSTRING(COLUMNB FROM 4 FOR 99)
    WHERE ...
        

Please refer also to the later section on External Functions (UDFs) for details of changes and additions to external substring functions in the standard UDF library.

Prev: Data Definition Language (DDL)Firebird Documentation IndexUp: SQL Language EnhancementsNext: Stored Procedure and Trigger Language (PSQL)
Firebird Documentation IndexFirebird 1.5.6 Release NotesSQL Language Enhancements → Data Manipulation Language (DML)