IBM InfoSphere CDC Vs Long Running Transactions

InfoSphere Change Data Capture (ISCDC) which is part of IBM InfoSphere Data Replication has the concept of Restart Position via bookmark which keeps advancing as the records are replicated and applied at the target that helps tracking CDC to keep the record of how much data have already been replicated so that CDC when restarted can start from this position.

There was a customer using Oracle as a source from which the data was replicated, and it was observed that InfoSphere CDC was holding up transaction inspite of not having any open transactions in the V$TRANSACTION. Later it turned out to be that the open transaction was identified from the GV$TRANSACTION table. These are global transactions that can occur in the Oracle RAC environment.
So, InfoSphere CDC holding up the transactions were found to be very much legitimate.

Long transactions in the database (and hence restart position not advancing in the ISCDC) sometime is a concern because
  1. when ISCDC is to be restarted for some reason, ISCDC will demand from the log files that corresponds to the Restart Position
  2. since the data was not committed for a long time, they are not replicated by ISCDC as expected and it is perceived as being high latency

Comments

  1. Unknown25 August 2011 at 22:21

    Thanks for the post..was very helpful..could you explain if there are any methods to find out what are the transactions currently being built in ISCDC and is there any tool / utility to analyze the TxQueues of CDC ?

    ReplyDelete
  2. Srinivasan R Mottupalli27 September 2011 at 20:48

    Yes Amit! There is a new tool that will be part of 6.5.1 to analyze the open and long running transactions. It requires the instance to be Normally Shutdown and dump the staging store TXQ files which can further be analyzed with the new tool.

    ReplyDelete
  3. Unknown20 October 2015 at 13:24

    How the client resolve their long running transactions issue. We are in similar situation where ISCDC complain of XA/global transactions that are not committed while we see them commit. So the bookmarks are not moving and the latency is in excess of 8 hours. We are using Oracle 11.2 RAC as source DB with ISCDC 9.

    ReplyDelete

Post a Comment

Popular posts from this blog

Informix 14.10 - EAR - Encryption At Rest (Dbspaces and Backups) with multiple levels of ciphers

Disk Encryption - Operational Data Only Up to IDS 12.10 the encryption was allowed only on the operational data on the dbspaces that the server actively performing read/write operations. The Backup data are not encrypted although there is a provision to give control to the end-user to do it via BACKUP_FILTER and RESTORE_FILTER . IDS does not control any encryption that is done with this method and hence takes no responsibility if the encryption key is lost or due to some other reason the decryption becomes impossible. In order to encrypt the operational data, all needed is to configure the onconfig parameter DISK_ENCRYPTION . It uses keystore file ($INFORMIXDIR/etc/*.p12) and stash file ($INFORMIXDIR/etc/*.sth) to support the encryption. These files are created automatically in 12.10 when DISK_ENCRYPTION is configured. The keystore file is used to store the encryption key and stash file contains the password to open the keystore file where encryption key is stored.  Thes...
Read more

InfoSphere CDC (ISCDC) - FAQ

1. What is Latency in Change Data Capture? Latency information is normally for Mirroring subscription. As InfoSphere CDC measures latency as the amount of time that passes between when data changes on a source table and when it changes on the target table. For example, if an application inserts a row into the source table at 10:00 and CDC applies that row to the target able at 10:15, then the latency for the subscription is 15 minutes. 2. What is in-scope and out-of-scope tables in InfoSphere CDC terminology? The tables are inscope tables if they are replicated to the target by InfoSphere CDC. The tables are out-of-scope when those tables are not part of any replication. 3. How to ignore error that occurs during refresh/mirrorring? Prior 6.3 D_MIRROR_REFRESH_ERROR_STOP=OFF - refresh continues after an error has occurred. D_MIRROR_MIRROR_ERROR_STOP=OFF - mirroring continues after an error has occurred. Post 6.3 Set the following 2 system parameters via Management Console. mirror_end...
Read more

HCL Informix 14.10 - onkstore utility - Remote Key Store

Read my blog on Encryption At Rest (EAR) before this article HCL Informix 14.10 has come up with a new utility called onkstore to generate the keystore (.p12) and stash file (.sth) for types such as local AWS_EAR AWS_BAR Type: local This is provided to store the .p12 and .sth files locally and to have our backward compatibilty with earlier versions which were storing these files locally. These files were essential to perform the encryption of the data and eventually decrypt them when needed. No connectivity to the external AWS site is needed to perform this however since the encryption key files are saved locally on the disk, there is a danger of loosing them and hence loosing the encrypted data that remains encrypted for ever and no way to decrypt them Type: AWS_EAR In order to avoid loosing of the encryption key, we have a new type called AWS_EAR to make sure the encryption key is saved safely and remotely so that can be accessed from amazon web-services, w...
Read more