Coordinated Replicats: The Best Way to Do Initial Load

TL;DR

For Oracle GoldenGate initial loads, the coordinated replicat is the better choice over the parallel replicat. Here is the short version:

• Use the REST API approach for initial loads of data — it is scriptable and the best method available.
• SPECIALRUN is gone. With the Microservices/REST architecture, every replicat reads both EXTTRAIL and EXTFILE, so any replicat can serve as an initial load replicat.
• Parallel replicat serializes on a single table, which means a big single-table load does not scale and takes longer than planned.
• Coordinated replicat ships with 25 threads by default, and with THREAD or THREADRANGE() you can load even a single table in parallel.
• The process barely changes. 

Back in May of 2018, I wrote up how to perform an initial load with Oracle GoldenGate using the RESTful APIs. That post still holds true today, and it is still the best way I know to do an initial load of data — not DDL. Everything can be scripted, which makes it developer friendly and repeatable, and that is exactly what you want when you are moving real volumes of data.

What I want to do here is pick up where that post left off and talk about the replicat side of the house: specifically, why the coordinated replicat is the one I reach for when I am doing an initial load.

Two Trail File Types: EXTTRAIL vs. EXTFILE

The key to understanding the initial load process is knowing that it uses two different kinds of trail files. They look similar on the surface, but they do very different jobs.

In the earlier post, I showed how this worked using standard extracts and replicats. Here, the spotlight is on the replicat.

The SPECIALRUN Change

One of the bigger changes Oracle made when moving to the RESTful API architecture was removing the SPECIALRUN parameter from replicats. The practical effect is significant: every replicat can now read both the EXTTRAIL and the EXTFILE formats.

There is a trade-off, and you should know about it going in. Without SPECIALRUN, the replicat will not automatically stop once it finishes loading the EXTFILE. You have to account for that in how you run and monitor the process. Oracle calls the unified behavior a feature, and it is — you just lose the built-in “stop when done” convenience.

One note worth keeping in your back pocket: the last time SPECIALRUN shows up is in the 19.1 release of Oracle GoldenGate. If you are on a current release, it is not coming back.

The upshot of all this is simple: with this change in replicat architecture, you can now use any replicat as an initial load replicat.

Why Coordinated Replicat Beats Parallel Replicat for Initial Load

Across multiple implementations and tests of replicats for initial loads, I keep landing on the same answer: the coordinated replicat is the best fit for the initial load of data. Both replicat types are useful here, so let me give you the straight story on why I pick coordinated.

Parallel Replicat

Parallel replicat lets you define the minimum and maximum parallelism for the process. That sounds ideal until you hit the catch: if the parallel replicat is loading a single table, the parallelism becomes serialized and does not scale. In plain terms, your initial load takes longer than planned or expected — exactly when you can least afford it.

Coordinated Replicat

Coordinated replicat does the same kind of work as parallel replicat — it is, in fact, the precursor to it — but it has an advantage that matters for initial loads. By default, the coordinated replicat has 25 threads available. A single table can then be loaded using the THREAD or THREADRANGE() parameter, which leverages those pre-allocated threads. The result is a single table that is properly loaded in parallel, which is the thing parallel replicat could not give you.

How This Changes Your Initial Load Process

Here is the honest answer: it really doesn’t. If you already have the REST API initial load process I described earlier, the only change is that you are switching out one, maybe two, replicats depending on the size of the environment you are loading. Same approach, better engine under the hood.

FAQ

What is the difference between EXTTRAIL and EXTFILE in Oracle GoldenGate?

EXTTRAIL is a binary file that houses Change Data Capture (CDC) transactions. EXTFILE is a file used for full table dumps of all the data. Initial loads rely on EXTFILE, while ongoing replication relies on EXTTRAIL.

Why is the coordinated replicat better than the parallel replicat for initial load?

The coordinated replicat ships with 25 threads by default and can load a single table in parallel using THREAD or THREADRANGE(). The parallel replicat serializes parallelism when loading a single table, so large single-table loads do not scale and take longer.

What happened to the SPECIALRUN parameter?

SPECIALRUN was removed when Oracle moved to the RESTful API architecture, and its last appearance is in the 19.1 release. Because of this, every replicat now reads both EXTTRAIL and EXTFILE, which means any replicat can act as an initial load replicat.

Does removing SPECIALRUN have any downside?

Yes. Without SPECIALRUN, the replicat will not automatically stop once it finishes loading the EXTFILE. You need to plan for stopping or monitoring the process yourself.

Do I have to rebuild my initial load process to use a coordinated replicat?

No. The overall REST API initial load process stays the same. You are only swapping out one, possibly two, replicats depending on the size of the environment being loaded.

Closing Thoughts

The REST API initial load approach has aged well, and the move away from SPECIALRUN quietly made it more flexible. If you are loading data — especially large single tables — reach for the coordinated replicat and put those 25 threads to work. If you are running initial loads at scale and want to talk through your specific environment, reach out. I am always happy to compare notes.

— Bobby

Please follow and like: