Return to Newsletter Contents...

Data Integrator Multi User & Migration Strategies

by: Thomas Hinkle, BI Consultant, CRCP, DI Instructor

Data Integrator has the ability to support a number of environments, such as development, test, and production environments. This includes large enterprises with many developers working on multiple projects. Data Integrator also supports multi-site architectures whether centralized or local.

The ETL application development process typically involves three distinct phases; Design, Test and Production. You can use Data Integrator in all three phases. Because each phase might require a different repository to control environment differences, Data Integrator provides controlled mechanisms for moving objects from phase to phase. Each phase could involve a different computer in a different environment with different security settings. For example, design and initial test may only require limited sample data and low security, while final testing may require a full emulation of the production environment including strict security.

Design phase

In this phase, you define objects and build diagrams that instruct Data Integrator in your data movement requirements. Data Integrator stores these specifications so you can reuse them or modify them as your system evolves. Design your project with migration to testing and final production in mind. Consider these basic guidelines as you design your project:

• Construct design steps as independent, testable modules

• Use meaningful names for each step you construct

• Make independent modules that can be used repeatedly to handle common operations

• Use test data that reflects all the variations in your production data

Test phase

In this phase, you use Data Integrator to test the execution of your application. At this point, you can test for errors and trace the flow of execution without exposing production data to any risk. If you discover errors during this phase, return the application to the design phase for correction, then test the corrected application.

Testing has two parts:

• The first part includes designing the data movement using your local repository

• The second part includes fully emulating your production environment, including data volume

Data Integrator provides feedback through trace, error, and statistics logs during both parts of this phase.  The testing repository should emulate your production environment as closely as possible, including scheduling jobs rather than manually starting them.

Production phase

In this phase, you set up a schedule in Data Integrator to run your application as a job. Evaluate results from production runs and when necessary, return to the design phase to optimize performance and refine your target requirements. After moving a Data Integrator application into production, monitor it in the Administrator for performance and results. During production: Monitor your jobs and the time it takes for them to complete.

Migration basics

The trace and monitoring logs provide information about each job as well as the work flows and data flows contained within the job. You can customize the log details. However, the more information you request in the logs, the longer the job runs. Balance job run-time against the information necessary to analyze job performance.

• Check the accuracy of your data

To enhance or correct your jobs:

• Make changes in your design environment

• Repeat the object testing

• Move changed objects back into production

Data Integrator provides two migration mechanisms:

• Export/import migration works best with small to medium-sized projects where a small number of developers work on somewhat independent Data Integrator applications through all phases of development

• Multi-user development works best in larger projects where two or more developers or multiple teams are working on interdependent parts of Data Integrator applications through all phases of development

Regardless of which migration mechanism you choose, Business Objects recommends you prepare for migration using one or more tools that best fit your development environment. The mechanism and tools you use will depend on the needs of your development environment.

Export/Import migration

Export/import is the basic mechanism for migrating Data Integrator applications between phases. First, you export jobs from the local repository to another local repository or to an intermediate file which you can then import into another local repository. For example, when moving from design repository to test repository, you export from the design repository to a file, then import the file to your test repository. If you find application errors during testing, you can correct them in the development environment, then export the corrected version and import it back into the test repository for retesting.

Multi-user migration

You can also migrate Data Integrator applications between phases in more complex development environments. Instead of exporting and importing applications, multi-user development provides a more secure check-in, check-out, and get mechanism, using a central repository to store the master copies of your application elements. Multi-user development includes other advanced features like labeling and filtering to provide you more flexibility and control in managing application objects.

For more detailed information refer to the Data Integrator Advanced Development and Migration Guide located with the technical manuals.

Go to Top  |  Return to Newsletter Contents