Data Migration Best Practices
Content
Data Migration Best Practices
Organizations migrate data all the time. They might migrate data when they acquire new hardware or software, when they relocate their data centers, when they make changes to their facilities, or when they optimize server or storage performance. Data migration may also be a component of Business Intelligence (BI), backup, and disaster recovery strategies. Perhaps because of its frequency, most IT managers consider data migration a routine chore requiring little attention. They mistakenly believe they can simply copy files from one volume to another or use replication tools like SRDF, TDMF, or even the host native volume manager—at best, they plan to use one of a variety of Extra, Transform, Load (ETL) tools intended to automate, simplify, and streamline the data migration process. Unfortunately, data migration is more complex than the mere movement of data from one storage device to another. Typically, true data migration is a one-time event in which data undergoes a format change as it moves to a new system—so it will be compatible with the new system. According to a recent survey conducted by Bloor, the IT research company, these so-called “routine” data migrations have a 60 percent failure rate. 1 They fail because the copy process fails, the server or the target storage device crashes, the data center experiences an array failure, or—the horror—the data center experiences a complete systems failure. Sometimes the data is just bad to begin with, or the migration process itself corrupts it. Most often, however, data migrations fail because companies simply underestimate the expertise, processes, and resources required. When data migrations fail, projects relying on that data run over budget and beyond their completion dates. Far more seriously, however, companies lose data integrity with failed migrations. Without data integrity, businesses can’t be confident their data is current, accurate, or complete. They may not even be able to identify the last valid copy of their data. To avoid data integrity loss, companies should always keep the focus of migrations on their data and make establishing data integrity their primary goal. In short, they need a methodical, data-focused migration process.
Top Five Data Migration Best Practices: 1. Understand, select, and locate data to migrate. 2. Extract, clean, and transform the data. 3. Move the data systematically. 4. Test and validate. 5. Audit and document.
It’s All About the Data
At a basic level, data migration affects several key IT infrastructure components. For example, it’s often an expected step when companies acquire new hardware, deploy new applications, or upgrade existing applications. It also can play a role in process redesign, especially when the movement and placement of data affect process performance. Because data migration is an integral step in these frequent IT operations, any migration problems hinder your agility and responsiveness. If your data migration fails, or if it even slows processes, your costs increase. You have no choice but correct the migration, draining your IT resources.
Unfortunately, data migration is more complex than the mere movement of data from one storage device to another. Typically, true data migration is a one-time event in which data undergoes a format change as it moves to a new system—so it will be compatible with the new system.
1 Howard, Phillip. “Bloor Research White Paper: Data Migration.” Informatica. October 2007. September 2009. <http://vip.informatica.com/?elqPURLPage=552>.
Furthermore, your data may be your most valuable asset. According to Bloor, companies spend over $5 billion a year on data migration. Add that figure to the 60% failure rate, and you have a powerful financial incentive to make sure your data migration is successful.
Given the value of your data, it pays to take a systematic approach to data migration. Follow these best practices so you can migrate the right data, the right way, while protecting data integrity.
To begin, IT managers shouldn’t undertake data migration alone. Successful data migrations require a specialized team within the organization that includes: • • • • Database administrators and application people who know the data and its current form Storage and network people who know where the data resides and how to access it Testing and Quality Assurance people to validate data quality Compliance staff to approve the necessary documentation and audit provisions
Start your successful data migration by knowing about the sources of data you’ll move to the new system. The best place to begin is by talking to your users.
Besides the internal team members, you may want to bring in outside experts experienced in data migration and the best available tools. Additionally, an outside expert frees up your IT staff to focus on their regular work. Once you’ve assembled your data migration team, you can complete the migration process quickly and efficiently—while you eliminate data migration failures.
1
Understand, Select, and Locate Data to Migrate
Do you know what data you are migrating, where it resides, and what form it takes? Start your successful data migration by knowing about the sources of data you’ll move to the new system. The best place to begin is by talking to your users. When you know where to locate the data and how it is stored, understand which elements from each of your sources you require. Finally, consider the form your data currently takes, and the form it will take when it arrives at its destination.
2
Extract, Clean, and Transform Data
Almost all data contains problems of sorts. Take your data migration as an opportunity to clean your data—improve the quality by identifying any errors, omissions, or issues. This might include standardizing formats or enforcing naming conventions, for example. Then you need to transform your data. You want to assess the format of the data and the format required for the new system. Then you need to determine how to get the old data into the new format. Fortunately, ETL and other automated tools can handle most of this process.
3
Move Data Systematically
Establish and enforce policies for migrating data. For example, you might restrict data migrations to overnight hours when network usage is low so it doesn’t interfere with regular business operations.
Once you’ve determined how to transform your data, you begin the largely manual process of actually transforming it. Often, various people perform data migrations ad hoc, taking various approaches at various times. Establish and enforce policies for migrating data. For example, you might restrict data migrations to overnight hours when network usage is low so it doesn’t interfere with regular business operations. Automated tools can also help confirm a consistent process. Although it may seem obvious, you should also try to avoid irreversible transformations, and definitely store a copy of the original data. If, for some reason, the migration doesn’t work, you’ll want the ability to undo mistakes.
4
Test and Validate
Migrating data over the network itself can introduce errors. That’s why you should check and test the migrated data to verify that it’s an accurate representation of the original data, and it’s currently in the expected format. Organizations, in their rush to complete a data migration, often skip this step. But without testing and validating the migrated data, you can’t be confident of its integrity. Ultimately, you want to be prepared for the most rigorous examination—user acceptance testing.
Primary Data Replication Approaches:
Synchronously Replicated Data Mirrors host data between a primary and secondary array, usually for disaster recoveries, especially those with very tight recovery point objectives (RPO) and recovery time objectives (RTO). Asynchronously Replicated Data Mirrors host data between a primary and secondary array, also used for disaster recoveries with less demanding RPO and RTO. Replicated Data with Local Snapshot Replicates data remotely with a local snapshot for instant data recovery, though insufficient for true disaster recovery. Three-site Replication Replicates data through an intermediate site. A specific vendor technology creates a multi-hop configuration, perhaps for enabling a mixture of synchronous and asynchronous replication.
5
Audit and Document
Once you’ve ensured the integrity of your data, you need to prove it. In this age of compliance mandates, you must document what you did at each stage of the migration process. Then you must create and preserve a clear, traceable audit trail of who did what to which data and when. In effect, you want to document what you did and preserve the evidence that you did it.
Good Data Migration Pays Off
You probably won’t migrate data often, but considering the increasing value of data, it’s worth doing right the first time. By implementing these simple best practices, you can dramatically reduce the likelihood of failure. At the same time, you can help your company become more responsive and avoid compliance problems. But the biggest payback you’ll reap, however, is better data integrity—and because your data is important, improved data integrity is priceless.
About the Author
Mark Teter is the Chief Technology Officer at Advanced Systems Group. He is an internationally recognized authority on information technology who regularly advises IT organizations, vendors, and government agencies on a broad range of information management issues. Each year, Mark conducts dozens of seminars and training programs for corporate and government institutions. He sits on several financial industry advisory boards and has recently published Paradigm Shift: Seven Keys of Highly Successful Linux and Open Source Adoptions.
About Advanced Systems Group
Advanced Systems Group (ASG) is a Denver-based IT consulting, integration, and project management firm—fully equipped with a high-end computing facility that provides testing, benchmarks, demonstrations and an executive briefing center. Acknowledged by Computer Reseller News as one of the Top Ten Storage Solution Providers, ASG pursues active involvement in the industry, maintaining the highest level of engineering certifications with partners and the vendor community.
Call us at 800.894.3619 www.virtual.com Denver . Baton Rouge . Boise . Colorado Springs . Dallas . Houston . Los Angeles . New Orleans Oklahoma City . Orange County . Phoenix . Portland . Salt Lake City . San Diego . Seattle
©2012 Advanced Systems Group. All rights reserved.
Organizations migrate data all the time. They might migrate data when they acquire new hardware or software, when they relocate their data centers, when they make changes to their facilities, or when they optimize server or storage performance. Data migration may also be a component of Business Intelligence (BI), backup, and disaster recovery strategies. Perhaps because of its frequency, most IT managers consider data migration a routine chore requiring little attention. They mistakenly believe they can simply copy files from one volume to another or use replication tools like SRDF, TDMF, or even the host native volume manager—at best, they plan to use one of a variety of Extra, Transform, Load (ETL) tools intended to automate, simplify, and streamline the data migration process. Unfortunately, data migration is more complex than the mere movement of data from one storage device to another. Typically, true data migration is a one-time event in which data undergoes a format change as it moves to a new system—so it will be compatible with the new system. According to a recent survey conducted by Bloor, the IT research company, these so-called “routine” data migrations have a 60 percent failure rate. 1 They fail because the copy process fails, the server or the target storage device crashes, the data center experiences an array failure, or—the horror—the data center experiences a complete systems failure. Sometimes the data is just bad to begin with, or the migration process itself corrupts it. Most often, however, data migrations fail because companies simply underestimate the expertise, processes, and resources required. When data migrations fail, projects relying on that data run over budget and beyond their completion dates. Far more seriously, however, companies lose data integrity with failed migrations. Without data integrity, businesses can’t be confident their data is current, accurate, or complete. They may not even be able to identify the last valid copy of their data. To avoid data integrity loss, companies should always keep the focus of migrations on their data and make establishing data integrity their primary goal. In short, they need a methodical, data-focused migration process.
Top Five Data Migration Best Practices: 1. Understand, select, and locate data to migrate. 2. Extract, clean, and transform the data. 3. Move the data systematically. 4. Test and validate. 5. Audit and document.
It’s All About the Data
At a basic level, data migration affects several key IT infrastructure components. For example, it’s often an expected step when companies acquire new hardware, deploy new applications, or upgrade existing applications. It also can play a role in process redesign, especially when the movement and placement of data affect process performance. Because data migration is an integral step in these frequent IT operations, any migration problems hinder your agility and responsiveness. If your data migration fails, or if it even slows processes, your costs increase. You have no choice but correct the migration, draining your IT resources.
Unfortunately, data migration is more complex than the mere movement of data from one storage device to another. Typically, true data migration is a one-time event in which data undergoes a format change as it moves to a new system—so it will be compatible with the new system.
1 Howard, Phillip. “Bloor Research White Paper: Data Migration.” Informatica. October 2007. September 2009. <http://vip.informatica.com/?elqPURLPage=552>.
Furthermore, your data may be your most valuable asset. According to Bloor, companies spend over $5 billion a year on data migration. Add that figure to the 60% failure rate, and you have a powerful financial incentive to make sure your data migration is successful.
Given the value of your data, it pays to take a systematic approach to data migration. Follow these best practices so you can migrate the right data, the right way, while protecting data integrity.
To begin, IT managers shouldn’t undertake data migration alone. Successful data migrations require a specialized team within the organization that includes: • • • • Database administrators and application people who know the data and its current form Storage and network people who know where the data resides and how to access it Testing and Quality Assurance people to validate data quality Compliance staff to approve the necessary documentation and audit provisions
Start your successful data migration by knowing about the sources of data you’ll move to the new system. The best place to begin is by talking to your users.
Besides the internal team members, you may want to bring in outside experts experienced in data migration and the best available tools. Additionally, an outside expert frees up your IT staff to focus on their regular work. Once you’ve assembled your data migration team, you can complete the migration process quickly and efficiently—while you eliminate data migration failures.
1
Understand, Select, and Locate Data to Migrate
Do you know what data you are migrating, where it resides, and what form it takes? Start your successful data migration by knowing about the sources of data you’ll move to the new system. The best place to begin is by talking to your users. When you know where to locate the data and how it is stored, understand which elements from each of your sources you require. Finally, consider the form your data currently takes, and the form it will take when it arrives at its destination.
2
Extract, Clean, and Transform Data
Almost all data contains problems of sorts. Take your data migration as an opportunity to clean your data—improve the quality by identifying any errors, omissions, or issues. This might include standardizing formats or enforcing naming conventions, for example. Then you need to transform your data. You want to assess the format of the data and the format required for the new system. Then you need to determine how to get the old data into the new format. Fortunately, ETL and other automated tools can handle most of this process.
3
Move Data Systematically
Establish and enforce policies for migrating data. For example, you might restrict data migrations to overnight hours when network usage is low so it doesn’t interfere with regular business operations.
Once you’ve determined how to transform your data, you begin the largely manual process of actually transforming it. Often, various people perform data migrations ad hoc, taking various approaches at various times. Establish and enforce policies for migrating data. For example, you might restrict data migrations to overnight hours when network usage is low so it doesn’t interfere with regular business operations. Automated tools can also help confirm a consistent process. Although it may seem obvious, you should also try to avoid irreversible transformations, and definitely store a copy of the original data. If, for some reason, the migration doesn’t work, you’ll want the ability to undo mistakes.
4
Test and Validate
Migrating data over the network itself can introduce errors. That’s why you should check and test the migrated data to verify that it’s an accurate representation of the original data, and it’s currently in the expected format. Organizations, in their rush to complete a data migration, often skip this step. But without testing and validating the migrated data, you can’t be confident of its integrity. Ultimately, you want to be prepared for the most rigorous examination—user acceptance testing.
Primary Data Replication Approaches:
Synchronously Replicated Data Mirrors host data between a primary and secondary array, usually for disaster recoveries, especially those with very tight recovery point objectives (RPO) and recovery time objectives (RTO). Asynchronously Replicated Data Mirrors host data between a primary and secondary array, also used for disaster recoveries with less demanding RPO and RTO. Replicated Data with Local Snapshot Replicates data remotely with a local snapshot for instant data recovery, though insufficient for true disaster recovery. Three-site Replication Replicates data through an intermediate site. A specific vendor technology creates a multi-hop configuration, perhaps for enabling a mixture of synchronous and asynchronous replication.
5
Audit and Document
Once you’ve ensured the integrity of your data, you need to prove it. In this age of compliance mandates, you must document what you did at each stage of the migration process. Then you must create and preserve a clear, traceable audit trail of who did what to which data and when. In effect, you want to document what you did and preserve the evidence that you did it.
Good Data Migration Pays Off
You probably won’t migrate data often, but considering the increasing value of data, it’s worth doing right the first time. By implementing these simple best practices, you can dramatically reduce the likelihood of failure. At the same time, you can help your company become more responsive and avoid compliance problems. But the biggest payback you’ll reap, however, is better data integrity—and because your data is important, improved data integrity is priceless.
About the Author
Mark Teter is the Chief Technology Officer at Advanced Systems Group. He is an internationally recognized authority on information technology who regularly advises IT organizations, vendors, and government agencies on a broad range of information management issues. Each year, Mark conducts dozens of seminars and training programs for corporate and government institutions. He sits on several financial industry advisory boards and has recently published Paradigm Shift: Seven Keys of Highly Successful Linux and Open Source Adoptions.
About Advanced Systems Group
Advanced Systems Group (ASG) is a Denver-based IT consulting, integration, and project management firm—fully equipped with a high-end computing facility that provides testing, benchmarks, demonstrations and an executive briefing center. Acknowledged by Computer Reseller News as one of the Top Ten Storage Solution Providers, ASG pursues active involvement in the industry, maintaining the highest level of engineering certifications with partners and the vendor community.
Call us at 800.894.3619 www.virtual.com Denver . Baton Rouge . Boise . Colorado Springs . Dallas . Houston . Los Angeles . New Orleans Oklahoma City . Orange County . Phoenix . Portland . Salt Lake City . San Diego . Seattle
©2012 Advanced Systems Group. All rights reserved.
