Data Recovery - Best Practices
Content
Data Recovery Best Practices
Building a responsible backup and recovery system for your databases
A White Paper by Stephen Wynkoop Microsoft SQL Server MVP Founder The SQL Server Worldwide User’s Group
Data Recovery Best Practices – White Paper
Table of Contents
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 Why Backup is Necessary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 Full Database Recovery and Restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 Point-in-Time Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 Specific Transaction Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 Disaster Planning and Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 How Much Data Can You Afford to Lose? . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 About Transaction Logs and Keeping Historical Backups . . . . . . . . . . . . . . . . .3 Optimize Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 Plan for the Future, Don’t Fail to Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 Pointers to Keep in Mind for the Restoration Process Planning . . . . . . . . . . . .6 Disk to Disk = Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 Being Prepared for Recovery – The Backup Process . . . . . . . . . . . . . . . . . . . . . . . . . . .8 Summary/Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
Data Recovery Best Practices – White Paper
Introduction
When people think about Data Recovery, they think largely about backups and the actual act of both backing up the database and associated files and the process of restoring those files to the server. Without a solid plan in place that reviews the best approaches for setting up a plan, testing the plan and executing on that plan, you can quickly get into trouble. Planning for data recovery is more than just making sure your database is backed up. You need to understand how the process works, you need to have the right tools in place, and you need to have practice in using those tools. When the time comes to restore information to your production systems, you won’t want to be learning about how things work; you’ll want to get the job done as quickly as possible. There are many different components to a competent backup and recovery plan. In addition, there are many types of recovery plans available. Each of these different approaches may suit what you need for different types of issues that arise. You need to understand and plan for the differences between a full system restore and a point-in-time recovery. At the most precise level, you may even need to recover a specific transaction or data element. As you can imagine, understanding each of these, and how to execute on them, is critical to managing your data resources. In this white paper, we’ll explain each of these items, talk about what they mean and how they apply. We’ll also provide key planning points, and investigate how some different tools can help you accomplish these tasks.
Why Backup is Necessary
Backup provides you a recovery avenue when things go wrong. Hard drives fail, connections between systems fail and have to be restored, people make mistakes, all causing the need to recover at different levels.
NOTE
Backup processes and planning often revolve around the unsettling question of “how much can you afford to lose.” This is because you need to determine the frequency that you backup the transaction logs and databases, while at the same time paying attention to disk and/or tape space constraints. In addition, you’ll need to decide how you store backups, how many days of backups you retain and lastly, whether you want to maintain a sub-set of your backups off-site. Remember, in the worst possible scenario, if your backups are stored right next to your computer and there is a fire, the backups will go up in smoke too, right along with your computer. It’s important to have at least a skeleton off-site storage plan.
Keep in mind that responsible planning and management of your systems includes more than just backing up to a device and then restoring the database should systems fail. There are really three different types of recoveries you may be faced with, and several shades of gray between each of these. The major restore options are explained in the next three sections.
1
Data Recovery Best Practices – White Paper
FULL
DATABASE RECOVERY AND RESTORE
Full database backup and restore is what many people think of when they consider their backup strategy, and it’s the most drastic recovery path. This requires that you restore the most recent full database backup, and then apply all transaction logs that were backed up after that backup was taken. At the end of the process, your database will be in the same state it was as of the time of the last transaction log backup. Your data loss in this scenario will amount to that information that was not in the most recent transaction log backup.
POINT-IN-TIME
RECOVERY
Point-in-Time Recovery lets you recover, typically using transaction logs, to a specific time when you know the data was valid. This typically means you’ve discovered data issues after some time has passed. This usually means restoring the most recent backup, then applying transaction logs to the system up to just before the time when you know the data began to have issues. This lets you restore to a known good point in time. You can also perform differential database backups—these allow you to backup just the changes since the last backup was performed.
SPECIFIC
TRANSACTION RECOVERY
Transaction-based recovery is typically done in one of two different ways. First, your application can be managing transactions in the code by starting transactions, doing a bit of work, and then committing the work to the database with an end transaction call. If the transaction fails, it can be rolled-back, putting the information in the tables into the same state that it was in when the transaction was started. In addition, if the server were forced to restart during the transaction, SQL Server would roll back the transaction, putting the database into a known state—the values representing the values in the database at the time that the transaction was started. It’s also possible to roll back specific transactions (either literal transactions or merely changes to the data in the database) using third party tools. Lumigent’s Log Explorer product will let you peruse data changes, along with a whole host of information about those changes. This includes who made the change, what was the value before the change, etc. From this information, the tool will allow you to restore specific values, in essence rolling-back data modifications, even without the benefit of transactions.
Disaster Planning and Recovery
Disaster planning must take into account the types of recovery you want and need to support. You need to have a written plan, and you need to test the plan to make sure it addresses the different facets of any restore process. Remember, you won’t control when the process is needed. You want to be able to provide for how the process is done, what the expected outcome will be, and how to provide for support for these processes up to the time you need the recovery efforts to begin. What follows are some guidelines to thinking through your plan.
2
Data Recovery Best Practices – White Paper
HOW
MUCH DATA CAN YOU AFFORD TO LOSE?
As mentioned above, this is perhaps the most telling question you need to be sure you can answer. If you can’t lose a single transaction or a single change, your disaster planning and recovery efforts will need to include fail-over systems. This means you’ll be looking into clustering solutions, and you’ll be working with hot stand-by systems and real-time replication and archival solutions. These tend to lead to rather large budgets, so depending on your budget, “no data loss whatsoever” may not be a reality. That said, and assuming that you’re not looking into a clustered solution, you’ll need to know how much data you have in the actual database(s) you’re backing up, and you’ll need to know what size the transaction logs get to as the database is used. One of the most common approaches to backups, and one which allows for only a maximum one hour data loss window, is to backup the database nightly and the transaction logs hourly. Typically, you’ll set up SQL Server to keep a specific number of days worth of backup as archive. When you set up this type of backup structure, you’ll tell SQL Server “Keep 14 days of backups, backup the database each morning at 3AM and the transaction logs every hour for all other times.” Keep in mind that, if you’re using this approach, you need to have disk (or tape, if you’re backing up directly to tape) space equal to more than 14 times the size of your database since you’ll be keeping 14 archival copies in the queue. In addition, you need to plan for enough space to support the 13 transaction log dumps. The size of transaction log dumps varies wildly and is entirely dependent on the volume of information processed by SQL Server.
ABOUT
TRANSACTION LOGS AND KEEPING HISTORICAL BACKUPS
Many people make the mistake of thinking that as long as they have several days of backups, they can restore to any point in time during those several days. It can be a painful lesson to learn that this may not be the case, depending on your archive solution. Consider the following backup policy:
• Nightly backups • Hourly transaction log dumps • Database backups are kept online for five days, then archived to a secondary source • Transaction logs are rotated to keep the most recent 24 hours available
At first glance, this is great. You can recover to the last database backup, then apply the transaction logs to recover beyond that to the current state, or any time in between. If your system fails, and you recognize the failure within 24 hours of the last database backup, you’re correct in saying that you’re covered. Keep in mind, though, that if you have the possibility of needing to restore further back than that last database backup, you will be faced with data loss.
3
Data Recovery Best Practices – White Paper
This situation comes from the fact that you’ll restore the database from three days ago (as an example), which would be available online. But if you follow the history configuration for the transaction logs, you’ll find that the transaction logs are only available for the last 24 hours. This would mean you wouldn’t be able to move forward beyond that three-day old backup. You’d be restoring to that point and no further in the database. Keep this in mind as you architect your recovery solution. You need to consider your transaction log rotation schedule in addition to your backup rotation schedule. It all goes back to “how much data can you lose” and how far back are you willing to support in the need to recover that data? If the answer is that you need to be able to restore to a point in time during that five day window (from our example of five days online backup storage), you’ll need to also be storing five days of hourly transaction logs.
OPTIMIZE
AVAILABILITY
When you’re building out your plan, be sure to consider the impact on your users and those dependent on access to the database. If you’re in a situation that requires access at all times (financial applications are an example of this), you’ll want to look not only at a recovery plan, but also a failover plan. Failover will protect you in cases where a hard drive fails, or other instances where the server goes offline, taking your database systems with it. Failover typically includes clustered server capabilities, where you have more than one server working against a given set of data. If one server does fail, the other server is able to pick up where the failing server left off and the user experience is largely unaffected by the downtime.
NOTE
In a clustered environment, if a failover situation does occur, the application working against the database may need to be restarted to “see” the recovery server. Typically this is merely a restart of the application, or a reconnection to the web site or other resource working with your SQL Server. The important point here is that your recovery plan in a clustered environment should include several phases: • Bring the applications back online against the recovery server(s). • Take the server offline that is down and/or experiencing trouble. • Correct the issue with the original server. • Bring the original server back into the cluster to begin supporting the cluster again.
On the other hand, if you don’t need to make sure you have full access, all the time to the server, you can work out your plan so you know exactly what you need to do to recover your system, get people back working again in the shortest period of time, and how to address problems that may arise during that process.
4
Data Recovery Best Practices – White Paper
PLAN
FOR THE FAILURE, DON’T FAIL TO PLAN
Executing on your plans will be key—below you’ll find different things you’ll need to consider and work through as you design your recovery plans.
Backup Procedure Checks • Are they working?
- Check your scheduled task’s history entries. - Check the backup directory for the related database and transaction log dump files.
• Are they archiving appropriate numbers of past copies of the backups?
- Check the directory for past copies of the database and transaction log dump files—if you’re expecting a rotation of files, perhaps several days worth or more of these files, make sure they’re in the directory.
• Are the transaction logs backing up on time?
- Check the job history. - Check the directory that is used for the backups; make sure the transaction logs dumps are there.
TIP
When you review the backup file sizes, if you see that your transaction log dump files are rather large, you may want to consider making the time between transaction log backups smaller. Remember, in the case of a restore, you’ll be restoring the database, then the transaction logs to get caught up. If the transaction logs are large, this can mean that you are running a large number of transactions, which translates into losing a large number of transactions (since the last transaction log backup) between backup processes. • If you’re using SQL LiteSpeed, try running LiteSpeed with the debug option turned on. This will enable you to see the various messages as the backups are performed. You’ll need to manually run the backups to be able to review/see these messages. Alternatively, you can have the output of the backup operations directed to a log file, external to SQL Server. You can then review this log file for any issues that may arise. For more information, read about the @logfile option with LiteSpeed.
Perhaps the most important check is whether your backup files can be restored. It sounds silly, but there are a large number of people that can attest to the fact that they thought they were successfully backing up and were protected from disaster. When it came time to recover and restore their files from backup, they found that they didn’t know how (didn’t know the commands), the backup files were either missing or corrupt, or they couldn’t find the correct hardware/software combination to get the files back onto the server for restoration. (This last point is one that pertains largely to tape backup systems.)
5
Data Recovery Best Practices – White Paper
Once you have your backup files, you need to make absolutely certain they are valid, that you know how to restore them, and that the restoration process is documented. Remember, if you’re encrypting or password protecting your backups, the password should be stored somewhere safe, but somewhere where the right person knows how to get to it. If you’re away on vacation and the system must be restored, there should be a procedure that can be followed to complete the restoration, complete with passwords. Keep in mind that just because you may not be taking vacations, this doesn’t mean you don’t need a plan. When things go wrong, the last thing you want to be doing is trying to remember the steps you need to follow to get your systems back online. Take the time now to write out the steps…then practice them.
HERE
ARE SOME POINTERS TO KEEP IN MIND FOR THE RESTORATION PROCESS PLANNING:
• Have a written plan with steps to follow for the restoration and recovery process. One very important thought on this topic has surfaced given the recent mass power outage in New York City and the surrounding areas. If you consider that, if you were the DBA, the phones and many transportation systems were out of commission, and you quickly see that you can’t count on getting back to the office to address issues. While this is extreme, it does point out that it’s possible that whoever happens to be in the office at the time a critical issue arises needs to be able to address that issue. You need to have a written plan. • Try performing your restores against a second server. Make sure you know the process and that you’ve gone through the steps of restoring the database, checking user permissions, applying transaction logs. • If you’re working in a clustered environment, run through a test with a failed node. Note of course that unless you have an extra clustered environment this can be tricky relative to downtime. Make sure you have a planned maintenance window and that you’re prepared for issues that may arise. While this will take some meticulous planning to avoid complications, all the planning and studying to understand the failover technologies will pay off—not just in the dry run, but in the real thing when the knowledge is needed most.
Disk to Disk = Best Practices
You have several options when considering the actual approach to backing up your system, especially as it relates to how you’ll store the backups, how you make them available for restores, and how you archive those backups. Typically, you can expect your backups to be needed for a restoration process within a reasonably short time. This is because backups are used to recover a system after a system failure—not to “go back in time” to see data. This is an important distinction because you’ll want to make sure your most recent backups are both the most protected and the most readily available. As a general rule of thumb, you’ll find that disk-to-disk backup is a much better solution than tape-based alternatives when it comes to recovery options and processes. Some of the benefits of this approach include:
6
Data Recovery Best Practices – White Paper
• Speed – with no tape transfer process to work with, you can access your database and transaction log backups immediately, providing a much faster path to recovery. • Additional recovery options – you can use products like Lumigent’s Log Explorer to work with the transaction logs, making transaction and specific data element recovery possible. This may be possible with tape backup, but would force a restore to your server or other location. • More reliable data storage medium – since you’re backing up to disk, you stand a better chance of not having the media go “bad” for your backups. That said, of course, make sure you’re backing up your backup devices, just in case. Keep in mind too that the “Acts of God” issues still remain—if you’re backing up to the disk on the same server that has your SQL Server, or you’re backing up to another server physically located near your SQL Server, you can still be in danger of not being able to recover from fire or other catastrophic disaster. For this reason, it’s good to keep archive copies (perhaps weekly, for example) off-site as a last-step recovery mechanism.
By backing up to disk, and keeping those backups online and available, you are able to use world class tools to quickly provide recovery options. Time is of the essence when you’re working to bring systems or data elements back online. Backing up to tape requires locating the tape, restoring to your server—both of which require time and introduce variables that can stand in the way of your recovery process’ success. If given a choice, it’s always a better solution to backup to disk. The table below shows some examples and recovery approaches you can employ with this type of system in place, based on the scenario you’re facing.
Recover a database
Restore the database; restore the logs, in order, from the point in time of the last backup. The resulting system will include all updates up to the time of the more recent transaction log backup. If you only want to recover to a specific point in time, determine which log file occurs closest to the point in time before your target time period. Restore the database, restore the log files up to that point.
Recover a specific data element change
Using Log Explorer, you can review the transaction logs, locate the change that was in error and restore the data to the value prior to the change. Restore your database and log files to a new, temporary database. From this database, you can copy the lost table back to the production database. Alternate solution, use Lumigent’s Log Explorer product to recover the lost table—recovery is possible for DROPped or TRUNCATEd tables, depending on your transaction logs.
Recover a dropped table
7
Data Recovery Best Practices – White Paper
Being Prepared for Recovery—The Backup Process
By utilizing disk-based backup procedures, you can optimize your responsiveness and available up time to support the recovery methods you’ll need. By using the right tools, you will have a full circle of options when it comes to restoring and recovering from system and database issues. Backing up your information, and how you do it is just as important as having the tools and knowledge available to you to recover your data. Backing up your data with tools or technologies that can become faulty or cause time delays in your recovery cycles are simply not good practice. A very significant tool you can use to optimize your system—both on the backup and recovery sides of the equation is the SQL LiteSpeed product from DBassociatesIT. The product offers fast, non-CPU-intensive, encrypted and compressed backups. One objection to backing up to disk has been the amount of disk space required to support a solid recovery model. With LiteSpeed’s compression technologies, you’ll not have to use thirdparty archive and compression utilities, and you can save drastically on the disk space you need to store and manage your database and transaction log backups. LiteSpeed runs just like the native backup routines in SQL Server and syntax is nearly identical to native backup options in all but just a few new commands. In addition, you can address the security issues associated with traditional backups by encrypting your database and transaction log backups with true encryption that protects the whole of your backup set. To be best prepared, set up a backup server—the destination for your backups. Install a good amount of disk space and use this as the destination for your backups. Don’t store the backups on the same drive as your databases. This is a solution that would provide no recovery path when the disk fails.
Summary/Conclusion
There is much to consider as you build out your backup, restore and recovery plans. It’s more than the ability to simply restore your database; you need to manage the recovery options and make sure you have all available options available to you. Be sure to write out your plan. Test the plan, practice the plan, and make sure others that may be in contact with the servers in your absence are also aware of and familiar with your plans. While restoration of a single point in time transaction isn’t something you need to train everyone one, you should consider training on full system restores, transaction log restores and how to work with the backup media you use. Use 3rd party tools as appropriate to make sure your systems are both optimized and providing the highest level of functionality you need. Having too many options is just not possible when the users are screaming, the boss is sweating and you’re in the hot seat to get things right again with your database server.
8
Data Recovery Best Practices – White Paper
IF
YOU’RE INTERESTED IN MORE INFORMATION ON EITHER OF THE PRODUCTS MENTIONED, YOU CAN VISIT :
Lumigent Technologies, Log Explorer, http://www.lumigent.com DBassociatesIT, SQL LiteSpeed, http://www.dbassociatesit.com About Stephen Wynkoop Stephen Wynkoop is the founder of The SQL Server Worldwide Users Group (www.sswug.org) where he writes a daily database column and newsletter, and a Microsoft SQL Server MVP. Stephen is a best-selling SQL Server author and a well-known speaker at technical conferences. Stephen first started working with SQL Server when it was first introduced in 1993 and has worked with SQL Server ever since. In addition, Stephen has authored online and offline columns, books, and other references on Office Development Technologies, web site design and deployment technologies and Microsoft Access. To contact Stephen, email [email protected].
9
Lumigent Technologies, Inc. is the leader in enterprise data auditing solutions for organizations that need to reduce risk associated with regulatory compliance and the use of corporate data assets.
Lumigent Technologies, Inc. 289 Great Road Acton, MA 01720 USA Toll Free Phone E-mail 1 866-LUMIGENT 1 866-586-4436 1 978-206-3700 [email protected]
www.lumigent.com
Copyright © 2003 Lumigent Technologies, Inc. All rights reserved. Lumigent, the Lumigent logo and Log Explorer are registered trademarks or trademarks of Lumigent Technologies, Inc. All other names and marks are property of their respective owners.
Building a responsible backup and recovery system for your databases
A White Paper by Stephen Wynkoop Microsoft SQL Server MVP Founder The SQL Server Worldwide User’s Group
Data Recovery Best Practices – White Paper
Table of Contents
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 Why Backup is Necessary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 Full Database Recovery and Restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 Point-in-Time Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 Specific Transaction Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 Disaster Planning and Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 How Much Data Can You Afford to Lose? . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 About Transaction Logs and Keeping Historical Backups . . . . . . . . . . . . . . . . .3 Optimize Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 Plan for the Future, Don’t Fail to Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 Pointers to Keep in Mind for the Restoration Process Planning . . . . . . . . . . . .6 Disk to Disk = Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 Being Prepared for Recovery – The Backup Process . . . . . . . . . . . . . . . . . . . . . . . . . . .8 Summary/Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
Data Recovery Best Practices – White Paper
Introduction
When people think about Data Recovery, they think largely about backups and the actual act of both backing up the database and associated files and the process of restoring those files to the server. Without a solid plan in place that reviews the best approaches for setting up a plan, testing the plan and executing on that plan, you can quickly get into trouble. Planning for data recovery is more than just making sure your database is backed up. You need to understand how the process works, you need to have the right tools in place, and you need to have practice in using those tools. When the time comes to restore information to your production systems, you won’t want to be learning about how things work; you’ll want to get the job done as quickly as possible. There are many different components to a competent backup and recovery plan. In addition, there are many types of recovery plans available. Each of these different approaches may suit what you need for different types of issues that arise. You need to understand and plan for the differences between a full system restore and a point-in-time recovery. At the most precise level, you may even need to recover a specific transaction or data element. As you can imagine, understanding each of these, and how to execute on them, is critical to managing your data resources. In this white paper, we’ll explain each of these items, talk about what they mean and how they apply. We’ll also provide key planning points, and investigate how some different tools can help you accomplish these tasks.
Why Backup is Necessary
Backup provides you a recovery avenue when things go wrong. Hard drives fail, connections between systems fail and have to be restored, people make mistakes, all causing the need to recover at different levels.
NOTE
Backup processes and planning often revolve around the unsettling question of “how much can you afford to lose.” This is because you need to determine the frequency that you backup the transaction logs and databases, while at the same time paying attention to disk and/or tape space constraints. In addition, you’ll need to decide how you store backups, how many days of backups you retain and lastly, whether you want to maintain a sub-set of your backups off-site. Remember, in the worst possible scenario, if your backups are stored right next to your computer and there is a fire, the backups will go up in smoke too, right along with your computer. It’s important to have at least a skeleton off-site storage plan.
Keep in mind that responsible planning and management of your systems includes more than just backing up to a device and then restoring the database should systems fail. There are really three different types of recoveries you may be faced with, and several shades of gray between each of these. The major restore options are explained in the next three sections.
1
Data Recovery Best Practices – White Paper
FULL
DATABASE RECOVERY AND RESTORE
Full database backup and restore is what many people think of when they consider their backup strategy, and it’s the most drastic recovery path. This requires that you restore the most recent full database backup, and then apply all transaction logs that were backed up after that backup was taken. At the end of the process, your database will be in the same state it was as of the time of the last transaction log backup. Your data loss in this scenario will amount to that information that was not in the most recent transaction log backup.
POINT-IN-TIME
RECOVERY
Point-in-Time Recovery lets you recover, typically using transaction logs, to a specific time when you know the data was valid. This typically means you’ve discovered data issues after some time has passed. This usually means restoring the most recent backup, then applying transaction logs to the system up to just before the time when you know the data began to have issues. This lets you restore to a known good point in time. You can also perform differential database backups—these allow you to backup just the changes since the last backup was performed.
SPECIFIC
TRANSACTION RECOVERY
Transaction-based recovery is typically done in one of two different ways. First, your application can be managing transactions in the code by starting transactions, doing a bit of work, and then committing the work to the database with an end transaction call. If the transaction fails, it can be rolled-back, putting the information in the tables into the same state that it was in when the transaction was started. In addition, if the server were forced to restart during the transaction, SQL Server would roll back the transaction, putting the database into a known state—the values representing the values in the database at the time that the transaction was started. It’s also possible to roll back specific transactions (either literal transactions or merely changes to the data in the database) using third party tools. Lumigent’s Log Explorer product will let you peruse data changes, along with a whole host of information about those changes. This includes who made the change, what was the value before the change, etc. From this information, the tool will allow you to restore specific values, in essence rolling-back data modifications, even without the benefit of transactions.
Disaster Planning and Recovery
Disaster planning must take into account the types of recovery you want and need to support. You need to have a written plan, and you need to test the plan to make sure it addresses the different facets of any restore process. Remember, you won’t control when the process is needed. You want to be able to provide for how the process is done, what the expected outcome will be, and how to provide for support for these processes up to the time you need the recovery efforts to begin. What follows are some guidelines to thinking through your plan.
2
Data Recovery Best Practices – White Paper
HOW
MUCH DATA CAN YOU AFFORD TO LOSE?
As mentioned above, this is perhaps the most telling question you need to be sure you can answer. If you can’t lose a single transaction or a single change, your disaster planning and recovery efforts will need to include fail-over systems. This means you’ll be looking into clustering solutions, and you’ll be working with hot stand-by systems and real-time replication and archival solutions. These tend to lead to rather large budgets, so depending on your budget, “no data loss whatsoever” may not be a reality. That said, and assuming that you’re not looking into a clustered solution, you’ll need to know how much data you have in the actual database(s) you’re backing up, and you’ll need to know what size the transaction logs get to as the database is used. One of the most common approaches to backups, and one which allows for only a maximum one hour data loss window, is to backup the database nightly and the transaction logs hourly. Typically, you’ll set up SQL Server to keep a specific number of days worth of backup as archive. When you set up this type of backup structure, you’ll tell SQL Server “Keep 14 days of backups, backup the database each morning at 3AM and the transaction logs every hour for all other times.” Keep in mind that, if you’re using this approach, you need to have disk (or tape, if you’re backing up directly to tape) space equal to more than 14 times the size of your database since you’ll be keeping 14 archival copies in the queue. In addition, you need to plan for enough space to support the 13 transaction log dumps. The size of transaction log dumps varies wildly and is entirely dependent on the volume of information processed by SQL Server.
ABOUT
TRANSACTION LOGS AND KEEPING HISTORICAL BACKUPS
Many people make the mistake of thinking that as long as they have several days of backups, they can restore to any point in time during those several days. It can be a painful lesson to learn that this may not be the case, depending on your archive solution. Consider the following backup policy:
• Nightly backups • Hourly transaction log dumps • Database backups are kept online for five days, then archived to a secondary source • Transaction logs are rotated to keep the most recent 24 hours available
At first glance, this is great. You can recover to the last database backup, then apply the transaction logs to recover beyond that to the current state, or any time in between. If your system fails, and you recognize the failure within 24 hours of the last database backup, you’re correct in saying that you’re covered. Keep in mind, though, that if you have the possibility of needing to restore further back than that last database backup, you will be faced with data loss.
3
Data Recovery Best Practices – White Paper
This situation comes from the fact that you’ll restore the database from three days ago (as an example), which would be available online. But if you follow the history configuration for the transaction logs, you’ll find that the transaction logs are only available for the last 24 hours. This would mean you wouldn’t be able to move forward beyond that three-day old backup. You’d be restoring to that point and no further in the database. Keep this in mind as you architect your recovery solution. You need to consider your transaction log rotation schedule in addition to your backup rotation schedule. It all goes back to “how much data can you lose” and how far back are you willing to support in the need to recover that data? If the answer is that you need to be able to restore to a point in time during that five day window (from our example of five days online backup storage), you’ll need to also be storing five days of hourly transaction logs.
OPTIMIZE
AVAILABILITY
When you’re building out your plan, be sure to consider the impact on your users and those dependent on access to the database. If you’re in a situation that requires access at all times (financial applications are an example of this), you’ll want to look not only at a recovery plan, but also a failover plan. Failover will protect you in cases where a hard drive fails, or other instances where the server goes offline, taking your database systems with it. Failover typically includes clustered server capabilities, where you have more than one server working against a given set of data. If one server does fail, the other server is able to pick up where the failing server left off and the user experience is largely unaffected by the downtime.
NOTE
In a clustered environment, if a failover situation does occur, the application working against the database may need to be restarted to “see” the recovery server. Typically this is merely a restart of the application, or a reconnection to the web site or other resource working with your SQL Server. The important point here is that your recovery plan in a clustered environment should include several phases: • Bring the applications back online against the recovery server(s). • Take the server offline that is down and/or experiencing trouble. • Correct the issue with the original server. • Bring the original server back into the cluster to begin supporting the cluster again.
On the other hand, if you don’t need to make sure you have full access, all the time to the server, you can work out your plan so you know exactly what you need to do to recover your system, get people back working again in the shortest period of time, and how to address problems that may arise during that process.
4
Data Recovery Best Practices – White Paper
PLAN
FOR THE FAILURE, DON’T FAIL TO PLAN
Executing on your plans will be key—below you’ll find different things you’ll need to consider and work through as you design your recovery plans.
Backup Procedure Checks • Are they working?
- Check your scheduled task’s history entries. - Check the backup directory for the related database and transaction log dump files.
• Are they archiving appropriate numbers of past copies of the backups?
- Check the directory for past copies of the database and transaction log dump files—if you’re expecting a rotation of files, perhaps several days worth or more of these files, make sure they’re in the directory.
• Are the transaction logs backing up on time?
- Check the job history. - Check the directory that is used for the backups; make sure the transaction logs dumps are there.
TIP
When you review the backup file sizes, if you see that your transaction log dump files are rather large, you may want to consider making the time between transaction log backups smaller. Remember, in the case of a restore, you’ll be restoring the database, then the transaction logs to get caught up. If the transaction logs are large, this can mean that you are running a large number of transactions, which translates into losing a large number of transactions (since the last transaction log backup) between backup processes. • If you’re using SQL LiteSpeed, try running LiteSpeed with the debug option turned on. This will enable you to see the various messages as the backups are performed. You’ll need to manually run the backups to be able to review/see these messages. Alternatively, you can have the output of the backup operations directed to a log file, external to SQL Server. You can then review this log file for any issues that may arise. For more information, read about the @logfile option with LiteSpeed.
Perhaps the most important check is whether your backup files can be restored. It sounds silly, but there are a large number of people that can attest to the fact that they thought they were successfully backing up and were protected from disaster. When it came time to recover and restore their files from backup, they found that they didn’t know how (didn’t know the commands), the backup files were either missing or corrupt, or they couldn’t find the correct hardware/software combination to get the files back onto the server for restoration. (This last point is one that pertains largely to tape backup systems.)
5
Data Recovery Best Practices – White Paper
Once you have your backup files, you need to make absolutely certain they are valid, that you know how to restore them, and that the restoration process is documented. Remember, if you’re encrypting or password protecting your backups, the password should be stored somewhere safe, but somewhere where the right person knows how to get to it. If you’re away on vacation and the system must be restored, there should be a procedure that can be followed to complete the restoration, complete with passwords. Keep in mind that just because you may not be taking vacations, this doesn’t mean you don’t need a plan. When things go wrong, the last thing you want to be doing is trying to remember the steps you need to follow to get your systems back online. Take the time now to write out the steps…then practice them.
HERE
ARE SOME POINTERS TO KEEP IN MIND FOR THE RESTORATION PROCESS PLANNING:
• Have a written plan with steps to follow for the restoration and recovery process. One very important thought on this topic has surfaced given the recent mass power outage in New York City and the surrounding areas. If you consider that, if you were the DBA, the phones and many transportation systems were out of commission, and you quickly see that you can’t count on getting back to the office to address issues. While this is extreme, it does point out that it’s possible that whoever happens to be in the office at the time a critical issue arises needs to be able to address that issue. You need to have a written plan. • Try performing your restores against a second server. Make sure you know the process and that you’ve gone through the steps of restoring the database, checking user permissions, applying transaction logs. • If you’re working in a clustered environment, run through a test with a failed node. Note of course that unless you have an extra clustered environment this can be tricky relative to downtime. Make sure you have a planned maintenance window and that you’re prepared for issues that may arise. While this will take some meticulous planning to avoid complications, all the planning and studying to understand the failover technologies will pay off—not just in the dry run, but in the real thing when the knowledge is needed most.
Disk to Disk = Best Practices
You have several options when considering the actual approach to backing up your system, especially as it relates to how you’ll store the backups, how you make them available for restores, and how you archive those backups. Typically, you can expect your backups to be needed for a restoration process within a reasonably short time. This is because backups are used to recover a system after a system failure—not to “go back in time” to see data. This is an important distinction because you’ll want to make sure your most recent backups are both the most protected and the most readily available. As a general rule of thumb, you’ll find that disk-to-disk backup is a much better solution than tape-based alternatives when it comes to recovery options and processes. Some of the benefits of this approach include:
6
Data Recovery Best Practices – White Paper
• Speed – with no tape transfer process to work with, you can access your database and transaction log backups immediately, providing a much faster path to recovery. • Additional recovery options – you can use products like Lumigent’s Log Explorer to work with the transaction logs, making transaction and specific data element recovery possible. This may be possible with tape backup, but would force a restore to your server or other location. • More reliable data storage medium – since you’re backing up to disk, you stand a better chance of not having the media go “bad” for your backups. That said, of course, make sure you’re backing up your backup devices, just in case. Keep in mind too that the “Acts of God” issues still remain—if you’re backing up to the disk on the same server that has your SQL Server, or you’re backing up to another server physically located near your SQL Server, you can still be in danger of not being able to recover from fire or other catastrophic disaster. For this reason, it’s good to keep archive copies (perhaps weekly, for example) off-site as a last-step recovery mechanism.
By backing up to disk, and keeping those backups online and available, you are able to use world class tools to quickly provide recovery options. Time is of the essence when you’re working to bring systems or data elements back online. Backing up to tape requires locating the tape, restoring to your server—both of which require time and introduce variables that can stand in the way of your recovery process’ success. If given a choice, it’s always a better solution to backup to disk. The table below shows some examples and recovery approaches you can employ with this type of system in place, based on the scenario you’re facing.
Recover a database
Restore the database; restore the logs, in order, from the point in time of the last backup. The resulting system will include all updates up to the time of the more recent transaction log backup. If you only want to recover to a specific point in time, determine which log file occurs closest to the point in time before your target time period. Restore the database, restore the log files up to that point.
Recover a specific data element change
Using Log Explorer, you can review the transaction logs, locate the change that was in error and restore the data to the value prior to the change. Restore your database and log files to a new, temporary database. From this database, you can copy the lost table back to the production database. Alternate solution, use Lumigent’s Log Explorer product to recover the lost table—recovery is possible for DROPped or TRUNCATEd tables, depending on your transaction logs.
Recover a dropped table
7
Data Recovery Best Practices – White Paper
Being Prepared for Recovery—The Backup Process
By utilizing disk-based backup procedures, you can optimize your responsiveness and available up time to support the recovery methods you’ll need. By using the right tools, you will have a full circle of options when it comes to restoring and recovering from system and database issues. Backing up your information, and how you do it is just as important as having the tools and knowledge available to you to recover your data. Backing up your data with tools or technologies that can become faulty or cause time delays in your recovery cycles are simply not good practice. A very significant tool you can use to optimize your system—both on the backup and recovery sides of the equation is the SQL LiteSpeed product from DBassociatesIT. The product offers fast, non-CPU-intensive, encrypted and compressed backups. One objection to backing up to disk has been the amount of disk space required to support a solid recovery model. With LiteSpeed’s compression technologies, you’ll not have to use thirdparty archive and compression utilities, and you can save drastically on the disk space you need to store and manage your database and transaction log backups. LiteSpeed runs just like the native backup routines in SQL Server and syntax is nearly identical to native backup options in all but just a few new commands. In addition, you can address the security issues associated with traditional backups by encrypting your database and transaction log backups with true encryption that protects the whole of your backup set. To be best prepared, set up a backup server—the destination for your backups. Install a good amount of disk space and use this as the destination for your backups. Don’t store the backups on the same drive as your databases. This is a solution that would provide no recovery path when the disk fails.
Summary/Conclusion
There is much to consider as you build out your backup, restore and recovery plans. It’s more than the ability to simply restore your database; you need to manage the recovery options and make sure you have all available options available to you. Be sure to write out your plan. Test the plan, practice the plan, and make sure others that may be in contact with the servers in your absence are also aware of and familiar with your plans. While restoration of a single point in time transaction isn’t something you need to train everyone one, you should consider training on full system restores, transaction log restores and how to work with the backup media you use. Use 3rd party tools as appropriate to make sure your systems are both optimized and providing the highest level of functionality you need. Having too many options is just not possible when the users are screaming, the boss is sweating and you’re in the hot seat to get things right again with your database server.
8
Data Recovery Best Practices – White Paper
IF
YOU’RE INTERESTED IN MORE INFORMATION ON EITHER OF THE PRODUCTS MENTIONED, YOU CAN VISIT :
Lumigent Technologies, Log Explorer, http://www.lumigent.com DBassociatesIT, SQL LiteSpeed, http://www.dbassociatesit.com About Stephen Wynkoop Stephen Wynkoop is the founder of The SQL Server Worldwide Users Group (www.sswug.org) where he writes a daily database column and newsletter, and a Microsoft SQL Server MVP. Stephen is a best-selling SQL Server author and a well-known speaker at technical conferences. Stephen first started working with SQL Server when it was first introduced in 1993 and has worked with SQL Server ever since. In addition, Stephen has authored online and offline columns, books, and other references on Office Development Technologies, web site design and deployment technologies and Microsoft Access. To contact Stephen, email [email protected].
9
Lumigent Technologies, Inc. is the leader in enterprise data auditing solutions for organizations that need to reduce risk associated with regulatory compliance and the use of corporate data assets.
Lumigent Technologies, Inc. 289 Great Road Acton, MA 01720 USA Toll Free Phone E-mail 1 866-LUMIGENT 1 866-586-4436 1 978-206-3700 [email protected]
www.lumigent.com
Copyright © 2003 Lumigent Technologies, Inc. All rights reserved. Lumigent, the Lumigent logo and Log Explorer are registered trademarks or trademarks of Lumigent Technologies, Inc. All other names and marks are property of their respective owners.
