Wednesday, August 25, 2010

SharePoint 2010 Upgrade Failure - Executing DDL Script

I'm in the process of upgrading our SharePoint 2007 farm to 2010, using the DBAttach method. I created a Server 2008R2 VM, installed SharePoint 2010 and configured the farm with default settings.

I backed up our existing content database and restored it to the new server. Our content database is approximately 150 GB, so it's quite large. I ran Test-SPContentDatabase. This reported some missing features, but indicated that none would cause the upgrade to fail. Since this is a test environment, I decided to go ahead and upgrade the database.

I then executed Mount-SPContentDatabase to upgrade and attach the content database. The process reached 19.69% after an hour and appeared to hang. I was able to log into Central Admin and monitor the process as designed. After several hours of what appeared to be no activity, I ran Sp_Who2 in SQL Management Studio and found that I had two Spids listed as SUSPENDED. One was an INSERT and one was a SELECT. One was blocking the other. SQL was deadlocked. After 13 hours, the upgrade failed.

Reviewing the upgrade log files indicated several issues:

  1. The first issue I noticed is that the issues identified by Test-SPContentDatabase (missing features) as not being fatal errors were listed in the log file as "The missing feature may cause upgrade to fail. Please install any solution which contains the feature and restart upgrade if necessary." It would appear that there is disagreement between Test-SPContentDatabase and the actual upgrade process.
  2. Secondly, and more importantly, it appears that while executing step 84 of 128, a fatal exception occurred. The following entry was recorded:
Action 4.0.98.0 of Microsoft.SharePoint.Upgrade.SPContentDatabaseSequence failed.


[STSADM] [SPContentDatabaseSequence] [ERROR] [8/25/2010 3:50:59 AM]: Exception: Could not allocate space for object 'dbo.AllDocStreams' in database 'MOSS_WSS_Portal_Content' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.
Autogrow on the SQL log file was set to 10% and had a max file growth limit.  When the upgrade process began populating the AllDocStreams Content Column, two things happened:

  1. The larger the transaction log got, the longer it began taking to autogrow. Once the autogrow process approached two minutes, errors began.
  2. Next, the transaction log grew so large that my partition ran out of disk space. 
I believe this is happening because none of the transactions are being committed during the upgrade process; thus allowing a rollback in case of failure. Paradoxically, this very failsafe is actually causing it to fail in my case.

I started with a 140 GB content database and it completely filled my 300 GB partition. When performing the upgrade, you need to plan on having plenty of drive space available. Once I moved the database to a terrabyte partition, the error was resolved. I found that, in order to reliably upgrade a SharePoint 2007 content databases to 2010, you'll need to plan for allowing the content database to triple in size during the upgrade. If your content database is 100 GB, you'll need at least 300 GB for the log files to grow.

9 comments:

  1. Plese let me know how it went I had the excat same thing happen.

    ReplyDelete
  2. Hi,

    This is really important for me. I'm also trying to upgrade 2007 farm to 2010. If you can share your experience (procedures you followed and etc...) it would be really helpful.

    -Sithara-

    ReplyDelete
  3. Update:

    This was all related to a lack of available drive space. See my post here http://practicalsharepoint.blogspot.com/2010/09/sharepoint-2010-database-upgrade.html for more details.

    Once I had adequate drive space (at least triple the size of the content database) this issue was resolved.

    ReplyDelete
  4. Thanks for the post I'm encountering the exact same problem -- also at 19.69% complete.

    ReplyDelete
  5. I am trying to upgrade our sharepoint from MOSS 2007 to SharePoint 2010. It errors or stalls at 19.69%. This is at step 84 steps in this action is 128. Drive space is not an issue. Any other ideas?

    ReplyDelete
  6. Dan:

    All my experience points to problems with the log file expanding during the DDL steps, causing a drive space issue.

    Try running DBCC shrinkfile on the log, putting the db in simple recovery model and trying it again. I'd focus on the log file growth.

    ReplyDelete
  7. There are many Intranets solutions on the market, however if you are looking for a solid SharePoint based Intranet solution, check out SharePoint Implementeds' product. I think it's the best solution out for the price.They seem to have put a lot of thought into usability and filling in gaps that you would not know exsist in sharepoint until you start your implementation.

    They offer a turnkey solution which provides a custom Home Site, Department Sites and Project Sites, installation, configuration and training all under $10,000 and even have a source code option.

    One thing that I would love to see that they don't have now is a hosted solution

    You can get more details at http://sharepointimplemented.com/AwesomeIntranetGorilla.html

    ReplyDelete
  8. I'm running into this exact same problem as well in testing our upgrade process except it hangs at 18.67%. From the research I have done I have found that this is due to the fact that documents are getting moved from the dbo.AllDocVersions table to the dbo.AllDocStreams table which will cause them to occupy more space in SQL. So your growth is going to be in direct correlation with how many versions of documents you have within your site. The more document versions you have the greater the growth you will experience. We had one site with a large number of versions grow by 103% total (including the log) whereas another site with a lot less versions grew by 29% total. Still trying to find a way to reduce this growth as much as possible but so far I have been unsuccessful so if anyone knows it would be greatly appreciated if they could share the info.

    ReplyDelete
  9. $feature = Get-SPFeature | ? { $_.Id –eq “819e9287-263a-462d-90ff-48e1097867c1” }
    &feature.delete()

    ReplyDelete