If you are planning to migrate your Google App Engine application to the “new” High Replication Database scheme, then you probably know that the Google migration process won’t handle Blobstore files.
Here you will find some python scripts that will help you to move the Blobstore files from the old application to the new one, and correct all the references in the new database. As always, use at your own risk, don’t try to do anything without reading and understanding the scripts, otherwise you may loose your data permanently.
We did used these scripts in our very own application, so I can say it works. However, I had to edit the original scripts to make them more “generic”, remove my database references and names, etc.
I will try to describe the steps of the migration, but please don’t try to do anything before reading the whole article first.
This is not by any means intended to be a fully automatic migration, in fact, the steps needs to be manually executed one by one.
The scripts won’t migrate big files. We have migrated files up to 40Mb without problems, but if you have files bigger than that, it will probably fail. For example we tried a 300Mb file unsuccessfully, the reason is that the file needs to be downloaded from the old app to the new, and there is a 1 minute timeout for the app requests, so if the download takes more than that it will fail.
How does the script works
You will need to put the script in your applications, both the new (HRD) and the old one. It basically exposes four URLs and you will need to access two of them from the HRD application.
By doing this you will download all the blob files from the old application to the new one, the script will create a model that keeps a map between the old and new references.
The final step is to migrate the old references to the new one in the HRD databases.
All the references to blob files in your models must be of type blobstore.BlobReferenceProperty, if not you will need to adjust them first.
Modify the MODELS variable adding the models and blob info fields that you need to update
Replace myoldapp URLs with the actual URL of your old application
Synchronize the code on both apps
Open a browser and point it to http://mynewapp.appspot.com/mig/__migrateblobs. Of course, mynewapp must be replaced with your actual HRD application name. This will download the blobs from the old app to the new one. Because there is a timeout limitation you will need to call it many times. WARNING: You must make only one request at time, never call it from two browsers / tabs at the same time or you will mess the migration database!!!. So this step must be carefully executed, one request at time and wait for it to finish before calling the next one. Repeat this until you see the message Nothing to migrate!. This step can be very slow depending on how many files you have in the blobstore and the size of them, it might take a long time to get all the files downloaded. Be patient!
Once all the files are copied to the new app, point the browser to http://mynewapp.appspot.com/mig/__migratereferences. Should be quick, and it needs to be executed only one time. However doesn’t harm to call it more than once.
The last step, remove the migration code and URLs and synchronize the code again. You can also remove the mig_BlobMig database.