How to Migrate PHP, WordPress and Drupal Sites to New Servers
When it came time for my Ubuntu server upgrade, I initially tried to upgrade in place from Ubuntu 14.0x to Ubuntu 16.0x, but
do-release-upgrade failed me. And, Digital Ocean support warned me that it’s not really working well for their customers; other websites concur. I tried it on a couple of different droplets and met long slow debilitating failures. Snapshots and recovery are time consuming.
When it comes time for a major server upgrade and you can’t upgrade in place, here’s how to migrate your web applications to a new Ubuntu server.
This tutorial will highlight procedures for moving all your sites as quickly and easily as possible, with all their complexities. I’ll also cover areas that most people don’t think about when they prepare for server migration.
Moving Sites to Different Servers is Hard
When migrating servers, there are a number of components we have to move. Many people don’t realize the complexity that may occur.
Most people know about these tasks:
- The Apache web directories in /var/www
- The MySQL databases
- When ready, updating the DNS for domain names to point to the new server
Most people do not anticipate these lesser known complications:
- The Apache website configuration files (e.g. /etc/apache2/sites-available/*.conf)
- MySQL User Privileges. I’d never realized that when you dump all-databases to migrate a server and its sites, user privileges are left behind. Oops.
- SSL Certificates such as Certbot (formerly known as Let’s Encrypt). Now that Certbot has made running SSL much easier, we are beginning to run into the challenge of migrating existing certificates to new servers. It’s complicated.
- Cron files for background tasks
And there are the complexities buried within the tools we use that you may not be as familiar with. For example, I ran into tar’s complications with absolute and relative paths and had to find a workaround. And there are often changes in the new servers such as MySQL 5.7 running strict mode.
I’ve divided this migration tutorial into six parts (you can also navigate using the table of contents in the side column):
- Migrating the Apache Site Directories
- Migrating the Apache Website Configuration Files
- Migrating the MySQL Databases
- Updating Your DNS Records for Migration
- Migrating the Cron File
- Migrating Certbot Let’s Encrypt Certificates
Let’s get started!
1. Migrating the Apache Site Directories
First, let’s start on the source server and compress the website files. We’ll migrate the /var/www trees for our sites and their configuration files.
Packaging and Compressing Website Files
Moving Individual Sites
Here’s the code I’m using to compress an individual site tree:
$ cd ~ $ sudo tar --absolute-names -czvf ./onesite.tgz /var/www/onesite/
And, here’s what I use to capture my configuration files outside of the visible apache www area:
$ cd ~ $ sudo tar --absolute-names -czvf ~/vsecure.tgz /var/config/
Initially, I ran into this error:
tar: Removing leading `/' from member names
I solved it by using “–absolute-names” or “-P”, or “-C /”. You can also use the very verbose mode (-vv) in tar to track down problems:
$ sudo tar -czvvf ./onesite.tgz /var/www/onesite/
The target files were placed in my home directory:
$ ls onesite.tgz vsecure.tgz
Moving Multiple Sites
When I moved all of my WordPress site trees, I used this command:
$ sudo tar --absolute-names -czvf ./wpsites.tgz /var/www/ # This also worked $ sudo tar fcz ./wpsites.tgz -C / var/www
If we’re moving lots of sites, we may generate a very large file. Mine was about 1GB.
Transferring the Files to the New Server
If you are willing to place your new server account’s private key on the old server, at least temporarily, you can save some steps and use scp directly from the old server to the new server.
$ scp -r -P 22 -i ~/.ssh/id_privkey ~/*.tgz firstname.lastname@example.org:~ onesite.tgz 100% 27MB 27.0MB/s 00:01 vsecure.tgz 100% 1597 1.6KB/s 00:00
If you don’t want to place your private key on the old server, you can scp from the old server to your local system and then from your local system to the new server. There’s just extra steps and in the past I’ve had problems where my Mac will translate file ownership and permissions a bit differently than Ubuntu does.
Or, perhaps you’re using less secure password authentication, in which case you’ll be able to transfer from old server to new server pretty easily.
Extracting the Files on the New Server
Once at the new server, you can extract them. Because I used –absolute-names, I was able to extract into the / (root) directory and have them appear properly in /var/www/onesite:
$ sudo tar -zxvf ~/onesite.tgz --directory /
Similarly, I was able to export the configuration .tgz package to the /var/config directory:
$ sudo tar -zxvf ~/vsecure.tgz --directory /
Trying to extract these in the target directories actually created unwanted directory paths and I had to remove them and start over. Extract to root (/).
2. Migrating the Apache Website Configuration Files
Now, let’s move the configuration (.conf) files. Using scp, I can quickly move them over to the new server:
$ scp -P 22 -i ~/.ssh/id_privkey /etc/apache2/sites-available/*.conf email@example.com:~ 000-yoursite.conf 100% 789 0.8KB/s 00:00 003-yoursite-le-ssl.conf 100% 828 0.8KB/s 00:00
On the new server, this moves them into the Apache sites-available folder:
$ sudo mv *.conf /etc/apache2/sites-available/
I use a sudo privileged account by habit which requires I move everything first to my home directory when using scp. Perhaps, if you use a root account for scp at the destination server, you can scp directly into that target directory and save steps.
3. Migrating the MySQL Databases
3a. Moving the MySQL Databases
If both your servers are on a secure, internal network, you can use this awesome single line piped MySQL dump. It impressively dumps any number of databases over an ssh connection directly into your new server’s database:
mysqldump -uroot -p'sourcesqlpwd' -a --databases db_a db_b db_c | ssh -p 22 -i ~/.ssh/id_privkey firstname.lastname@example.org "mysql -uroot -p'destmysqlpwd'"
This was my favorite find (thanks WisdmLabs). It worked super well for me. But, if you prefer to be more secure, it’s best to dump the databases and transfer the packages via scp. Something like this dumps the databases:
$ sudo mysqldump -u root -p --all-databases > all_databases.sql
Then, you can use scp to transfer them to the new server:
$ scp -P 22 -i ~/.ssh/id_privkey ./all_databases.sql email@example.com:~
3b. Migrating the MySQL User Privileges
Most people don’t realize that dumping all the databases doesn’t include the user privileges. I found this very helpful script below that builds a file of your privileges so you can import them to the new server.
$ mysql -B -N -uroot -p'sourcemysqlpwd' -e "SELECT CONCAT('\'', user,'\'@\'', host, '\'') FROM user WHERE user != 'debian-sys-maint' AND user != 'root' AND user != ''" mysql > mysql_all_users.txt $ while read line; do mysql -B -N -uroot -p'sourcemysqlpwd' -e "SHOW GRANTS FOR $line"; done < mysql_all_users.txt > mysql_all_users_sql.sql $ sed -i 's/$/;/' mysql_all_users_sql.sql
Ultimately, the file looks like this:
GRANT USAGE ON *.* TO 'db1'@'localhost' IDENTIFIED BY PASSWORD '*XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'; GRANT ALL PRIVILEGES ON `db1`.* TO 'db_user'@'localhost'; GRANT USAGE ON *.* TO 'db_2'@'localhost' IDENTIFIED BY PASSWORD '*XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'; GRANT ALL PRIVILEGES ON `db2`.* TO 'db_2'@'localhost'; ...
It’s also a nice find. Then, I was able to scp that file over to the new server:
$ scp -P 22 -i ~/.ssh/id_privkey ./mysql_all_users_sql.sql firstname.lastname@example.org:~ mysql_all_users_sql.sql 100% 5252 5.1KB/s 00:00
Next, you import the file:
$ mysql -u root -p < mysql_all_users_sql.sql
Pretty much every SQL server backup and restore script on the web uses mysql-dump all-databases as if it’s a panacea but it leaves out vital user access privileges. Surprising to me.
4. Updating Your DNS Records for Migration
Updating Your Host Records
At my host, NameCheap, I just edited the A recorded to point to the new server’s ip address. And, I updated the CNAME for * or www to point to the server’s identity URL. I use something like wordpress.newserver.com or apps.newserver.com. I point my CNAMEs to that.
I also point all my domains for sale at a parking page. Fortunately, I discovered that NameCheap has a bulk update UX now. This worked very well for migrating the IP addresses and CNAMES of 100 domain names.
Reactivating the Apache Configuration Files
Either before or after you update your DNS records, you need to re-enable your Apache sites. Basically I did this one by one:
$ ls /etc/apache2/sites-available
Then, for each site, I just did this:
$ sudo a2ensite <sitename>.conf # after all of the sites enabled on the new server $ sudo systemctl reload apache2
5. Migrating the Cron File
I used cut and paste for this. On the source server, I listed out the current file and copied it:
$ sudo crontab -l
Then, I just copied what came up below.
# m h dom mon dow command 15 2 * * 1 certbot renew --no-self-upgrade */3 * * * * /var/www/yiiapp/yii daemon/frequent */15 * * * * /var/www/yiiapp/yii daemon/quarter 0 * * * * /var/www/yiiapp/yii daemon/hourly 15 1 * * * /var/www/yiiapp/yii daemon/overnight 15 3 * * 5 /var/www/yiiapp/yii daemon/weekly
Then, I pasted it into the new server. You can verify it here.
$ sudo crontab -e # and then paste
At first, I chose to comment them out on the new server until I could ensure the DNS had migrated and the sites were running.
6. Migrating Certbot Let’s Encrypt Certificates
I’m in the process of working on a tutorial describing how to transfer and migrate your Let’s Encrypt Certbots to a new server. I’ll link to that soon.