Here I migrate a GitLab installation from one server to another. The hostname and IP addresses will be different. Both were installed from the Omnibus version. However, currently the exact version numbers differ.
In order to migrate successfully, both versions must be exactly the same type and version. For example, both installed using the Omnibus version and exactly the same version number.
If you are unsure whether you installed the Omnibus version, you can confirm with the below command. This will fail if not. If it returns output similar to the below, you have the Omnibus version. It also shows me that currently have version 10.2.3 installed.
root@repo:~# gitlab-rake gitlab:env:info System information System: Ubuntu 16.04 Current User: git Using RVM: no Ruby Version: 2.3.5p376 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.1.0 Redis Version: 3.2.5 Git Version: 2.13.6 Sidekiq Version:5.0.4 Go Version: unknown GitLab information Version: 10.2.3 Revision: 3141105 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://gitlab.openit-uk.com HTTP Clone URL: https://gitlab.openit-uk.com/some-group/some-project.git SSH Clone URL: git@gitlab.openit-uk.com:some-group/some-project.git Using LDAP: no Using Omniauth: no GitLab Shell Version: 5.9.4 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
If you do not have the Omnibus version, consult the official documentation on how to migrate. You need to make sure the version you are going to and from are the same.
The Strategy
- Install GitLab on new server [10.7.3]
- Update old GitLab to the latest version [10.7.3]
- Backup old GitLab
- Transfer backup files to new host
- Restore from backup on new instance
Install GitLab
Install GitLab on the new server.
Update GitLab
This can simply be achieved by updating the system.
apt-get update apt-get dist-upgrade
GitLab should now be updated to the latest version. It should also have created an application backup in the below location as part of the process.
root@repo:~# ls -l /var/opt/gitlab/backups/ total 204 -rw------- 1 git git 71680 May 16 19:29 1526498956_2018_05_16_10.2.3_gitlab_backup.tar
Check the version is now the latest and the same as the origin server.
root@repo:~# gitlab-rake gitlab:env:info | grep "GitLab information" -A1 GitLab information Version: 10.7.3
Backup GitLab
To fully backup, you need to take both a configuration and a application backup. The configuration backup includes all of GitLabs configuration including the external URL. The application backup includes all of your repositories.
Configuration Backup
All GitLab configuration is inside /etc/gitlab. To backup this directory, run:
sudo sh -c 'umask 0077; tar -cf $(date "+etc-gitlab-%s.tar") -C / etc/gitlab'
This creates the following file in the current working directory.
root@repo:~# ls -la etc-gitlab-1526500078.tar -rw------- 1 root root 102400 May 16 19:47 etc-gitlab-1526500078.tar
With the following content.
root@repo:~# tar -tvf etc-gitlab-1526500078.tar drwxrwxr-x root/root 0 2017-12-09 23:32 etc/gitlab/ drwxr-xr-x root/root 0 2017-12-06 14:58 etc/gitlab/trusted-certs/ drwx------ root/root 0 2017-12-09 23:41 etc/gitlab/ssl/ -rw-r--r-- root/root 2025 2017-12-09 23:09 etc/gitlab/ssl/repo.pikedom.com.crt -rw-r--r-- root/root 1708 2017-12-09 23:08 etc/gitlab/ssl/repo.pikedom.com.key -rw-r--r-- root/root 2025 2017-12-09 23:39 etc/gitlab/ssl/gitlab.openit-uk.com.crt -rw-r--r-- root/root 1708 2017-12-09 23:41 etc/gitlab/ssl/gitlab.openit-uk.com.key -rw------- root/root 14906 2018-05-16 19:31 etc/gitlab/gitlab-secrets.json -rw------- root/root 72812 2017-12-09 23:32 etc/gitlab/gitlab.rb
Application Backup
By default, backups are kept in /var/opt/gitlab/backups.
root@repo:~# grep backup_path /etc/gitlab/gitlab.rb # gitlab_rails['manage_backup_path'] = true # gitlab_rails['backup_path'] = "/var/opt/gitlab/backups"
If you change this location, be sure to run the below after.
sudo gitlab-ctl reconfigure
To actually take the application backup, run the following.
sudo gitlab-rake gitlab:backup:create
This creates a backup in the following location.
root@repo:~# ls -ltr /var/opt/gitlab/backups/ total 204 -rw------- 1 git git 71680 May 16 19:29 1526498956_2018_05_16_10.2.3_gitlab_backup.tar -rw------- 1 git git 133120 May 16 20:14 1526501659_2018_05_16_10.7.3_gitlab_backup.tar
Host Backup (optional)
If rebuilding the machine and keeping the same IP, to avoid having to delete the host key entry in the ~/.ssh/know_hosts file, run the following to backup the SSH host keys.
sudo tar -cvf $(date "+hostkeys-%s.tar") $(grep HostKey /etc/ssh/sshd_config | grep -v ^'#' | awk '{print $2}')
Transfer
Lets zip and compress these files into a single archive for transfer. Although I don’t plan on restoring the host keys, here I zip and compress all three all the same.
/var/opt/gitlab/backups/1526501659_2018_05_16_10.7.3_gitlab_backup.tar /root/etc-gitlab-1526500078.tar /root/hostkeys-1526501148.tar
Zip and compress with the following.
sudo tar -cvzf ~/gitlab-backup.tar.gz /var/opt/gitlab/backups/1526501659_2018_05_16_10.7.3_gitlab_backup.tar /root/etc-gitlab-1526500078.tar /root/hostkeys-1526501148.tar
Use scp to copy the backup securely over the internet. Here I run the below from the new GitLab server to pull the above backup to the current working directory.
sudo scp -v root@old.dummydomains.org.uk:/root/gitlab-backup.tar.gz .
Restore
As I am mainly concerned about the Git repositories themselves, I will only be doing an application restore.
If you are moving to a new host with the same hostname etc., you should also restore the configuration backup to /etc/gitlab/. If you want everything to be the same but the hostname is different, you can manually amend anything such as external_url before running sudo gitlab-ctl reconfigure.
Application Backup Restore
Stop processes using the database.
sudo gitlab-ctl stop unicorn sudo gitlab-ctl stop sidekiq sudo gitlab-ctl status
Check they’re actually down.
sudo gitlab-ctl status | grep ^down
Move the application backup to the backups directory as defined in gitlab.rb and correct the owner if necessary.
sudo mv -v ~/temp/1526501659_2018_05_16_10.7.3_gitlab_backup.tar /var/opt/gitlab/backups/ sudo chown -v git:git /var/opt/gitlab/backups/1526501659_2018_05_16_10.7.3_gitlab_backup.tar
Then restore.
sudo gitlab-rake gitlab:backup:restore BACKUP=1526501659_2018_05_16_10.7.3
Now restart GitLab.
sudo gitlab-ctl start
And check for problems.
sudo gitlab-rake gitlab:check SANITIZE=true
If all was successful, you should be able to log into the new site as before with whatever users you had setup. If you had 2FA enabled, you may need to restore the gitlab-secrets.json.
Configuration Backup Restore
Exactly what you choose to restore from the /etc/gitlab/ directory is up to you. None maybe necessary. If any of your users had 2FA setup, you will need to restore the gitlab-secrects.json file or else they will lose their access. If unsure, you could always do a diff to compare the differences.
diff old-gitlab.rb new-gitlab.rb
If moving to a new host but the hostname and everything should remain the same, you can simply move the old directory out the way and replace with the old.
First, stop unicorn and sidekiq at a minimum.
sudo gitlab-ctl stop
Then restore the configuration directory.
mv -v /etc/gitlab{,.original}
mv -v ~/temp/etc/gitlab /etc/gitlab
Then start GitLab again and check.
sudo gitlab-ctl start sudo gitlab-rake gitlab:check SANITIZE=true
Fix any issue and then re-run if necessary. Try re-configuring if you have issues.
sudo gitlab-ctl reconfigure
If even the IP address remains the same as before and you want to prevent your users from seeing an SSH man-in-the-middle attack warning, you will also need to restore the HostKeys.
You need to replace the HostKeys defined in your sshd_config.
andy@repo:~$ grep HostKey /etc/ssh/sshd_config # HostKeys for protocol version 2 HostKey /etc/ssh/ssh_host_rsa_key HostKey /etc/ssh/ssh_host_dsa_key HostKey /etc/ssh/ssh_host_ecdsa_key HostKey /etc/ssh/ssh_host_ed25519_key
…with the ones from your backup.
andy@repo:~$ tar -tvf temp/hostkeys-1526501148.tar.gz -rw------- root/root 1679 2017-12-05 13:37 etc/ssh/ssh_host_rsa_key -rw------- root/root 672 2017-12-05 13:37 etc/ssh/ssh_host_dsa_key -rw------- root/root 227 2017-12-05 13:37 etc/ssh/ssh_host_ecdsa_key -rw------- root/root 399 2017-12-05 13:37 etc/ssh/ssh_host_ed25519_key
Rename the existing keys.
for i in $(grep ^HostKey /etc/ssh/sshd_config | awk '{print $2}'); do sudo mv -v $i $i.bkp-$(date +%s);done
Then restore the old ones in-place.
sudo tar -xvzf temp/hostkeys-1526501148.tar.gz -C /
Restart GitLab and check for errors.
sudo gitlab-ctl restart sudo gitlab-rake gitlab:check SANITIZE=true
Resources
https://www.fabian-keller.de/blog/migrating-a-gitlab-instance-to-another-server
https://www.tienle.com/2016/05-24/migrate-gitlab-to-new-server.html
https://stackoverflow.com/questions/31534293/how-to-migrate-gitlab-to-a-new-server
