Migrate GitLab Instance to new Host

Here I migrate a GitLab installation from one server to another. The hostname and IP addresses will be different. Both were installed from the Omnibus version. However, currently the exact version numbers differ.

In order to migrate successfully, both versions must be exactly the same type and version. For example, both installed using the Omnibus version and exactly the same version number.

If you are unsure whether you installed the Omnibus version, you can confirm with the below command. This will fail if not. If it returns output similar to the below, you have the Omnibus version. It also shows me that I currently have version 10.2.3 installed.

[email protected]:~# gitlab-rake gitlab:env:info

System information
System:         Ubuntu 16.04
Current User:   git
Using RVM:      no
Ruby Version:   2.3.5p376
Gem Version:    2.6.13
Bundler Version:1.13.7
Rake Version:   12.1.0
Redis Version:  3.2.5
Git Version:    2.13.6
Sidekiq Version:5.0.4
Go Version:     unknown

GitLab information
Version:        10.2.3
Revision:       3141105
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://gitlab.openit-uk.com
HTTP Clone URL: https://gitlab.openit-uk.com/some-group/some-project.git
SSH Clone URL:  [email protected]:some-group/some-project.git
Using LDAP:     no
Using Omniauth: no

GitLab Shell
Version:        5.9.4
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab-shell/hooks
Git:            /opt/gitlab/embedded/bin/git

If you do not have the Omnibus version, consult the official documentation on how to migrate. You need to make sure the version you are going to and from are the same.

The Strategy

  1. Install GitLab on new server [10.7.3]
  2. Update old GitLab [10.2.3] to the latest version [10.7.3]
  3. Backup old GitLab
  4. Transfer backup files to new host
  5. Restore from backup on new instance

Install GitLab

Install GitLab on the new server.

Update GitLab

This can simply be achieved by updating the system.

[email protected]:~# apt-get update
[email protected]:~# apt-get dist-upgrade

GitLab should now be updated to the latest version. It should also have created an application backup in the below location as part of the process.

[email protected]:~# ls -l /var/opt/gitlab/backups/
total 204
-rw------- 1 git git  71680 May 16 19:29 1526498956_2018_05_16_10.2.3_gitlab_backup.tar

Check the version is now the latest and the same as the origin server.

[email protected]:~# gitlab-rake gitlab:env:info | grep "GitLab information" -A1
GitLab information
Version:	10.7.3

Backup GitLab

To fully backup, you need to take both a configuration and a application backup. The configuration backup includes all of GitLabs configuration including the external URL. The application backup includes all of your repositories.

Configuration Backup

All GitLab configuration is inside /etc/gitlab. To backup this directory, run:

[email protected]:~# sh -c 'umask 0077; tar -cf $(date "+etc-gitlab-%s.tar") -C / etc/gitlab'

This creates the following file in the current working directory.

[email protected]:~# ls -la etc-gitlab-1526500078.tar
-rw------- 1 root root 102400 May 16 19:47 etc-gitlab-1526500078.tar

With the following content.

[email protected]o:~# tar -tvf etc-gitlab-1526500078.tar
drwxrwxr-x root/root         0 2017-12-09 23:32 etc/gitlab/
drwxr-xr-x root/root         0 2017-12-06 14:58 etc/gitlab/trusted-certs/
drwx------ root/root         0 2017-12-09 23:41 etc/gitlab/ssl/
-rw-r--r-- root/root      2025 2017-12-09 23:09 etc/gitlab/ssl/repo.pikedom.com.crt
-rw-r--r-- root/root      1708 2017-12-09 23:08 etc/gitlab/ssl/repo.pikedom.com.key
-rw-r--r-- root/root      2025 2017-12-09 23:39 etc/gitlab/ssl/gitlab.openit-uk.com.crt
-rw-r--r-- root/root      1708 2017-12-09 23:41 etc/gitlab/ssl/gitlab.openit-uk.com.key
-rw------- root/root     14906 2018-05-16 19:31 etc/gitlab/gitlab-secrets.json
-rw------- root/root     72812 2017-12-09 23:32 etc/gitlab/gitlab.rb

Application Backup

By default, backups are kept in /var/opt/gitlab/backups.

[email protected]:~# grep backup_path /etc/gitlab/gitlab.rb
# gitlab_rails['manage_backup_path'] = true
# gitlab_rails['backup_path'] = "/var/opt/gitlab/backups"

If you change this location, be sure to run the below after.

[email protected]:~# gitlab-ctl reconfigure

To actually take the application backup, run the following.

[email protected]:~# gitlab-rake gitlab:backup:create

This creates a backup in the following location.

[email protected]:~# ls -ltr /var/opt/gitlab/backups/
total 204
-rw------- 1 git git  71680 May 16 19:29 1526498956_2018_05_16_10.2.3_gitlab_backup.tar
-rw------- 1 git git 133120 May 16 20:14 1526501659_2018_05_16_10.7.3_gitlab_backup.tar

Host Backup (optional)

If rebuilding the machine and keeping the same IP, to avoid having to delete the host key entry in the ~/.ssh/know_hosts file, run the following to backup the SSH host keys.

[email protected]:~# tar -cvf $(date "+hostkeys-%s.tar") $(grep HostKey /etc/ssh/sshd_config | grep -v ^'#' | awk '{print $2}')

Transfer

Lets zip and compress these files into a single archive for transfer. Although I don’t plan on restoring the host keys, here I zip and compress all three all the same.

/var/opt/gitlab/backups/1526501659_2018_05_16_10.7.3_gitlab_backup.tar
/root/etc-gitlab-1526500078.tar
/root/hostkeys-1526501148.tar

Zip and compress with the following.

[email protected]:~# tar -cvzf ~/gitlab-backup.tar.gz /var/opt/gitlab/backups/1526501659_2018_05_16_10.7.3_gitlab_backup.tar /root/etc-gitlab-1526500078.tar /root/hostkeys-1526501148.tar

Use scp to copy the backup securely over the internet. Here I run the below from the new GitLab server to pull the above backup to the current working directory.

[email protected]:~# scp -v [email protected]:/root/gitlab-backup.tar.gz .

If you’d rather push it from the old server to the new, the command would be:

[email protected]:~# scp -v gitlab-backup.tar.gz  [email protected]:/root/

Restore

As I am mainly concerned about the Git repositories themselves, I will only be doing an application restore.

If you are moving to a new host with the same hostname etc., you should also restore the configuration backup to /etc/gitlab/. If you want everything to be the same but the hostname is different, you can manually amend anything such as external_url before running sudo gitlab-ctl reconfigure.

Extract the Restore

From the new GitLab host, as root, create the restore files.

[email protected]:~# mkdir gitlab-restore
[email protected]:~# mv gitlab-backup.tar.gz gitlab-restore/
[email protected]:~# cd gitlab-restore/
[email protected]:~/gitlab-restore# tar -xzvf gitlab-backup.tar.gz
[email protected]:~/gitlab-restore# ls -l
total 244
-rw-r--r-- 1 root root 240531 Nov 24 19:43 gitlab-backup.tar.gz
drwxr-xr-x 2 root root   4096 Nov 24 19:50 root
drwxr-xr-x 3 root root   4096 Nov 24 19:50 var

You should see something like that.

Application Backup Restore

Stop processes using the database.

[email protected]:~/gitlab-restore# gitlab-ctl stop unicorn
[email protected]:~/gitlab-restore# gitlab-ctl stop sidekiq
[email protected]:~/gitlab-restore# gitlab-ctl status

Check they’re actually down.

[email protected]:~/gitlab-restore# gitlab-ctl status | grep ^down

Move the application backup to the backups directory as defined in gitlab.rb and correct the owner if necessary.

[email protected]:~/gitlab-restore# cp -v var/opt/gitlab/backups/1526501659_2018_05_16_10.7.3_gitlab_backup.tar /var/opt/gitlab/backups/
[email protected]:~/gitlab-restore# chown -v git:git /var/opt/gitlab/backups/1526501659_2018_05_16_10.7.3_gitlab_backup.tar

Then restore. The restore name is the same as the above file name but without the _gitlab_backup.tar bit.

[email protected]:~/gitlab-restore# gitlab-rake gitlab:backup:restore BACKUP=1526501659_2018_05_16_10.7.3

Now restart GitLab.

[email protected]:~/gitlab-restore# gitlab-ctl start

And check for problems.

[email protected]:~/gitlab-restore# gitlab-rake gitlab:check SANITIZE=true

If all was successful, you should be able to log into the new site as before with whatever users you had setup. If you had 2FA enabled, you may need to restore the gitlab-secrets.json.

Configuration Backup Restore

Exactly what you choose to restore from the /etc/gitlab/ directory is up to you. Like me, none maybe necessary. If any of your users had 2FA setup, you will need to restore the gitlab-secrects.json file or else they will lose their access. If unsure, you could always do a diff to compare the differences.

[email protected]:~/gitlab-restore# cd root/
[email protected]:~/gitlab-restore/root# tar -vxf etc-gitlab-15*
[email protected]:~/gitlab-restore/root# diff -y --suppress-common-lines etc/gitlab/gitlab.rb /etc/gitlab/gitlab.rb

If moving to a new host but the hostname and everything should remain the same, you can simply move the old directory out the way and replace with the old.

First, stop unicorn and sidekiq at a minimum.

[email protected]:~/gitlab-restore/root# gitlab-ctl stop

Then restore the configuration directory.

[email protected]:~/gitlab-restore/root# mv -v /etc/gitlab{,.original}
[email protected]:~/gitlab-restore/root# mv etc/gitlab /etc/gitlab

Then start GitLab again and check.

[email protected]:~/gitlab-restore/root# gitlab-ctl start
[email protected]:~/gitlab-restore/root# gitlab-rake gitlab:check SANITIZE=true

Fix any issue and then re-run if necessary. Try re-configuring if you have issues.

 
[email protected]:~/gitlab-restore/root# gitlab-ctl reconfigure

If even the IP address remains the same as before and you want to prevent your users from seeing an SSH man-in-the-middle attack warning, you will also need to restore the HostKeys.

First extract the restored keys.

[email protected]:~/gitlab-restore/root# pwd
/root/gitlab-restore/root
[email protected]:~/gitlab-restore/root# ls -la hostkeys-1526501148.tar 
-rw-r--r-- 1 root root 10240 Nov 24 19:36 hostkeys-1526501148.tar
[email protected]:~/gitlab-restore/root# tar -xvf hostkeys-1526501148.tar 
etc/ssh/ssh_host_rsa_key
etc/ssh/ssh_host_dsa_key
etc/ssh/ssh_host_ecdsa_key
etc/ssh/ssh_host_ed25519_key

Then replace the HostKeys defined in your sshd_config.

[email protected]:~/gitlab-restore/root# grep HostKey /etc/ssh/sshd_config
# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
[/code]

…with the ones from your backup.

[email protected]:~/gitlab-restore/root# tar -tvf hostkeys-1526501148.tar.gz 
-rw------- root/root      1679 2017-12-05 13:37 etc/ssh/ssh_host_rsa_key
-rw------- root/root       672 2017-12-05 13:37 etc/ssh/ssh_host_dsa_key
-rw------- root/root       227 2017-12-05 13:37 etc/ssh/ssh_host_ecdsa_key
-rw------- root/root       399 2017-12-05 13:37 etc/ssh/ssh_host_ed25519_key

Rename the existing keys.

[email protected]:~/gitlab-restore/root#  for i in $(grep ^HostKey /etc/ssh/sshd_config | awk '{print $2}'); do sudo mv -v $i $i.bkp-$(date +%s);done

Then restore the old ones in-place.

[email protected]:~/gitlab-restore/root#  tar -xvzf hostkeys-1526501148.tar.gz -C /

Restart GitLab and check for errors.

[email protected]:~/gitlab-restore/root# gitlab-ctl restart
[email protected]:~/gitlab-restore/root# gitlab-rake gitlab:check SANITIZE=true

Resources

https://www.fabian-keller.de/blog/migrating-a-gitlab-instance-to-another-server
https://www.tienle.com/2016/05-24/migrate-gitlab-to-new-server.html
https://stackoverflow.com/questions/31534293/how-to-migrate-gitlab-to-a-new-server

Be the first to comment

Leave a Reply